Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ESQL: Support loading shapes from source into WKB blocks #104269

Merged
merged 22 commits into from
Jan 17, 2024
Merged

ESQL: Support loading shapes from source into WKB blocks #104269

merged 22 commits into from
Jan 17, 2024

Conversation

iverase
Copy link
Contributor

@iverase iverase commented Jan 11, 2024 •
edited by craigtaverner
Loading

Similarly to #103698, this PR adds support for reading geo_shape and shape fields into wkb blocks so they can use top project the data to the result of a ES|QL query.

One of the points for discussion in this PR is the naming use to the ES|QL fields where geo_shape has been renamed to geography and shape has been renamed to geometry, to follow more closely industry standards. Although this is debatable.

Note to the reviewer: There is a limitation on csv test because they cannot deal with WKT,

Fixes #103586

@iverase iverase added >enhancement :Analytics/Geo Indexing, search aggregations of geo points and shapes :Analytics/ES|QL AKA ESQL v8.13.0 labels Jan 11, 2024
@github-actions GitHub Actions
Copy link

Documentation preview:

@elasticsearchmachine
Copy link
Collaborator

Hi @iverase, I've created a changelog YAML for you.

@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jan 11, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@iverase iverase changed the title ESQL: Support loading points from source into WKB blocks ESQL: Support loading shapes from source into WKB blocks Jan 11, 2024
craigtaverner
craigtaverner requested changes Jan 15, 2024
View reviewed changes
Copy link
Contributor

@craigtaverner craigtaverner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really great. Just need to add ToGeometryTests and ToGeographyTests.

docs/changelog/104269.yaml Outdated Show resolved Hide resolved
docs/changelog/104269.yaml Outdated Show resolved Hide resolved
server/src/main/java/org/elasticsearch/index/mapper/AbstractPointGeometryFieldMapper.java Show resolved Hide resolved
x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial_shapes.csv-spec Outdated Show resolved Hide resolved
x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial_shapes.csv-spec Outdated Show resolved Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java Outdated Show resolved Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java Outdated Show resolved Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/planner/PlannerUtils.java Outdated Show resolved Hide resolved
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java Show resolved Hide resolved
x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java Show resolved Hide resolved
@iverase iverase mentioned this pull request Jan 16, 2024
Merged
craigtaverner
craigtaverner approved these changes Jan 16, 2024
View reviewed changes
Copy link
Contributor

@craigtaverner craigtaverner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

craigtaverner
craigtaverner reviewed Jan 16, 2024
View reviewed changes
Copy link
Contributor

@craigtaverner craigtaverner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that you have added types/to_geometry.asciidoc but not the parent to_geometry.asciidoc that should include that, nor any additions to the function list. Am I missing something, or are the docs incomplete?

@iverase
Copy link
Contributor Author

iverase commented Jan 16, 2024

That was generated, I did not add it.

@nik9000
Copy link
Member

nik9000 commented Jan 17, 2024

I noticed that you have added types/to_geometry.asciidoc but not the parent to_geometry.asciidoc that should include that, nor any additions to the function list. Am I missing something, or are the docs incomplete?

If the "parent" asciidoc file is missing then the docs are incomplete. But that doesn't have a block a huge PR.

nik9000
nik9000 approved these changes Jan 17, 2024
View reviewed changes
...ugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java
Map.entry(AIRPORTS.indexName, AIRPORTS),
Map.entry(AIRPORTS_WEB.indexName, AIRPORTS_WEB),
Map.entry(COUNTRIES_BBOX.indexName, COUNTRIES_BBOX),
Map.entry(COUNTRIES_BBOX_WEB.indexName, COUNTRIES_BBOX_WEB)
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

@iverase iverase merged commit 585face into elastic:main Jan 17, 2024
15 checks passed
@iverase iverase deleted the shapesESQL branch January 17, 2024 13:55
@craigtaverner craigtaverner mentioned this pull request Jan 17, 2024
Merged
@nreese nreese mentioned this pull request Jan 18, 2024
Merged
nreese added a commit to elastic/kibana that referenced this pull request Jan 22, 2024
@nreese @kibanamachine
[Maps] ESQL geo_shape support (#175156)
f6caec7 
elastic/elasticsearch#104269 adds geo_shape
support to ESQL

This PR updates maps ESQL source to support geo_shape column type
<img width="800" alt="Screenshot 2024-01-18 at 1 15 31 PM"
src="https://github.com/elastic/kibana/assets/373691/844251a6-4409-45f4-89f5-513f4537d833">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@craigtaverner craigtaverner mentioned this pull request Jan 23, 2024
Closed
lcawl pushed a commit to lcawl/kibana that referenced this pull request Jan 26, 2024
@nreese @kibanamachine @lcawl
[Maps] ESQL geo_shape support (elastic#175156)
f37c576 
elastic/elasticsearch#104269 adds geo_shape
support to ESQL

This PR updates maps ESQL source to support geo_shape column type
<img width="800" alt="Screenshot 2024-01-18 at 1 15 31 PM"
src="https://github.com/elastic/kibana/assets/373691/844251a6-4409-45f4-89f5-513f4537d833">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
@nreese @kibanamachine @CoenWarmer
[Maps] ESQL geo_shape support (elastic#175156)
55bec2e 
elastic/elasticsearch#104269 adds geo_shape
support to ESQL

This PR updates maps ESQL source to support geo_shape column type
<img width="800" alt="Screenshot 2024-01-18 at 1 15 31 PM"
src="https://github.com/elastic/kibana/assets/373691/844251a6-4409-45f4-89f5-513f4537d833">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
@nreese @kibanamachine @CoenWarmer
[Maps] ESQL geo_shape support (elastic#175156)
8dc22fb 
elastic/elasticsearch#104269 adds geo_shape
support to ESQL

This PR updates maps ESQL source to support geo_shape column type
<img width="800" alt="Screenshot 2024-01-18 at 1 15 31 PM"
src="https://github.com/elastic/kibana/assets/373691/844251a6-4409-45f4-89f5-513f4537d833">

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@craigtaverner craigtaverner craigtaverner approved these changes

@nik9000 nik9000 nik9000 approved these changes

Assignees
No one assigned
Labels
:Analytics/ES|QL AKA ESQL :Analytics/Geo Indexing, search aggregations of geo points and shapes >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v8.13.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support geo_shape in ES|QL
4 participants
@iverase @elasticsearchmachine @nik9000 @craigtaverner