Add Z value support to geo_point and geo_shape #25738

nknize · 2017-07-15T17:54:36Z

This enhancement adds Z value support (source only) to geo_shape fields. If vertices are provided with a third dimension, the third dimension is ignored for indexing but returned as part of source. Like before, any values greater than the 3rd dimension are ignored.

closes #22917

jpountz · 2017-07-17T15:59:47Z

core/src/main/java/org/elasticsearch/common/geo/builders/ShapeBuilder.java

    }

    protected static void writeCoordinateTo(Coordinate coordinate, StreamOutput out) throws IOException {
        out.writeDouble(coordinate.x);
        out.writeDouble(coordinate.y);
+        out.writeOptionalDouble(Double.isNaN(coordinate.z) ? null : coordinate.z);


you need to add a version check here?

jpountz · 2017-07-17T16:00:02Z

core/src/main/java/org/elasticsearch/common/geo/builders/ShapeBuilder.java

-        return new Coordinate(in.readDouble(), in.readDouble());
+        double x = in.readDouble();
+        double y = in.readDouble();
+        Double z = in.readOptionalDouble();


you need to add a version check here?

jpountz · 2017-07-17T16:02:05Z

core/src/test/java/org/elasticsearch/common/geo/ShapeBuilderTests.java

+            .coordinate(new Coordinate(-45, -30, 101))
+            .coordinate(new Coordinate(-45, 30, 110)));
+
+        assertEquals(expected, pb.toString());


we should test parsing too?

added testParse3DPolygon to GeoJSONShapeParserTests

hanoch · 2017-07-17T21:23:36Z

@nknize #22917 talks about both geo_point and geo_shape. does this PR include support for store-and-retrieve Z values for geo_point as well, or will that support be separate from this PR?

nknize · 2017-08-02T20:14:19Z

@hanoch geo_point already supports Z values in source.

This PR effectively fixes a bug where Z values for geo_shape coordinates were omitted in source after indexing.

javanna · 2017-11-06T20:20:06Z

@nknize are you still working on this?

hanoch · 2017-11-06T21:04:49Z

@nknize we are still waiting for this functionality to be merged into the master branch.

hanoch · 2017-11-06T21:05:19Z

cc: @sherry-ger

nknize · 2017-11-10T00:57:12Z

@colings86 This should be good to go. Could you give it a final once over since @jpountz is out? Thx

clintongormley · 2017-11-10T15:11:06Z

Discussed this in FixItFriday. We're not crazy about accepting parameters that we will silently ignore, but given that Z values are part of the geojson spec, we should probably make an effort to extract what info we can from valid geojson files.

It should, however, be enabled by opting in using a mapping parameter like accept_but_ignore_z_values, which makes clear to the user what they are opting in to.

colings86

@nknize this looks ok to me but could you add the option that @clintongormley mentioned and a tests with 3 dimensional polygons with and without that setting?

hanoch · 2018-02-09T00:11:07Z

are we finally ready to merge this to master anytime soon?

hanoch · 2018-02-14T16:02:23Z

@colings86 @jpountz @nknize is this now ready to be merged to master to be able to address #22917?

nknize · 2018-02-20T21:18:32Z

Rebased to master and cleaned up some geo_point parsing. This is ready for another review and should be just about ready to merge. /cc @jpountz @colings86

colings86

LGTM, its a shame the changes are so pervasive in the code and the acceptZValue option has to be passed around so much but unfortunately I don't think there is a better way to do it.

jpountz

Thanks for making the docs very explicit about how the 3rd dimension is handled. I left some comments but it looks like it is on the right track to me.

jpountz · 2018-01-18T22:25:11Z

server/src/main/java/org/elasticsearch/common/geo/builders/CircleBuilder.java

@@ -173,6 +173,10 @@ public String toWKT() {
        throw new UnsupportedOperationException("The WKT spec does not support CIRCLE geometry");
    }

+    public int numDimensions() {
+        return center.z == Double.NaN ? 2 : 3;


beware it might not do what you think: https://stackoverflow.com/questions/8819738/why-does-double-nan-double-nan-return-false

ah, yes... thank you. I still slip up w/ this one occasionally.

jpountz · 2018-01-18T22:26:24Z

server/src/main/java/org/elasticsearch/common/geo/builders/CoordinatesBuilder.java

-        this.points.add(coordinate);
+        if (points.isEmpty() == false && Double.isNaN(points.get(0).z) != Double.isNaN(coordinate.z)) {
+            throw new ElasticsearchException("unable to add coordinate to CoordinateBuilder: " +
+                "coordinate dimensions do not match");


Can you add the expected number of dims and the new coordinate to the error message?

jpountz · 2018-01-18T22:31:50Z

server/src/main/java/org/elasticsearch/index/mapper/GeoShapeFieldMapper.java

@@ -452,12 +472,15 @@ public Query termQuery(Object value, QueryShardContext context) {

    protected Explicit<Boolean> coerce;
    protected Explicit<Boolean> ignoreMalformed;
+    protected Explicit<Boolean> storeZValue;


we don't seem to be making decisions based on this option? or do I miss it?

jpountz · 2018-02-21T14:21:08Z

docs/reference/mapping/types/geo-point.asciidoc

@@ -105,6 +105,13 @@ The following parameters are accepted by `geo_point` fields:
    If `true`, malformed geo-points are ignored. If `false` (default),
    malformed geo-points throw an exception and reject the whole document.

+<<accept_z_value,`accept_z_value`>>::


should it be called something like ignore_z_value instead to make it clear it's ignored?

jpountz · 2018-02-21T14:25:33Z

server/src/main/java/org/elasticsearch/common/Strings.java

+     * @param ch  the char to count
+     * @return the number of occurrences, 0 if the CharSequence is {@code null}
+     */
+    public static int countMatches(final CharSequence str, final char ch) {


reading the rest of the PR, it doesn't seem like we really need this method? We could just check whether the string contains a comma to know whether it is a geohash and then use the length of the array after splitting to know the number of dimensions?

jpountz · 2018-02-21T14:28:55Z

server/src/main/java/org/elasticsearch/common/geo/GeoUtils.java

        double lat = Double.NaN;
        double lon = Double.NaN;
+        double alt = Double.NaN;


no need for this alt variable

jpountz · 2018-02-21T14:32:40Z

server/src/main/java/org/elasticsearch/common/geo/builders/ShapeBuilder.java

+        double x = in.readDouble();
+        double y = in.readDouble();
+        Double z = null;
+        if (in.getVersion().onOrAfter(Version.V_6_0_0_beta1)) {


this needs to be version 6.3 now

jpountz · 2018-02-21T14:32:51Z

server/src/main/java/org/elasticsearch/common/geo/builders/ShapeBuilder.java

@@ -123,6 +130,9 @@ public void writeTo(StreamOutput out) throws IOException {
    protected static void writeCoordinateTo(Coordinate coordinate, StreamOutput out) throws IOException {
        out.writeDouble(coordinate.x);
        out.writeDouble(coordinate.y);
+        if (out.getVersion().onOrAfter(Version.V_6_1_0)) {


should be 6.3

jpountz · 2018-02-21T14:35:09Z

server/src/main/java/org/elasticsearch/common/geo/parsers/GeoJsonParser.java

+        if (token == XContentParser.Token.VALUE_NUMBER) {
+            GeoPoint.assertZValue(acceptZValue);
+            alt = parser.doubleValue();
+        }
        while (token == XContentParser.Token.VALUE_NUMBER) {
            token = parser.nextToken();
        }
        // todo support z/alt


remove todo

jpountz · 2018-02-21T15:01:40Z

server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java

    }

    public static class Defaults {
        public static final Explicit<Boolean> IGNORE_MALFORMED = new Explicit<>(false, false);
        public static final GeoPointFieldType FIELD_TYPE = new GeoPointFieldType();
+        public static final Explicit<Boolean> ACCEPT_Z_VALUE = new Explicit<>(true, false);


true by default? shouldn't it be false like the docs say? (we might need some logic to make it true for pre-6.3 indices in case we used to accept it in the past)

jpountz

Logic looks good to me overall. I think we should make the default of ignore_z_value false as it could otherwise be a bit trappy? For shapes we can do it directly in this PR since they never supported 3 dimensions (if my understanding is correct). And for points which were always lenient until now (again, if my understanding is correct), we would make the default true in this PR (like you did) but do a follow-up PR to change the default to false on 7.0+ indices

jpountz · 2018-02-23T10:31:48Z

server/src/main/java/org/elasticsearch/common/geo/GeoPoint.java

+
+    public GeoPoint resetFromString(String value, final boolean ignoreZValue) {
+        if (value.contains(",")) {
+            String[] vals = value.split(",");


please fail if vals.length > 3

@jpountz @nknize Wouldn't it be better to be more lenient and accept the first 3 (or 2) dimensions, rather than completely fail? Adding failure now might break compatibility with previous behavior.

jpountz · 2018-02-23T10:40:15Z

server/src/main/java/org/elasticsearch/index/mapper/GeoShapeFieldMapper.java

@@ -100,6 +100,7 @@
        public static final double LEGACY_DISTANCE_ERROR_PCT = 0.025d;
        public static final Explicit<Boolean> COERCE = new Explicit<>(false, false);
        public static final Explicit<Boolean> IGNORE_MALFORMED = new Explicit<>(false, false);
+        public static final Explicit<Boolean> ACCEPT_Z_VALUE = new Explicit<>(true, false);


s/ACCEPT/IGNORE/

jpountz · 2018-02-23T10:43:16Z

server/src/main/java/org/elasticsearch/common/geo/parsers/GeoJsonParser.java

+        double alt = Double.NaN;
+        if (token == XContentParser.Token.VALUE_NUMBER) {
+            alt = GeoPoint.assertZValue(ignoreZValue, parser.doubleValue());
+        }
        while (token == XContentParser.Token.VALUE_NUMBER) {
            token = parser.nextToken();
        }


let's remove this leniency?

hanoch · 2018-02-23T16:38:45Z

@jpountz @nknize what is the overall design goal wrt the default behavior with Z values with geo_shape and geo_point?
It would be best if the default setting and behavior will continue to be that if the data value (csv-string/array/GeoJson/etc.) has a third Z dimension - it will be stored without needing to set any special settings. Also, it will be best to continue to auto ignore any 4th (e.g. M value) and above dimension in the data by default (without adjusting setting to accept the data) and store the first 3 (or 2 if the Z value is not present) dimensions, without failing if there is a 4th or more dimension.
That behavior will be consistent with the existing geo_point [x,y,z] or [x,y] array data values (and we should match the csv-string values with a Z value "y,x,z" or "y,x" as well) of existing and previous releases, where if the data has a Z value it will be stored, without needing to set some special settings, or modify the default settings.
Is that the design?

colings86 · 2018-02-23T17:33:23Z

@hanoch we do not want the Z Values to be accepted by default because it can be surprising to users. The fact that the z value is accepted but is not indexed and therefore does not affect search results is not what a user would expect OOTB so we feel it is a better user experience for the user to opt-in to accepting but ignoring the Z-Value.

hanoch · 2018-03-20T21:33:06Z

is there anything else pending or are we ready to merge this PR?

This enhancement adds Z value support (source only) to geo_shape fields. If vertices are provided with a third dimension, the third dimension is ignored for indexing but returned as part of source. Like beofre, any values greater than the 3rd dimension are ignored. closes elastic#23747

nknize added :Analytics/Geo Indexing, search aggregations of geo points and shapes review v5.6.0 v6.0.0 labels Jul 15, 2017

jpountz requested changes Jul 17, 2017

View reviewed changes

nknize force-pushed the enhancement/22917 branch from 6182333 to bc29b96 Compare July 24, 2017 21:17

clintongormley added v6.1.0 and removed v5.6.0 labels Jul 25, 2017

nknize force-pushed the enhancement/22917 branch from bc29b96 to a2118c1 Compare August 2, 2017 19:42

nknize force-pushed the enhancement/22917 branch 2 times, most recently from 52ee020 to fdf07e0 Compare September 19, 2017 17:38

nknize force-pushed the enhancement/22917 branch from fdf07e0 to d9383f3 Compare November 2, 2017 17:56

nknize requested a review from colings86 November 9, 2017 03:06

nknize force-pushed the enhancement/22917 branch 2 times, most recently from 1744096 to 0a7868d Compare November 9, 2017 23:53

colings86 reviewed Nov 10, 2017

View reviewed changes

lcawl added v6.0.1 and removed v6.0.0 labels Nov 13, 2017

nknize removed review v6.0.1 labels Nov 21, 2017

nknize mentioned this pull request Nov 21, 2017

[Geo] Add Well Known Text (WKT) Parsing Support to ShapeBuilders #27417

Merged

lcawl added v6.2.0 and removed v6.1.0 labels Dec 12, 2017

nknize force-pushed the enhancement/22917 branch from 0a7868d to 300fde3 Compare January 16, 2018 15:27

colings86 added v6.3.0 and removed v6.2.0 labels Jan 22, 2018

nknize force-pushed the enhancement/22917 branch from 7e215b5 to e473d77 Compare February 20, 2018 21:15

nknize changed the title ~~Add Z value support to geo_shape~~ Add Z value support to geo_point and geo_shape Feb 20, 2018

nknize added the review label Feb 20, 2018

colings86 approved these changes Feb 21, 2018

View reviewed changes

jpountz reviewed Feb 21, 2018

View reviewed changes

jpountz reviewed Feb 23, 2018

View reviewed changes

nknize force-pushed the enhancement/22917 branch 2 times, most recently from 05cc21a to 59c65a6 Compare March 23, 2018 13:46

nknize force-pushed the enhancement/22917 branch from 59c65a6 to fede633 Compare March 23, 2018 13:51

nknize merged commit fede633 into elastic:master Mar 23, 2018

nknize removed the review label Mar 23, 2018

jpountz added v7.0.0 >feature labels Jun 13, 2018

russcam mentioned this pull request Jul 9, 2018

Support 3rd dimension (Z value) to points for geo_point and points in geo_shapes elastic/elasticsearch-net#3318

Closed

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Add Z value support to geo_point and geo_shape #25738

Add Z value support to geo_point and geo_shape #25738

Conversation

nknize commented Jul 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanoch commented Jul 17, 2017

nknize commented Aug 2, 2017

javanna commented Nov 6, 2017

hanoch commented Nov 6, 2017

hanoch commented Nov 6, 2017

nknize commented Nov 10, 2017

clintongormley commented Nov 10, 2017

colings86 left a comment

Choose a reason for hiding this comment

hanoch commented Feb 9, 2018

hanoch commented Feb 14, 2018 • edited

nknize commented Feb 20, 2018

colings86 left a comment

Choose a reason for hiding this comment

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanoch Feb 23, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanoch commented Feb 23, 2018 • edited

colings86 commented Feb 23, 2018 • edited

hanoch commented Mar 20, 2018

hanoch commented Feb 14, 2018 •

edited

hanoch Feb 23, 2018 •

edited

hanoch commented Feb 23, 2018 •

edited

colings86 commented Feb 23, 2018 •

edited