New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping: Geopoint with array broken (dynamic mapping): geo_point expected #6939

Closed
pierrre opened this Issue Jul 21, 2014 · 2 comments

Comments

Projects
None yet
5 participants
@pierrre

pierrre commented Jul 21, 2014

I use Elasticsearch with Logstash and dynamic mapping.
I my logs, I have geopoints with the array syntax.
It was OK with ES 1.0.X, but it's broken with ES 1.2.2.

Template:

curl -XPUT localhost:9200/_template/logstash -d '{
    "template": "logstash_*",
    "settings" : {
        "refresh_interval": "30s"
    },
    "mappings": {
        "logs": {
            "_all" : {
                "enabled": false
            },
            "dynamic_templates": [
                {
                    "location": {
                        "match": "location*",
                        "mapping": {
                            "type": "geo_point"
                        }
                    }
                },
                {
                    "generic": {
                        "match": "*",
                        "match_mapping_type": "string",
                        "mapping": {
                            "type": "string",
                            "index": "not_analyzed"
                        }
                    }
                }
            ],
            "dynamic_date_formats": [
                "dateOptionalTime",
                "yyyy-MM-dd",
                "yyyy-MM-dd HH:mm:ss"
            ]
        }
    }
}'

Delete index:

curl -XDELETE localhost:9200/logstash_test

Try to add a doc/log:

curl -XPOST localhost:9200/logstash_test/logs -d '{
  "location_array": [
    2.3069244,
    48.8881598
  ]
}'

It fails with this message:

[2014-07-21 11:31:49,168][INFO ][cluster.metadata         ] [fr-dev-01] [logstash_test] creating index, cause [auto(index api)], shards [5]/[1], mappings [logs]
[2014-07-21 11:31:49,465][DEBUG][action.index             ] [fr-dev-01] [logstash_test][1], node[zzMEQe9JSS6qEgw0oRgdVA], [P], s[STARTED]: Failed to execute [index {[logstash_test][logs][PhqP9fpgQkS4tHfOs105GQ], source[{"location_array":[2.3069244,48.8881598]}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:536)
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
    at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:373)
    at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:203)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:534)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:433)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.ElasticsearchParseException: geo_point expected
    at org.elasticsearch.common.geo.GeoUtils.parseGeoPoint(GeoUtils.java:421)
    at org.elasticsearch.index.mapper.geo.GeoPointFieldMapper.parse(GeoPointFieldMapper.java:530)
    at org.elasticsearch.index.mapper.object.ObjectMapper.parseDynamicValue(ObjectMapper.java:819)
    at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:639)
    at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:625)
    at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:482)
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
    ... 8 more

If I insert a doc with a geopoint as an object, it works, and the mapping is created dynamically.
After that, I can insert a doc with a geopoint as an array without error.

@bobpaulin

This comment has been minimized.

Show comment
Hide comment
@bobpaulin

bobpaulin Jul 26, 2014

This also appears to be happening on the master branch.

bobpaulin commented Jul 26, 2014

This also appears to be happening on the master branch.

@bobpaulin

This comment has been minimized.

Show comment
Hide comment
@bobpaulin

bobpaulin Jul 26, 2014

Seems like it's blowing up since the geo_point is only parsing the first value instead of the 2 item array as a whole. It works the second time through because the mapper object within org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray caches location_array as a geo_point so the array logic properly pulls the mapper. The first time through that entry is not there so the mapper comes up null and it tries to parse the array values individually. May I suggest adding some logic to the org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray method to run the template builder logic to find the GeoMapper

private void serializeArray(ParseContext context, String lastFieldName) throws IOException {
    String arrayFieldName = lastFieldName;
    Mapper mapper = mappers.get(lastFieldName);
    //Start Check TemplateBuilder for geo
    if (mapper == null) {
        BuilderContext builderContext = new BuilderContext(context.indexSettings(), context.path());
        Mapper.Builder builder = context.root().findTemplateBuilder(context, arrayFieldName, "geo_point");
        if (builder != null) {
            mapper = builder.build(builderContext);
            mappers.put(lastFieldName, mapper);
        }
    }
    //End Check TemplateBuilder for geo
    if (mapper != null && mapper instanceof ArrayValueMapperParser) {
        mapper.parse(context);
    } else {

bobpaulin commented Jul 26, 2014

Seems like it's blowing up since the geo_point is only parsing the first value instead of the 2 item array as a whole. It works the second time through because the mapper object within org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray caches location_array as a geo_point so the array logic properly pulls the mapper. The first time through that entry is not there so the mapper comes up null and it tries to parse the array values individually. May I suggest adding some logic to the org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray method to run the template builder logic to find the GeoMapper

private void serializeArray(ParseContext context, String lastFieldName) throws IOException {
    String arrayFieldName = lastFieldName;
    Mapper mapper = mappers.get(lastFieldName);
    //Start Check TemplateBuilder for geo
    if (mapper == null) {
        BuilderContext builderContext = new BuilderContext(context.indexSettings(), context.path());
        Mapper.Builder builder = context.root().findTemplateBuilder(context, arrayFieldName, "geo_point");
        if (builder != null) {
            mapper = builder.build(builderContext);
            mappers.put(lastFieldName, mapper);
        }
    }
    //End Check TemplateBuilder for geo
    if (mapper != null && mapper instanceof ArrayValueMapperParser) {
        mapper.parse(context);
    } else {

@jpountz jpountz removed the adoptme label Aug 1, 2014

colings86 added a commit to colings86/elasticsearch that referenced this issue Aug 11, 2014

Mapping: fixes dynamic mapping of geo_point fields
If a dynamic mapping for a geo_point field is defined and the first document specifies the value of the field as a geo_point array, the dynamic mapping throws an error as the array is broken into individual number before consulting the dynamic mapping configuration.  This change adds a check of the dynamic mapping before the array is split into individual numbers.

Closes #6939

@colings86 colings86 closed this in #7175 Aug 11, 2014

colings86 added a commit that referenced this issue Aug 11, 2014

Mapping: fixes dynamic mapping of geo_point fields
If a dynamic mapping for a geo_point field is defined and the first document specifies the value of the field as a geo_point array, the dynamic mapping throws an error as the array is broken into individual number before consulting the dynamic mapping configuration.  This change adds a check of the dynamic mapping before the array is split into individual numbers.

Closes #6939

@colings86 colings86 changed the title from Geopoint with array broken (dynamic mapping): geo_point expected to Mapping: Geopoint with array broken (dynamic mapping): geo_point expected Aug 12, 2014

colings86 added a commit that referenced this issue Sep 8, 2014

Mapping: fixes dynamic mapping of geo_point fields
If a dynamic mapping for a geo_point field is defined and the first document specifies the value of the field as a geo_point array, the dynamic mapping throws an error as the array is broken into individual number before consulting the dynamic mapping configuration.  This change adds a check of the dynamic mapping before the array is split into individual numbers.

Closes #6939
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment