Skip to content

Dynamic es.resource.write fails to find nested field #362

Closed
@aewhite

Description

@aewhite

Given the dynamic configuration...

hadoopConfiguration.set("es.resource.write", "{nested.field}/type");

And a document like...

{"nested" : { "field" : "value" } }

The load will fail with "Cannot find match for "{nested.field}". The work around is to either insert a dummy field before the target field or, if possible, re-order the nested sub fields.

I did manage to debug the issue down to org.elasticsearch.hadoop.serialization.ParsingUtils#doFind. In particular this section...

            if (token == Token.START_OBJECT) {
                token = parser.nextToken();  // <--- this eats the first FIELD_NAME token of the nested object
                if (matchingCurrentLevel == null) {
                    parser.skipChildren();
                }
                else {
                    doFind(parser, matchingCurrentLevel, active, inactive);
                }
            }

I expect moving the parser.nextToken() to inside the null check block would solve my particular problem since that is what's eating the FIELD_NAME token, but I don't know if it would break something else.

Edit: To clarify, if "field" is the first property inside the nested object, then the error is triggered. If "field" is the second or later field, it works as expected.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions