Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic es.resource.write fails to find nested field #362

Closed
aewhite opened this issue Jan 21, 2015 · 6 comments
Closed

Dynamic es.resource.write fails to find nested field #362

aewhite opened this issue Jan 21, 2015 · 6 comments

Comments

@aewhite
Copy link

aewhite commented Jan 21, 2015

Given the dynamic configuration...

hadoopConfiguration.set("es.resource.write", "{nested.field}/type");

And a document like...

{"nested" : { "field" : "value" } }

The load will fail with "Cannot find match for "{nested.field}". The work around is to either insert a dummy field before the target field or, if possible, re-order the nested sub fields.

I did manage to debug the issue down to org.elasticsearch.hadoop.serialization.ParsingUtils#doFind. In particular this section...

            if (token == Token.START_OBJECT) {
                token = parser.nextToken();  // <--- this eats the first FIELD_NAME token of the nested object
                if (matchingCurrentLevel == null) {
                    parser.skipChildren();
                }
                else {
                    doFind(parser, matchingCurrentLevel, active, inactive);
                }
            }

I expect moving the parser.nextToken() to inside the null check block would solve my particular problem since that is what's eating the FIELD_NAME token, but I don't know if it would break something else.

Edit: To clarify, if "field" is the first property inside the nested object, then the error is triggered. If "field" is the second or later field, it works as expected.

@costin
Copy link
Member

costin commented Jan 23, 2015

To confirm, you are using a JSON input, correct?

@aewhite
Copy link
Author

aewhite commented Jan 23, 2015

Correct.

costin added a commit that referenced this issue Jan 26, 2015
relates to #362
related to commits 28ccb29 and
 3aadfa9
Note that the JSON values are still all returned as strings to avoid
 too many disruptive changes to the 2.0.x branch
@costin
Copy link
Member

costin commented Jan 26, 2015

@aewhite Looks like the problem occurs on 2.0.x branch and it has been fixed in master. Can you please try the latest dev builds published for both 2.1 and 2.0 branch and report back?

Thanks!

costin added a commit that referenced this issue Jan 27, 2015
@costin
Copy link
Member

costin commented Feb 13, 2015

@aewhite ping...

@aewhite
Copy link
Author

aewhite commented Feb 13, 2015

This appears resolved in the latest branch.

@costin
Copy link
Member

costin commented Apr 28, 2015

Great. Closing the issue.

@costin costin closed this as completed Apr 28, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants