Fix es.output.json for Cascading.
#898
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added tests to verify functionality. fixes elastic#885
tlrx
reviewed
Nov 23, 2016
| Pipe pipe = new Pipe("copy"); | ||
|
|
||
| Tap out = new HadoopPrintStreamTap(Stream.NULL); | ||
| build(cfg(), in, out, pipe); |
There was a problem hiding this comment.
For my education, do we check the result somewhere?
There was a problem hiding this comment.
There's no great way to check the result in this test case since it's writing in a separate process. We're mostly just checking that the job completes instead of throwing.
| context[0] = sourceCall.getInput().createKey(); | ||
| context[1] = sourceCall.getInput().createValue(); | ||
| // as the tuple _might_ vary (some objects might be missing), we use a map rather then a collection | ||
| Settings settings = loadSettings(flowProcess.getConfigCopy(), true); | ||
| context[2] = CascadingUtils.alias(settings); | ||
| context[3] = settings.getOutputAsJson(); |
There was a problem hiding this comment.
Would it make sense to use a constant for the index in the context array, something like
context[OUTPUT_AS_JSON] = settings.getOutputAsJson();
That would help to read the context array as well as quickly identify where it is read & set?
tlrx
reviewed
Nov 23, 2016
jbaiera
added a commit
that referenced
this pull request
May 8, 2017
Added tests to verify functionality. fixes #885
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If
es.output.jsonis enabled, instead of deserializing JSON data into a map object ES-Hadoop will simply return the raw JSON string to the user. This is returned from the Hadoop RecordReader as aTextobject containing the string data. When the Cascading integration receives data from a RecordReader or from a ScrollQuery, it automatically assumes that the data is a Map object. This leads to a cast exception when it receives a Text object instead.This relates to issue #885.