Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON parsing should not fail on missing fields #209

Closed
lennartkoopmann opened this issue Sep 1, 2017 · 1 comment
Closed

JSON parsing should not fail on missing fields #209

lennartkoopmann opened this issue Sep 1, 2017 · 1 comment
Assignees
Milestone

Comments

@lennartkoopmann
Copy link
Member

@lennartkoopmann lennartkoopmann commented Sep 1, 2017

Problem description

When parsing a JSON message, the pipeline rule fails if a field does not exist, but it defined in select_jsonpath.

For example, this will fail:

let json_result = parse_json(to_string($message.message));
let json_fields = select_jsonpath(json_result,
            { source: "$.source",
              some_field: "$.i_dont_exist"
            });

The error message will be:

For rule 'f5_test_rule': In call to function 'select_jsonpath' at 8:22 an exception was thrown: null

I suggest ignoring missing fields by default and introducing a new optional boolean parameter to toggle this behaviour.

Environment

  • Graylog Version: 2.3.1
  • Pipeline Processor plugin version: 2.3.1
@lennartkoopmann lennartkoopmann added this to the 2.4.0 milestone Sep 1, 2017
@lennartkoopmann lennartkoopmann changed the title JSON parsing should not fail on missing fields. JSON parsing should not fail on missing fields Sep 1, 2017
@jalogisch jalogisch added the triaged label Sep 5, 2017
joschi pushed a commit that referenced this issue Sep 6, 2017
Jochen Schalanda
The `parse_json()` function is supposed to return a `JsonNode` but returned
`null` if the input wasn't valid JSON.

This change set changes the return type to `MissingNode` if the input wasn't
valid and couldn't be parsed.

Closes #209
@joschi
Copy link
Contributor

@joschi joschi commented Sep 6, 2017

@lennartkoopmann I'm unable to reproduce this issue with Graylog 2.3.1 and 2.4.0-SNAPSHOT.

The test case I've been using is directly derived from your example:

diff --git a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
index e036970..75c4b7c 100644
--- a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
+++ b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
@@ -7,4 +7,7 @@ then
               author_last: "$['store']['book'][-1:]['author']"
             });
   set_fields(new_fields);
+
+  // Don't fail on missing field
+  let missing_field = select_jsonpath(x, { some_field: "$.i_dont_exist" });
 end
\ No newline at end of file

It looks like json_result was null when you tried running the JSON path extraction on it.

The parse_json() function may return null if the input wasn't valid JSON:

try {
return objectMapper.readTree(value);
} catch (IOException e) {
log.warn("Unable to parse json", e);
}
return null;

This all being said, I think the parse_json() function shouldn't return null and the json_path() function should be able to handle null as an input.

@joschi joschi self-assigned this Sep 6, 2017
@ghost ghost added the in progress label Sep 6, 2017
@bernd bernd closed this in #210 Sep 11, 2017
bernd added a commit that referenced this issue Sep 11, 2017
The `parse_json()` function is supposed to return a `JsonNode` but returned
`null` if the input wasn't valid JSON.

This change set changes the return type to `MissingNode` if the input wasn't
valid and couldn't be parsed.

Closes #209
@ghost ghost removed the in progress label Sep 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants