New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON parsing should not fail on missing fields #209

Closed
lennartkoopmann opened this Issue Sep 1, 2017 · 1 comment

Comments

Projects
None yet
3 participants
@lennartkoopmann
Member

lennartkoopmann commented Sep 1, 2017

Problem description

When parsing a JSON message, the pipeline rule fails if a field does not exist, but it defined in select_jsonpath.

For example, this will fail:

let json_result = parse_json(to_string($message.message));
let json_fields = select_jsonpath(json_result,
            { source: "$.source",
              some_field: "$.i_dont_exist"
            });

The error message will be:

For rule 'f5_test_rule': In call to function 'select_jsonpath' at 8:22 an exception was thrown: null

I suggest ignoring missing fields by default and introducing a new optional boolean parameter to toggle this behaviour.

Environment

  • Graylog Version: 2.3.1
  • Pipeline Processor plugin version: 2.3.1

@lennartkoopmann lennartkoopmann added this to the 2.4.0 milestone Sep 1, 2017

@lennartkoopmann lennartkoopmann changed the title from JSON parsing should not fail on missing fields. to JSON parsing should not fail on missing fields Sep 1, 2017

@jalogisch jalogisch added the triaged label Sep 5, 2017

joschi added a commit that referenced this issue Sep 6, 2017

parse_json() returns MissingNode if input wasn't valid JSON
The `parse_json()` function is supposed to return a `JsonNode` but returned
`null` if the input wasn't valid JSON.

This change set changes the return type to `MissingNode` if the input wasn't
valid and couldn't be parsed.

Closes #209
@joschi

This comment has been minimized.

Contributor

joschi commented Sep 6, 2017

@lennartkoopmann I'm unable to reproduce this issue with Graylog 2.3.1 and 2.4.0-SNAPSHOT.

The test case I've been using is directly derived from your example:

diff --git a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
index e036970..75c4b7c 100644
--- a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
+++ b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
@@ -7,4 +7,7 @@ then
               author_last: "$['store']['book'][-1:]['author']"
             });
   set_fields(new_fields);
+
+  // Don't fail on missing field
+  let missing_field = select_jsonpath(x, { some_field: "$.i_dont_exist" });
 end
\ No newline at end of file

It looks like json_result was null when you tried running the JSON path extraction on it.

The parse_json() function may return null if the input wasn't valid JSON:

try {
return objectMapper.readTree(value);
} catch (IOException e) {
log.warn("Unable to parse json", e);
}
return null;

This all being said, I think the parse_json() function shouldn't return null and the json_path() function should be able to handle null as an input.

@joschi joschi self-assigned this Sep 6, 2017

@wafflebot wafflebot bot added the in progress label Sep 6, 2017

@bernd bernd closed this in #210 Sep 11, 2017

bernd added a commit that referenced this issue Sep 11, 2017

parse_json() returns MissingNode if input wasn't valid JSON (#210)
The `parse_json()` function is supposed to return a `JsonNode` but returned
`null` if the input wasn't valid JSON.

This change set changes the return type to `MissingNode` if the input wasn't
valid and couldn't be parsed.

Closes #209

@wafflebot wafflebot bot removed the in progress label Sep 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment