Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

json: accept more characters in extract-prefix for json-parser() #371

Conversation

vincentbernat
Copy link
Contributor

When receiving JSON objects whose keys have @ or - characters, it is
currently not possible to extract them with extract-prefix since they
are not considered as a valid JS identifier.

In JS, bracket notation should be used instead when accessing such
properties. This is not implemented in syslog-ng as only array indices
can be accessed with this notation.

This commit extends the set of supported characters to @ and -. -
can be easily found in the wild while @ is part of the event log
format of Logstash (@version and @timestamp).

This commit could be extended to support $ (which is a valid character
in a JS identifier) but this may conflict with variable expansion. Any
character other than [ and . could be accepted as they don't have a
meaning in the dot notation.

This PR can also be applied on previous versions of syslog-ng (at least 3.6).

@vincentbernat
Copy link
Contributor Author

As an example, I am able to transform a v0 json file to a v1 version with this parser:

parser p_jsoneventv0 {
  # Transform to v1
  channel {
    parser { json-parser(extract-prefix("@fields")); };
    parser { json-parser(prefix(".json.")); };
    rewrite {
      set("1" value("@version"));
      set("${.json.@timestamp}" value("@timestamp"));
      set("${.json.@message}" value("message"));
    };
  };
};

@bazsi
Copy link
Collaborator

bazsi commented Jan 21, 2015

I also like this change :) And I am really happy to see such well thought out contributions. And with tests :)

@lbudai
Copy link
Collaborator

lbudai commented Jan 21, 2015

Squashing the commits is good idea ;-)

When receiving JSON objects whose keys have `@` or other non-JS-approved
characters, it is currently not possible to extract them with
`extract-prefix` since they are not considered as a valid JS identifier.

In JS, bracket notation should be used instead when accessing such
properties. This is not implemented in syslog-ng as only array indices
can be accessed with this notation.

This commit extends the set of supported characters to:

     !"#$%&'()*+,-/:;<=>?@\^_`{|}~

Which is everything printable in the ASCII range except `.`, `[` and
`]` (just for equity with its sister).

Signed-off-by: Vincent Bernat <vincent@bernat.im>
@vincentbernat vincentbernat force-pushed the feature/accept-more-chars-in-json-extract branch from a842d9e to 2d04c7a Compare January 21, 2015 20:29
@vincentbernat
Copy link
Contributor Author

Squashed.

@lbudai
Copy link
Collaborator

lbudai commented Jan 21, 2015

Thanks!!

lbudai added a commit that referenced this pull request Jan 21, 2015
…in-json-extract

json: accept more characters in `extract-prefix` for `json-parser()`
@lbudai lbudai merged commit 9fd1f61 into syslog-ng:master Jan 21, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants