key_value ignoring whitespace

[robp1234]

I was directed here from the community forum, as this behaviour appears to be a bug (https://community.graylog.org/t/alert-processing-query/5964).

I have a set of data with key value pairs separated by a colon and a whitespace ": ". The whitespace is being ignored, resulting in data being split on a colon. This breaks the data, as it contains timestamps (e.g. 12:34:34).

Expected Behaviour

I have a data set coming into graylog, into a pipeline and then using a rule to extract the field names and data values.

email: rob@email.com, event: user login, time: 2018-07-18T08:02:19Z, user_id: 12345

My processing rule looks like this:

                    key_value(
                        value: to_string($message.message),
                        delimiters: ",",
                        kv_delimiters: ":",
                        allow_dup_keys:true,
                        handle_dup_keys: ","

This breaks, as it splits on the timestamp. I changed kv_delimiters to this:

kv_delimiters: ": "

This includes the whitespace, but the behaviour does not change. The split is done exclusively on the colon, breaking timestamps

I tried escaping the whitespace with \ but that results in an error.

Context

The separator is always a ": " but : can occur in a data field, as can whitespace. Being able to split on ": " would be extremely useful.

This is Graylog 2.4.5.

thanks
Rob

[radykal-com]

Hello, I'm trying to simulate the bug and noticed another bug.

Using the message in your example:

"email: rob@email.com, event: user login, time: 2018-07-18T08:02:19Z, user_id: 12345"

Not only the timestamp gets truncated in the first occurrence of the colon inside the timestamp but also the event gets truncated in the space character, so the event gets mapped to just "user" and the "login" gets lost.
Can you confirm this other bug is also happening in your environment?

[radykal-com]

Continuing with this, the problem resides in the class:

https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog/plugins/pipelineprocessor/functions/strings/KeyValue.java

Graylog is using CharMatcher from Guava library, that is specially built to work with single characters, not strings.
To solve this, it is required to avoid using the CharMatcher and use directly the Splitter passing strings instead of CharMatcher instances, otherwise if your delimiter strings have multiple characters it would split in any of them and not when all of them matches.

[radykal-com]

another approach would be to fix current behaviour to only apply the inner splitter on the first occurrence of the kv_delimiter char, so next occurrences inside the value string would not be treated.

This way it can still working with the CharMatcher.

So, how should I try to fix it?
Allow delimiters to be multi-char strings or just ensure that with single chars it works as expected?

[radykal-com]

I decided to just fix the behaviour using CharMatcher, after this fix, setting the kv_delimiter to ":" (without the space) will work as expected

[zez3]

@robp1234 Did you perhaps had '?' in your field values ?