Support adding fields from grok pattern

[nick-george]

Hi,

I have been making use of freely available logstash configs and patterns, like this (https://github.com/nxhack/logstash/blob/master/patterns/sshd) awesome one for SSHD. I've copied a small section of the file for illustrative purposes below:

# Normal
SSHD_SUCCESS        (?<sshd_result>Accepted) %{WORD:sshd_auth_type} for %{USERNAME:sshd_user} from %{IP:sshd_client_ip} port %{NUMBER:sshd_port} %{GREEDYDATA:sshd_protocol}
SSHD_DISCONNECT     Received disconnect from %{IP:sshd_client_ip}: (?<sshd_disconnect_code>\d+): %{GREEDYDATA:sshd_disconnect_reason}
SSHD_CONN_CLOSE     Connection closed by %{IP:sshd_client_ip}
SSHD_SESSION_OPEN   pam_unix\(sshd:session\): session opened for user %{USERNAME:sshd_user} by \(uid=\d+\)
SSHD_SESSION_CLOSE  pam_unix\(sshd:session\): session closed for user %{USERNAME:sshd_user}
SSHD_NORMAL_LOG %{SSHD_SUCCESS}|%{SSHD_DISCONNECT}|%{SSHD_CONN_CLOSE}|%{SSHD_SESSION_OPEN}|%{SSHD_SESSION_CLOSE}

What makes this so elegant is that we can have a single grok filter in logstash but match lots of potential log lines. This makes for a VERY clean logstash config.

Where this design lets us down is capturing the meaning/result of each of the log lines. You can see that there's an attempt in the top pattern to set the sshd_result field, and we could potentially add something similar to the rest of the patterns in the file. However, a much cleaner way would be if we could add arbitrary new fields from within the grok pattern itself.

An example of what I'm talking about is that if the SSHD_SUCCESS pattern matched, I'd like to create a field called 'action' with a value of 'login', and another field called 'result' with a value of 'success'. I'd like to do this (or similar) for all the patterns in the file. As far as I know, doing this at the moment requires creating a separate GROK for every different log line we want to support. This blows out the size of the logstash code big time.

I think the 'design pattern' above is the cleanest way to do grokking en masse. I'd really like to be able to use it all the time, but this apparent limitation makes is very difficult.

Should I be filing this issue against the https://github.com/jordansissel/ruby-grok repo instead?

Related discussion here: https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497

Regards,
Nick