insert fields based on grok matches

[cjcc1]

Would it be possible to assign a value to some extra field depending on which grok pattern matches. To slightly modify the example from another question (#44):

<source>
  @type tail
  path /path/to/log
  parsetype_key myeventtype
  <parse>
    @type grok
    <grok>
      pattern %{COMBINEDAPACHELOG}
      time_format "%d/%b/%Y:%H:%M:%S %z"
      parsetype_value apachelog
    </grok>
    <grok>
      pattern %{IP:ip_address}
      parsetype_value ipline
    </grok>
    <grok>
      pattern %{GREEDYDATA:message}
      parsetype_value message
    </grok>
  </parse>
</source>

The resulting event would have a new field named 'myeventtype' which would have the value of 'apachelog', 'ipline', 'message' depending on which of the grok patterns hit. If they all miss, maybe the default could be 'unknown' or something. The parsetype_key and parsetype_value configuration names are just suggestions on how it might be configured.

So, a good idea, or is there perhaps already a better way to achieve this sort of thing?

[okkez]

Could you show me the actual logs you expected and expected the result?

[cjcc1]

Don't misunderstand ... this is not a bug; it's more of a feature suggestion or question on what I'm doing wrong if I'm just missing how this sort of thing is done.

I just didn't know where else to post my question.

[okkez]

I know this is a feature request or question. But I could not understand your situation enough. I want to understand what you expected.

[okkez]

Do you want to add grok_failure_key for records that are not matched any grok patterns?
Do you want to add grok_success_key for records that are matched a grok pattern?

For example:

<source>
  @type dummy
  @label @dummy
  dummy [
    { "message1": "no grok pattern matched!", "prog": "foo" },
    { "message1": "/", "prog": "bar" }
  ]
  tag dummy.log
</source>

<label @dummy>
  <filter>
    @type parser
    key_name message1
    reserve_data true
    reserve_time true
    <parse>
      @type grok
      grok_failure_key grokfailure
      grok_success_key groksuccess
      <grok>
        pattern %{PATH:path}
      </grok>
    </parse>
  </filter>
  <match dummy.log>
    @type stdout
  </match>
</label>

Added grokfailure and groksuccess to parsed record:

2016-11-28 13:07:08.009131727 +0900 dummy.log: {"message1":"no grok pattern matched!","prog":"foo","message":"no grok pattern matched!","grokfailure":"No grok pattern matched"}
2016-11-28 13:07:09.010400923 +0900 dummy.log: {"message1":"/","prog":"bar","path":"/", groksuccess: "%{PATH:path}"}

[cjcc1]

I think what I had in mind was similar to your 'groksuccess' example. The difference is that the values would be user-determined and indicated at the level of the 'pattern' config in the grok block. I was calling this "parsetype_value" above - but there's likely a better name for it. For instance (and I don't understand what role 'prog' was playing for you ... but I left it):

<source>
  @type dummy
  @label @dummy
  dummy [
    { "message1": "no grok pattern matched!", "prog": "foo" },
    { "message1": "/", "prog": "bar" },
    { "message1": "/ /2", "prog": "bar" }
  ]
  tag dummy.log
</source>

<label @dummy>
  <filter>
    @type parser
    key_name message1
    reserve_data true
    reserve_time true
    <parse>
      @type grok
      grok_failure_key grokfailure
      grok_success_key groksuccess
      parsetype_key myeventtype  <--this is new
      <grok>
         pattern %{PATH:path}
         parsetype_value singlepath  <--this is new
      </grok>
      <grok>
        pattern %{PATH:path} %{PATH:path2}
        parsetype_value doublepath  <--this is new
      </grok>
    </parse>
  </filter>
  <match dummy.log>
    @type stdout
  </match>
</label>

and (leaving 'groksuccess' and 'grokfailure' in there - I'm not sure if you added those just for your example or if they are already part of the plugin) the result would look like:

2016-11-28 13:07:08.009131727 +0900 dummy.log: {"message1":"no grok pattern matched!","prog":"foo","message":"no grok pattern matched!","grokfailure":"No grok pattern matched","myeventtype": "default"}
2016-11-28 13:07:09.010400923 +0900 dummy.log: {"message1":"/","prog":"bar","path":"/", groksuccess: "%{PATH:path}","myeventtype": "singlepath"}
2016-11-28 13:07:09.010400923 +0900 dummy.log: {"message1":"/ /2","prog":"bar","path":"/", "path2":"/2", groksuccess: "%{PATH:path}","myeventtype": "doublepath"}

So basically inject a new field (here named myeventtype) which contains an indication of which grok block, if any, matched. The idea is that we could then later filter on that value to distinguish what type of event it is, or maybe just use it as a troubleshooting aid when trying to determine which pattern 'won'.

[cjcc1]

Now that I think about it some more, maybe it'd be more flexible to just have arbitrary extra fields included, akin to some of the the functionality promised by the filter_record_transformer.

In that case the grok part could look like this:

      <grok>
         pattern %{PATH:path}
         myeventtype singlepath <--this is a new user added field
      </grok>
      <grok>
        pattern %{PATH:path} %{PATH:path2}
        myeventtype doublepath <--this is a new user added field
        foo2 bar22<--this is a new user added field
      </grok>

and the output might be

2016-11-28 13:07:08.009131727 +0900 dummy.log: {"message1":"no grok pattern matched!","prog":"foo","message":"no grok pattern matched!","grokfailure":"No grok pattern matched"}
2016-11-28 13:07:09.010400923 +0900 dummy.log: {"message1":"/","prog":"bar","path":"/", groksuccess: "%{PATH:path}","myeventtype": "singlepath"}
2016-11-28 13:07:09.010400923 +0900 dummy.log: {"message1":"/ /2","prog":"bar","path":"/", "path2":"/2", groksuccess: "%{PATH:path}","myeventtype": "doublepath","foo2": "bar22"}

Not sure what would happen in the 'nomatch' case ... maybe there could be a similar list set a the parse level that gets applied to everything and may or may not be overridden in the event of a grok match.

[okkez]

Could you try #53?

[cjcc1]

Sorry ... I thought I had left a comment a few days ago saying the docs looked good.

I finally got around to trying it out and can say the added field does show up, so thank you.

From what I see it's working well. I thought there was an issue where the parse 'missed' unexpectedly, but it went away after I 'redeployed' so I must have messed it up the first time.

Will keep playing with it.

[cjcc1]

Actually, I think the example in your docs are off a bit:

grok_name_key grok_name
grok_failure_key grokfailure

should be under @type grok, not @type tail, right?

anyhow, trying the deployed instance out now. 2.2.0 was released yesterday, right?

[okkez]

Thanks.
Fixed d31f18e