Logstash 1.4.2 grok filter with multiple match statements not working as expected

[YuvalBenAri]

I tried to use a grok filter with multiple match statements to match various possible formats of a certain log file. The synatax I used was:

grok {
        break_on_match => false
        match => [ "message", "regex1" ]
        match => [ "message", "regex2" ]
        match => [ "message", "regex3" ]

}

However only the first match is working. If I change the order so regex2 is the first one only it works

The workaround I found is:

grok {
        match => [ "message", "regex1" ]
        tag_on_failure => []
}
grok {
        match => [ "message", "regex2" ]
        tag_on_failure => []
}
grok {
        match => [ "message", "regex3" ]
        tag_on_failure => []
}

I have seen other complaints about the same issue but no confirmation that it's still a known issue on 1.4.2

[YuvalBenAri]

I did, as mentioned in my code example, with no help

On Thu, Nov 20, 2014 at 4:19 PM, Wiibaa notifications@github.com wrote:

@YuvalBenAri https://github.com/YuvalBenAri did you try with break_on_match
=> false ??

—
Reply to this email directly or view it on GitHub
#2108 (comment)
.

[YuvalBenAri]

I did, as mentioned in my code example, with no help

On Thu, Nov 20, 2014 at 4:19 PM, Wiibaa notifications@github.com wrote:

@YuvalBenAri https://github.com/YuvalBenAri did you try with break_on_match
=> false ??

—
Reply to this email directly or view it on GitHub
#2108 (comment)
.

[wiibaa]

@YuvalBenAri sorry, I read too fast, it seems a long standing issue https://logstash.jira.com/browse/LOGSTASH-703

[wiibaa]

@YuvalBenAri sorry, I read too fast, it seems a long standing issue https://logstash.jira.com/browse/LOGSTASH-703

[YuvalBenAri]

Thanks. Any idea when is it fixed? I just spent few days banging my head with this :(

[YuvalBenAri]

Thanks. Any idea when is it fixed? I just spent few days banging my head with this :(

[jordansissel]

@YuvalBenAri Pretty sure this bug was fixed recently and will be available in the next release (1.5.0) of logstash.

[jordansissel]

@YuvalBenAri Pretty sure this bug was fixed recently and will be available in the next release (1.5.0) of logstash.

[YuvalBenAri]

Thanks

[YuvalBenAri]

Thanks

[wiibaa]

@YuvalBenAri I confirm that this works in current master but the change in grok are important so it would be difficult (at least for me) to do a hack-fix on 1.4, hoping you can work with your workaround until next release.

On current master with this config

input {
  stdin{}
}
filter {
  grok {
    break_on_match => false
    match => [ "message", "%{WORD:word1}" ]
    match => [ "message", "%{WORD:word2}" ]
    match => [ "message", "%{WORD:word3}" ]
  }
}
output {
  stdout { codec => rubydebug }
}

I get
{
"message" => "hello\r",
"@Version" => "1",
"@timestamp" => "2014-11-20T19:07:51.629Z",
"host" => "LU5CB147157W",
"word1" => "hello",
"word2" => "hello",
"word3" => "hello"
}

[wiibaa]

@YuvalBenAri I confirm that this works in current master but the change in grok are important so it would be difficult (at least for me) to do a hack-fix on 1.4, hoping you can work with your workaround until next release.

On current master with this config

input {
  stdin{}
}
filter {
  grok {
    break_on_match => false
    match => [ "message", "%{WORD:word1}" ]
    match => [ "message", "%{WORD:word2}" ]
    match => [ "message", "%{WORD:word3}" ]
  }
}
output {
  stdout { codec => rubydebug }
}

I get
{
"message" => "hello\r",
"@Version" => "1",
"@timestamp" => "2014-11-20T19:07:51.629Z",
"host" => "LU5CB147157W",
"word1" => "hello",
"word2" => "hello",
"word3" => "hello"
}

[suyograo]

@YuvalBenAri this was fixed in #1558

[suyograo]

@YuvalBenAri this was fixed in #1558

[tabs11]

Hi, It's possible to have two different logfiles (in my case logs with different number of columns) and create different matchs to each one inside the same grok?

How can I assume the match to respective file or to respective path where is stored?

[tabs11]

Hi, It's possible to have two different logfiles (in my case logs with different number of columns) and create different matchs to each one inside the same grok?

How can I assume the match to respective file or to respective path where is stored?

[untergeek]

@tabs11 Please ask usage questions in our discussion forums at https://discuss.elastic.co.

[untergeek]

@tabs11 Please ask usage questions in our discussion forums at https://discuss.elastic.co.