Grok assigns the wrong names to captures under some conditions

[jordansissel]

(This issue was originally filed by @TheFlimFlam at elastic/logstash#2072)

---

Description:
Composing grok patterns that share named captures will result in names to bind to the wrong capture in the context of the composition.

Reproduction steps:
Setup the following patterns file (/etc/logstash/patterns/general/test-patterns)

SSH_KEYFILE_ERROR (?<tags>error): (?<failure>Could not load host key): %{PATH:keyfile}
SSH_PASSWORD_FAIL (?<failure>Failed password) for %{USER:username} from %{IPORHOST:clientip} port %{INT:port} %{WORD:protocal}
AUTH_SSH          (%{SSH_KEYFILE_ERROR}|%{SSH_PASSWORD_FAIL})

Run the patterns file using the following logstash config

input {
    generator {
        count   => 1
        message => "Nov 14 14:50:23 puppet sshd[36930]: Failed password for magicaluser from 127.0.0.1 port 43333 ssh2"
    }
}

output {
    stdout { codec => "rubydebug" }
}

filter {
    grok {
        patterns_dir => '/etc/logstash/patterns/general/test-patterns'
        match => [ "message", "%{AUTH_SSH}" ]
    }
}

Will print the following to standard out:

{
       "message" => "Nov 14 14:50:23 puppet sshd[36930]: Failed password for magicaluser from 127.0.0.1 port 43333 ssh2",
      "@version" => "1",
    "@timestamp" => "2014-11-14T03:16:26.831Z",
          "host" => "1051a1523d6e",
      "sequence" => 0,
      "username" => "Failed password",
      "clientip" => "magicaluser",
      "protocal" => "43333"
}

Expected output:

• A field should exist called failure which captures the text 'Failed password'
• The field username should contain the text 'magicaluser'

[jordansissel]

I agree with you on the expected output. This behavior seems like a new bug, which may be expected given we did some internal changes to the grok library to improve performance and may have broken something.

I'd like to get some tests that show this bad behavior so we can ensure it stays fixed in the future.

[jordansissel]

I haven't yet confirmed this. Not sure if anyone else has reproduced it either.