Grok matching syslog

[lvthao]

I used Rsyslog send to logstash. I write grok like match => { "message" => " %{GREEDYDATA:OS_message}" }.That i get OS_message : controller 2014-10-06 12:27:47.536 1142 WARNING keystone.common.controller [-] RBAC: Bypassing authorization
But when i change match => { "message" => " %{HOSTNAME:hostname} %{GREEDYDATA:OS_message}" }. I get hostname is 00.So,I try to test with grokdebug, i get hostname controller. Anybody can help me explain it.

[magnusbaeck]

You're mixing your bug description with configuration snippets and Logstash output. Could you be a little bit more verbose and use full examples that include the complete input and output?

[lvthao]

Log line like as : <132>2014-10-06T10:15:25.285613+07:00 controller 2014-10-06 10:15:25.285 2388 WARNING nova.compute.api [req-3d87ad12-a13e-4641-8fd2-158cab28850a e3968c446a1a4c6cac12d0516259d714 a3f6dbd085a14fd6afc0c612db2a6960] instances quota exceeded for a3f6dbd085a14fd6afc0c612db2a6960, tried to run 1 instances. Cannot run any more instances of this type
Here is config:
input {
tcp {
port => 9000
type => syslog
}
}
filter {
if [type] == "syslog" {

     grok {
            patterns_dir=> "/opt/logstash/patterns"
            match => [ "message" , " %{HOSTNAME:hostname} %{GREEDYDATA:OS_message}" ]
            add_field=> { "received_at" => "%{@timestamp}" }
            remove_tag => [ "_grokparsefailure" ]
    }

    date {
            match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
            remove_field=> "timestamp"
            timezone=> "Etc/UTC"
    }
    syslog_pri { }
    }

}

[magnusbaeck]

Sorry, I don't understand. You have a message that begins with an ISO8601 timestamp and you're attempting to use a grok expression that begins with " %{HOSTNAME:hostname}"? I just don't see how this is supposed to work.

Perhaps this would be a better fit for the Logstash mailing list.

[lvthao]

I'm sorry about my mistake. Here is grok:
match => [ "message" , "( ?<sys_pri><%{POSINT}>)%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostname} %{TIMESTAMP_ISO8601:OS_timestamp} %{POSINT:OS_pid} %{LOGLEVEL:OS_loglevel} %{SYSLOGPROG:OS_program} (?<OS_id>[(req-%{UUID}|%{BASE16NUM}|-|%{SPACE})*])? %{GREEDYDATA:OS_message}" ]

[magnusbaeck]

I'm continuing the discussion in the mailing list thread (https://groups.google.com/d/topic/logstash-users/Ikz2xai-6Uc/discussion). So far nothing indicates that there's a bug in Logstash.

[jordansissel]

For Logstash 1.5.0, we've moved all plugins to individual repositories, so I have moved this issue to logstash-plugins/logstash-filter-grok#25. Let's continue the discussion there! :)