Examples

Processing syslog

input {
     loghub.receivers.Udp { port: 514, decoder: loghub.decoders.StringCodec}
} | $syslog

pipeline[syslog] {
      [type] = "syslog"
    | loghub.processors.Grok {
        pattern: "%{SYSLOG_LINE}",
        field: "message",
    }
    | loghub.processors.SyslogPriority {
        field: "syslog_pri",
    }
    | loghub.processors.DateParser {
        field: "timestamp",
        destination: "@timestamp",
        timezone: "CET",
        success: [timestamp]-
    }
    | loghub.processors.DateParser {
        fields: ["nexustimestamp", "utctimestamp"],
        patterns: ["MMM dd HH:mm:ss", "MMM dd yyyy HH:mm:ss"],
        destination: "@timestamp",
        timezone: "UTC",
        success: ( [nexustimestamp]- | [utctimestamp]- )
    }
    | loghub.processors.DateParser {
        field: "timestamp8601",
        patterns: ["ISO_INSTANT", "yyyy-MM-dd'T'HH:m:ss.SSSSSSXXX"],
        destination: "@timestamp",
        success: [timestamp8601]-
    }
    | (java.lang.Integer) [syslog5424_ver]
    # a set of subpipeline that can process further message according to the program
    | [program] == "centreon-engine" ? $nagios
    | [program] == "proftpd" ? $proftpd
    | [program] == "sendmail" ? $sendmail
    | [program] == "imap" || [program] == "imaps" ? $imap
} | $main

Processing Apache logs

This pattern uses a binary logger for apache, mod_log_net than encode request informations in a msgpack’ed UDP message.

It can be configured with

<IfModule !log_net_module>
  LoadModule log_net_module   /usr/lib64/httpd/modules/mod_log_net.so
</IfModule>

#SetEnvIf needed, setenv is evaluated too late
SetEnvIf Request_URI . instance=webfront

LognetHost loghost
LognetPort 1516
LognetEntries virtual_host remote_address request_duration_microseconds bytes_sent request_query request_method
LognetEntry header_in Host name=host_header
LognetEntry header_in User-agent name=user-agent
LognetEntry header_in Referer name=referrer
LognetEntry hostname name=servername
LognetEntry env instance name=instance
LognetEntry status request=final
LognetEntry request_uri request=original

And to use log messages, the loghub configuration is:

geoip2data: "/var/lib/loghub/GeoIP2-Country.mmdb"

input {
    loghub.receivers.Udp {
        port: 1516, 
        decoder: loghub.decoders.Msgpack,
    }
} | $http

pipeline[http] {
    [type] = "http"
    | loghub.processors.Geoip2 {
        types: ["country","city", "location"],
        field: "remote_address",
        destination: "location",
    }
    | loghub.processors.VarExtractor { 
        parser: "(?<name>\\p{Alnum}+)=(?<value>[^&]+)&?",
        path: "query",
        field: ".request_query",
    }
    | loghub.processors.DecodeUrl { 
        path: "query",
        fields: [ "*" ],
    }
    | loghub.processors.UserAgent { 
        field: "user-agent",
        destination: "user-agent-content",
    }
} | $main

It will ouput events likes:

{
    "date": "2016-08-12 21:16:09",
    "instance": "webfront",
    "query": {
      "q1": "someargs",
      "q2": "another"
    },
    "remote_address": "8.8.8.8",
    "request_method": "GET",
    "request_duration_microseconds": 2068691,
    "bytes_sent": 11884,
    "request_uri": "/process/",
    "host_header": "www.loghub.com",
    "referrer": "https://github.com/fbacchella/LogHub",
    "@timestamp": "2016-08-12T21:16:09.677+0200",
    "host": "sourcehost",
    "servername": "sourcehost",
    "location": {
      "continent": "Europe",
      "country": {
        "name": "Greece",
        "code": "GR"
      }
    },
    "virtual_host": "www.loghub.com",
    "user-agent": "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0"
    "user-agent-content": {
      "os": {
        "family": "Windows 8.1"
      },
      "device": "Other",
      "userAgent": {
        "family": "Firefox",
        "major": "39",
        "minor": "0"
      }
    },
    "status": 200
  },

From the request

[2016-08-12 21:16:09] 8.8.8.8 "www.loghub.com" "GET /process/?q1=someargs&q2=another HTTP/1.1" 2068691 200 - 11884 "https://github.com/fbacchella/LogHub" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0"