Examples

Processing syslog

input {
     loghub.receivers.Udp { port: 514, decoder: loghub.decoders.StringCodec}
} | $syslog

pipeline[syslog] {
      [type] = "syslog"
    | loghub.processors.Grok {
        pattern: "%{SYSLOG_LINE}",
        field: "message",
    }
    | loghub.processors.SyslogPriority {
        field: "syslog_pri",
    }
    | loghub.processors.DateParser {
        field: "timestamp",
        destination: "@timestamp",
        timezone: "CET",
        success: [timestamp]-
    }
    | loghub.processors.DateParser {
        fields: ["nexustimestamp", "utctimestamp"],
        patterns: ["MMM dd HH:mm:ss", "MMM dd yyyy HH:mm:ss"],
        destination: "@timestamp",
        timezone: "UTC",
        success: ( [nexustimestamp]- | [utctimestamp]- )
    }
    | loghub.processors.DateParser {
        field: "timestamp8601",
        patterns: ["ISO_INSTANT", "yyyy-MM-dd'T'HH:m:ss.SSSSSSXXX"],
        destination: "@timestamp",
        success: [timestamp8601]-
    }
    | (java.lang.Integer) [syslog5424_ver]
    # a set of subpipeline that can process further message according to the program
    | [program] == "centreon-engine" ? $nagios
    | [program] == "proftpd" ? $proftpd
    | [program] == "sendmail" ? $sendmail
    | [program] == "imap" || [program] == "imaps" ? $imap
} | $main

Processing Apache logs

This pattern uses a binary logger for apache, mod_log_net than encode request informations in a msgpack’ed UDP message.

It can be configured with

<IfModule !log_net_module>
  LoadModule log_net_module   /usr/lib64/httpd/modules/mod_log_net.so
</IfModule>

#SetEnvIf needed, setenv is evaluated too late
SetEnvIf Request_URI . instance=webfront

LognetHost loghost
LognetPort 1516
LognetEntries virtual_host remote_address request_duration_microseconds bytes_sent request_query request_method
LognetEntry header_in Host name=host_header
LognetEntry header_in User-agent name=user-agent
LognetEntry header_in Referer name=referrer
LognetEntry hostname name=servername
LognetEntry env instance name=instance
LognetEntry status request=final
LognetEntry request_uri request=original

And to use log messages, the loghub configuration is:

geoip2data: "/var/lib/loghub/GeoIP2-Country.mmdb"

input {
    loghub.receivers.Udp {
        port: 1516, 
        decoder: loghub.decoders.Msgpack,
    }
} | $http

pipeline[http] {
    [type] = "http"
    | loghub.processors.Geoip2 {
        types: ["country","city", "location"],
        field: "remote_address",
        destination: "location",
    }
    | loghub.processors.VarExtractor { 
        parser: "(?<name>\\p{Alnum}+)=(?<value>[^&]+)&?",
        path: "query",
        field: ".request_query",
    }
    | loghub.processors.DecodeUrl { 
        path: "query",
        fields: [ "*" ],
    }
    | loghub.processors.UserAgent { 
        field: "user-agent",
        destination: "user-agent-content",
    }
} | $main

It will ouput events likes:

{
    "date": "2016-08-12 21:16:09",
    "instance": "webfront",
    "query": {
      "q1": "someargs",
      "q2": "another"
    },
    "remote_address": "8.8.8.8",
    "request_method": "GET",
    "request_duration_microseconds": 2068691,
    "bytes_sent": 11884,
    "request_uri": "/process/",
    "host_header": "www.loghub.com",
    "referrer": "https://github.com/fbacchella/LogHub",
    "@timestamp": "2016-08-12T21:16:09.677+0200",
    "host": "sourcehost",
    "servername": "sourcehost",
    "location": {
      "continent": "Europe",
      "country": {
        "name": "Greece",
        "code": "GR"
      }
    },
    "virtual_host": "www.loghub.com",
    "user-agent": "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0"
    "user-agent-content": {
      "os": {
        "family": "Windows 8.1"
      },
      "device": "Other",
      "userAgent": {
        "family": "Firefox",
        "major": "39",
        "minor": "0"
      }
    },
    "status": 200
  },

From the request

[2016-08-12 21:16:09] 8.8.8.8 "www.loghub.com" "GET /process/?q1=someargs&q2=another HTTP/1.1" 2068691 200 - 11884 "https://github.com/fbacchella/LogHub" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0"

Watching isc-dhcp

pipeline[syslog] {
    ...
    | [program] == "dhcpd" ? $dhcpd
}

pipeline [dhcpd] {
    loghub.processors.Grok {
        pattern: "(?<verb>DHCPACK|DHCPREQUEST|DHCPOFFER) (?:for|on) %{IP:ip} (?:from|to) %{MAC:src_mac}(?: \\(%{HOSTNAME:dhcp_client_name}\\))?(?:(?: via %{IP:dhcp_proxy})|(?: via %{WORD:dhcp_interface}))",
        success: [message]-,
    }
    | loghub.processors.Grok {
        pattern: "DHCPDISCOVER from %{MAC:src_mac}(?:(?: via %{IP:dhcp_proxy})|(?: via %{WORD:dhcp_interface}))",
        success: {[message]- | [verb]="DHCPDISCOVER"},
    }
    | loghub.processors.Grok {
        pattern: "PXE boot: 1:%{PXEMAC:src_mac}",
        success: {[message]- | [verb]="PXE boot"},
        customPatterns: {
            "PXEMAC": "(?:[a-f0-9]{1,2}:){5}(?:[a-f0-9]{1,2})",
        },
    }
}

For example, given the following syslog message:

DHCPREQUEST for 169.254.1.1 from 92:18:02:71:49:f8 (hostname) via bond0

It will add to the event the following informations:

{
    "dhcp_client_name":"hostname",
    "dhcp_interface":"bond0",
    "ip":"169.254.1.1",
    "src_mac":"92:18:02:71:49:f8",
    "verb":"DHCPREQUEST"
}

Watching sudo

pipeline[syslog] {
    ...
    | [program] == "sudo" ? $sudo
}

pipeline[sudo] {
    loghub.processors.Grok {
        pattern: "%{WORD:user} : %{GREEDYDATA:message}",
        if: [message] ==~ /.*TTY=.*/,
    }
    | loghub.processors.VarExtractor { 
        parser: "(?<name>\\p{Alnum}+)=(?<value>.+?)(?:(?: ; )|$)",
        path: "sudo",
        field: ".message",
        if: [user] != null
    }
}

For example, given the following syslog message:

nrpe : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/reboot

It will add to the event the following informations:

{
    "sudo":{
        "COMMAND":"/sbin/reboot",
        "PWD":"/",
        "TTY":"unknown",
        "USER":"root"
    },
    "user":"nrpe"
}