Processors

Processors

Processors are the main worker of loghub. They takes events and transform them.

Many of them act on a given fields. For those kind of field processors, there is a few shortcuts that allow easier management.

The first use case is simply the declaration:

| loghub.processors.Someone {
    field: "message",
    ...
}

But also, many fields can be given using an array. It this case, glob are used. For more details, see Pattern Matching Notation.

For example:

| loghub.processors.Someone {
    fields: ["message", "sub*"],
    ...
}

This transformer will work on the field message, subMessage, but not other.

Events can be structured and contains map of key. It's not always convenient to repeat or define the full path. So a processor can use a path prefix. All the other fields will be relatives to this path and path component are separated using a dot. If a field name starts with a ., it refers to the root.

For example:

| loghub.processors.Someone {
    path: "query",
    fields: [".message", "param"],
    ...
}

This processor will works on the fields message and query.param.

The result will be stored in the same field by default. It someone want a copy instead it can use the bean destination. A pattern can be used that will be have access to the field variable.

Special bean.

Some special processor parameters (called bean) are used, they are common to all processor.

The bean if can be used to control execution of a processor. It's a groovy expression. If it return false, this processor step will not be executed.

Processors list

loghub.processors.Convert

A processor that take a String field and transform it to any object that can take a String as a constructor.

It's a field processor.

beans:

• className

loghub.processors.DecodeUrl

Used to decode string encoded for URL It's a field processor.

beans:

• encoding, the encoding used for the decoding, default to UTF-8.
• loop, loop until no more decoded is done, default to false.

loghub.processors.Geoip2

It use GeoIP2 API and database (.mmdb). The path to the database is given in the global geoip2data property, so all the possible instances will use the same.

It's a field processor.

beans:

• types, an array of value to extract from the database, can take the values country, registredcountry, representedcountry, city, location, continent, postal and subdivision.
• locale, GeoIP2 databases contains many locale names, this indicate which one to use, default to en.

loghub.processors.Grok

Uses Grok to parse a field. All pattern are found inside the patterns folder of the class path.

It's a field processor.

beans:

• grok: grok pattern to use.

loghub.processors.NameResolver

Used to resolve names from a field. It uses JNDI for that. Results are cached using ehcache.

It's a field processor.

beans:

• ttl, the TTL for queries in the ehcache,
• cacheSize, the ehcache size,
• resolver, the IP of a resolver, if not given, it uses systems one,

loghub.processors.DateParser

Used to parse a date in a string and store the result in a java.lang.Date object. It uses java.time.format.DateTimeFormatter for parsing. If the destination field @timestamp is used, the event's timestamp will be changed.

It's a field processor.

beans:

• timeZone, ensure the time zone of parsed value, if it can't be deduced from the string.
• locale, use this locale for parsing, default to en.
• patterns, an array of patterns to try until one matches. If not given, default to the named patterns in java.time.format.DateTimeFormatter.
• pattern, a single pattern to use for parsing.

loghub.processors.ParseJson

Uses com.fasterxml.jackson for parsing a json string and encode the result in the field.

It's a field processor.

loghub.processors.SyslogPriority

It's used to resolve a integer as a syslog priority field.

It's a field processor.

beans:

• resolve, does facility and severity numbers are resolver to string, default to true.
• facilities, an ordered array of string to uses for facilites values, default to [ "kernel", "user-level" "mail", "daemon", "security/authorization", "syslogd", "line printer", "network news", "uucp", "clock", "security/authorization", "ftp", "ntp", "log audit", "log alert", "clock", "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7" ]
• severities, an ordered array of string to uses for severities values, default to [ "emergency", "alert", "critical", "error", "warning", "notice", "informational", "debug"]

loghub.processors.UserAgent

Parse a user agent field, using the (ua-parser)[https://github.com/tobie/ua-parser] library and yaml files extracted fro it. It look for ua_parser/regexes.yaml in the class path, so a alternate file can be given. Results are cached using a ehcache.

It's a field processor.

beans:

• cacheSize, the ehcache size to use.

loghub.processors.VarExtractor

It iterate other a field using a pattern to extract a set of name/value. The pattern is a common java regex that must define two named capturing group name and value. After the parsing, the content not matched is put back in the original field. If nothing left, it's destroyed.

It's a field processor.

beans:

• parser, the parser to use to extract values, it defaults to (?<name>\\p{Alnum}+)\\p{Space}?[=:]\\p{Space}?(?<value>[^;,:]+)[;,:]?that should match common case.