Repository navigation
Processors
Processors are the main worker of loghub. They takes events and transform them.
They are defined using the syntax:
ClassName {
attribute: value;
...
}
Many of them needs operate on a specific field. For those kind of field processors, there is a few shortcuts that allow easier management.
The first use case is simply the declaration:
| loghub.processors.Someone {
field: "message",
...
}
But also, many fields can be given using an array. It this case, glob are used. For more details, see Pattern Matching Notation. For each field repeated, a instance of the processor is created.
For example:
| loghub.processors.Someone {
fields: ["message", "sub*"],
...
}
This transformer will work on the field message, subMessage, but not other.
Events can be structured and contains map of key. It's not always convenient to repeat or define the full path. So a processor can use a path prefix. All the other fields will be relatives to this path and path component are separated using a dot. If a field name starts with a ., it refers to the root.
For example:
| loghub.processors.Someone {
path: "query",
fields: [".message", "param"],
...
}
This processor will works on the fields message and query.param.
The result will be stored in the same field by default. It someone want a copy instead it can use the bean destination. A pattern can be used that will be have access to the field variable.
The processing status can be managed using 4 attributes.
The attribute if can be used to control execution of a processor. It's a groovy expression. If it return false, this processor will not be executed.
The attribute success refers to pipe element that will be executed if the processor succeeded.
The attribute failure refers to pipe element that will be executed if the processor failed by returning a false value.
The attribute exception refers to pipe element that will be executed if the processor failed throwing a ProcessorException.
Unmanaged exceptions can't be handled, they are stored in the JMX bean loghub:type=stats/Exceptions and the event is dropped.
A processor that take a String field and transform it to any object that can take a String as a constructor.
It's a field processor.
attributes:
- className
Used to decode string encoded for URL It's a field processor.
attributes:
- encoding, the encoding used for the decoding, default to "UTF-8".
- loop, loop until no more decoded is done, default to false.
It uses GeoIP2 API and database (.mmdb). The path to the database is given in the global geoip2data property, so all the possible instances will use the same.
It's a field processor.
attributes:
- types, an array of value to extract from the database, can take the values country, registredcountry, representedcountry, city, location, continent, postal and subdivision.
- locale, GeoIP2 databases contains many locale names, this indicate which one to use, default to "en".
Uses Grok to parse a field. All pattern are found inside the patterns folder of the class path. So it can be stored under the plugins property of the configuration file.
If in a pattern, the destination name is ., it will be put back in the original field. It allows to write a
expression like
| loghub.processors.Grok {
pattern: "%{HOSTNAME:.}\\.mydomain\\.com",
fields: ["remote_address", "host", "local_address"],
}
This will remove ".mydomain.com" from each fields remote_address, host, local_address only if it contains it and keep the hostname part.
Patterns can be tested in command line. If run with the argument -g or --grok, it specifies the pattern folder. It then read stdin. The first line is the pattern to check (the pattern bean) and then each following lines is checked against it. If the parsing failed, the line is printed back prefixed with ** failing ** . If succeeded, the found properties are printed.
It's a field processor.
attributes:
- grok: grok pattern to use.
Used to resolve names from a field. It uses JNDI for that. Results are cached using ehcache.
It's a field processor.
attributes:
- ttl, the TTL for queries in the ehcache,
- cacheSize, the ehcache size,
- resolver, the IP of a resolver, if not given, it uses systems one,
- timeout, the time out of queries in seconds.
Used to resolve names from a field. It uses Netty asynchronous resolver. It allows waiting requests to don't hold a processor thread.
It's a field processor.
attributes:
- resolver, the IP of a resolver, if not given, it uses the system's one,
- timeout, the time out of queries in seconds.
- cacheSize, the entry cache size.
- poller, the Netty poller to use.
- queueDepth, the number of simultaneous request allowed; -1 default to the
queueDepthproperty, 0 means no restrictions.
Used to parse a date in a string and store the result in a java.lang.Date object. It uses java.time.format.DateTimeFormatter for parsing. If the destination field @timestamp is used, the event's timestamp will be changed.
It's a field processor.
attributes:
- timeZone, ensure the time zone of parsed value, if it can't be deduced from the string.
- locale, use this locale for parsing, default to en.
- patterns, an array of patterns to try until one matches. If not given, default to the named patterns in
java.time.format.DateTimeFormatter. - pattern, a single pattern to use for parsing.
CEF (Common Event Format) is a syslog-like message format defined by ArcSight, current version is 25
At this message is quite strange, parsing it using a regex is not easy, and a dedicated parser is provided. This parse only resolve the message part. So from a message being:
Sep 19 08:26:10 host CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232
It's used to resolve the message CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232, so any transport or source format can be used, it just needs to be cleaned. The standar header will be resolved in the field cef_fields, and the extensions in cef_extensions`. This parser don't try to be smart and resolve values type, it must be explicitely handled.
A standard usage, that implements Elastic Common Schema could be:
pipeline[cef] {
loghub.processors.Grok {
pattern: "%{TIMESTAMP:cef_timestamp} %{WORD:cef_host} %{CEFDATA:cef_message}",
customPatterns: {
"TIMESTAMP": "%{MONTH} %{MONTHDAY} (%{YEAR} )?%{TIME}",
"CEFDATA": "CEF:%{GREEDYDATA}",
},
success: (
loghub.processors.ParseCef {
field: "cef_message",
success: (
[cef_message]- |
[message]- |
[cef_fields device_vendor] == "vendor1" ? path[cef_extensions] ($vendor1) |
[cef_fields device_vendor] == "vendor2" ? path[cef_extensions] ($vendor2) |
loghub.processors.DateParser {
field: [cef_timestamp],
patterns: ["MMM dd YYYY HH:mm:ss"],
} |
[cef timestamp] < [cef_timestamp] |
path[cef_fields](
[. cef version] < [version] |
[. event code] < [device_event_class_id] |
[. observer vendor] < [device_vendor] |
[. observer product] < [device_product] |
[. observer version] < [device_version] |
[. message] < [name] |
[. event severity] < [severity]
) |
path[cef_extensions](
[. message] < [msg] |
[. source interface alias] < [deviceInboundInterface] |
[. destination interface alias] < [deviceOutboundInterface] |
[. destination mac] < [smac] |
[. source mac] < [dmac] |
[. destination bytes] < [out] |
[. source bytes] < [in] |
(java.lang.Integer) [spt] | [. source port] < [spt] |
(java.lang.Integer) [dpt] | [. destination port] < [dpt]
) |
[cef extensions] < [cef_extensions] |
[observer hostname] < [cef_host]
)
}
),
}
}
Uses com.fasterxml.jackson for parsing a json string and encode the result in the field.
It's a field processor.
This processor parse a string or a number as a bits field. It can manage single bit (flag) bits field or split it as sub value with variable length sub fields..
It's a field processor.
attributes:
- bitsNames, an array of string, that give the name of individual bits. It's mandatory.
- fieldsLength, an array of integer ; if given, the input value is splited as subfield of the given length.
- asMap, a boolean ; a flags bit field can either be returned as a array of set bits or a map of all flags with there value.
For example, for the following configuration:
loghub.processors.PrintBinary {
bitsNames: ["PF_PROT", "PF_WRITE", "PF_USER", "PF_RSVD", "PF_INSTR"],
field: "error_code",
destination: "errors"
}
If given an event that contains the field "field": 13, it will put ["PF_PROT", "PF_USER", "PF_RSVD"] in errors because 13 resolve to 0x1101. Each bit is matched to the associated entry in the array, the first bit maps to first entry in the array and so one. Only bits set to 1 will keep the name.
If the attribute asMap is set to true, the processor will return a map of {"PF_PROT": 1, "PF_WRITE": 0, "PF_USER": 1, "PF_RSVD": 1, "PF_INSTR": 0}
It's used to resolve a integer as a syslog priority field.
It's a field processor.
attributes:
- resolve, does facility and severity numbers are resolver to string, default to true.
- facilities, an ordered array of string to uses for facilites values, default to [ "kernel", "user-level" "mail", "daemon", "security/authorization", "syslogd", "line printer", "network news", "uucp", "clock", "security/authorization", "ftp", "ntp", "log audit", "log alert", "clock", "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7" ]
- severities, an ordered array of string to uses for severities values, default to [ "emergency", "alert", "critical", "error", "warning", "notice", "informational", "debug"]
Parse a user agent field, using the ua-parser/uap-java library and yaml files extracted from ua-parser/uap-core. It look for the file ua_parser/regexes.yaml in the class path, so a alternate file can be given. Results are cached using a ehcache.
It's a field processor.
attributes:
- cacheSize, the ehcache size to use.
It iterate other a field using a pattern to extract a set of name/value. The pattern is a common java regex that must define two named capturing group name and value. After the parsing, the content not matched is put back in the original field. If nothing left, that field is destroyed.
It's a field processor.
attributes:
- parser, the parser to use to extract values, it defaults to
(?<name>\\p{Alnum}+)\\p{Space}?[=:]\\p{Space}?(?<value>[^;,:]+)[;,:]?that should match common case.