GELF Decoder

JSON is a convenient format to store application-specific key/value pairs. Furthermore, values are typed, making it an excellent fit for efficient indexing in a data store such as ArangoDB, Hyperdex, MongoDB or ElasticSearch.

The flipside of JSON's flexibility is that it doesn't encourage applications to follow any conventions and log even common properties in a consistent way.

Graylog's GELF specification partially solves this, by adding minimal constraints to JSON-based log entries:

• A value MUST contain an UTF-8 string, a boolean (true/false), a JavaScript number or null.
• A message MUST include a timestamp, whose key is timestamp, and whose value is a Unix timestamp.
• A message MUST include a source host name, in a string value whose key is host.
• A message MAY include a GELF version: "version":"1.1".
• A message MUST include a short description, in a string value whose key is short_message.
• A message MAY include a long description, in a string value whose key is full_message.
• A message MAY include a severity level, as a number (from 0 to 7, matching syslog-defined levels) whose key is level
• A message MAY include any number of additional key/value pairs provided that the key starts with an underscore (_), so that application-provided properties cannot collide with reserved properties.

Example of valid GELF message:

{
  "version":"1.1",
  "timestamp": 1385053862.3072,
  "host": "example.org",
  "short_message": "A short message that helps you identify what is going on",
  "full_message": "Backtrace here\n\nmore stuff",
  "level": 1,
  "_user_id": 9001,
  "_some_info": "foo",
  "_some_env_var": "bar"
}

Note: the example above is a raw JSON string, as produced by standard JSON libraries. In the JSON representation, a \n sequence is not equivalent to a single character. GELF records are always represented as valid JSON strings, and have to respect the same escaping rules.

The JSON (GELF) decoder can be selected in the input section of Flowgger's configuration file:

[input]
type = "gelf"

Notes on log4perl_gelf

Log::Log4perl::Layout::GELF is a module to add GELF support to Log4Perl, a logging framework for a language called Perl.

Unfortunately, this module:

• Doesn't implement the GELF specification. Lines numbers, timestamps and severity levels are sent as UTF-8 strings where the specification mentions that they MUST be numbers.
• Can send UTF-8 strings that cannot be parsed as UTF-8 strings.
• Can send invalid, unparsable JSON.
• Can only send messages over UDP. TCP support is documented but cannot possibly work with any GELF parsers: messages are concatenated without any delimiter, and the output of the concatenation gets compressed as a single chunk, which is unworkable on a persistent TCP connection.
• Hasn't been updated since 2011.

There are no compatibility hacks that Flowgger or any other GELF (or even JSON) parser could implement in order to reliably support the output of this module when used with TCP.

Log::Log4perl::Layout::LTSV is a recommended alternative.