Skip to content

makeros/gelf-transformer

Repository files navigation

GELF Transformer (ver. 2)

This project is based on Pino Gelf

GELF transformer is a tool which receives json formatted logs from the stdin and transforms them into GELF format GELF It can also use a custom mapping schema to fill the output log with more data.

CircleCI NPM NPM download

Contents

Installation

npm i -g gelf-transformer

Usage

Pipeline approach

If your application is pushing logs to the standard output then pipe them to gelf transformer.

node your-app.js | gelf-transformer log <options>

Getting Started

command log

gelf-transformer log --help
Switch Description Default Notes
-h Host 127.0.0.1 Graylog server host
-p Port 12201 Graylog server port
-m Maximum Chunk Size 1420
-c Custom schema false You can provide a schema which will define which information from your original logs will be visible in the graylog formatted log
-v Verbose mode false Output GELF to console
-t Start sending logs to Graylog false It will start to send logs to the defined graylog server

Examples

Custom Fields

Given the log message (formatted as JSON for readability):

{
  "pid":16699,
  "hostname":"han",
  "name":"gelf-test-app",
  "level":30,
  "time":1481840140708,
  "msg":"request completed",
  "customField":"test",
  "res":{"statusCode":304},
  "responseTime":8,
  "req":{
    "method":"GET",
    "headers":{
      "host":"localhost:3000",
      "user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14"}
    },
  "v":1
}

Given custom schema json file (my_custom_schema.json):

{
  "title": "GELF Schema",
  "type": "object",
  "properties": {
    "_status_code": {
      "type": "integer",
      "source": "res.statusCode"
    },
    "_user_agent": {
      "type": "string",
      "source": "req.headers.user-agent"
    },
    "customField": {
      "type": "string"
    }
  }
}

And the usage:

node server.js | gelf-transformer log -v -c my_custom_schema.json

Gelf Transformer will show the following message to your Graylog server (formatted here as JSON for readability):

{
  "version":"1.1",
  "host":"han",
  "short_message":"request completed",
  "full_message":"request completed",
  "timestamp":1481840140.708,
  "level":6,
  "facility":"gelf-test-app",
  "_status_code":304,
  "_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14",
  "customField":"test"
}

GELF

Right now automatic mapping of fields is done as follows:

Output GELF Input log Notes
version - Hardcoded to 1.1 per GELF docs
host hostname
short_message msg This message is truncated to 64 characters
full_message msg msg is not truncated
timestamp time
level level Default level codes from Pino are mapped to SysLog levels1
facility name deprecated

Log Level Mapping

Default behaviour

By default Gelf Transfomer will log level from a Pino format to syslog format:

Pino Log Level Value Pino Log Level Name SysLog Level
10 Trace Debug
20 Debug Debug
30 Info Info
40 Warn Warning
50 Error Error
60 Fatal Critical

Note: A log messages without a level map to SysLog Critical

Override log level from Schema

TBD

Acknowledgements

The implementation of Pino GELF is based in large part on pino-syslog and gelf-node.