Skip to content

Mission: Agree on a common format for application structured logs

Notifications You must be signed in to change notification settings

bitc/structured-logging-schema

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 

Repository files navigation

Schema Comparison between Structured Logging Libraries

The goal of this document is to summarize the differences between various structured logging libraries, in order to encourage a discussion so that a consensus can be reached for a common format.

Please use the GitHub issues for discussion, and submit GitHub Pull requests to add/fix the information in the tables.

Message and Payload

Behold the following logging command that you might find in an application. There is a "message" string, and a "payload" object:

logger.info("Hello World", {"animal": "cat", "numLegs": 4})
             ^^^^^^^^^^^   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
             Message       Payload object

Some libraries merge the payload object with the root object, so the event will look like this:

{
    "timestamp": "2018-06-18T23:16:45.000Z",
    "message": "Hello World",
    "animal": "cat",
    "numLegs": 4
}

While other libraries nest the payload object beneath some key, with a result like this:

{
    "timestamp": "2018-06-18T23:16:45.000Z",
    "message": "Hello World",
    "data": {
        "animal": "cat",
        "numLegs": 4
    }
}
Library Message JSON Key Payload Object JSON Key
bunyan (JavaScript) msg Merged with root
katip (Haskell) msg data
logrus (Go) msg Merged with root
logstash message Merged with root
ougai (Ruby) msg Merged with root
pygogo (Python) message Merged with root
roarr (JavaScript) message context
semantic logger (Ruby) message payload
serilog (C#) @m / @mt Merged with root
slog (Rust) msg Merged with root
structlog (Python) msg HELP NEEDED

Timestamp JSON Key and format

All libraries automatically add a timestamp to all events.

Library Timestamp JSON Key Timestamp format
bunyan (JavaScript) time ISO 8601
katip (Haskell) at ISO 8601
logrus (Go) time ISO 8601
logstash timestamp / @timestamp ISO 8601 / varies
ougai (Ruby) time ISO 8601
pygogo (Python) time ISO 8601
roarr (JavaScript) time Epoch milliseconds
semantic logger (Ruby) timestamp ISO 8601
serilog (C#) Timestamp / @t ISO 8601
slog (Rust) ts ISO 8601
structlog (Python) timestamp ISO 8601

Log Level

All libraries require that that all events are tagged with a level, chosen from a pre-defined list of available levels.

Log Level JSON Key

Library Log Level JSON Key
bunyan (JavaScript) level
katip (Haskell) sev
logrus (Go) level
logstash HELP NEEDED
ougai (Ruby) level
pygogo (Python) level
roarr (JavaScript) context.logLevel
semantic logger (Ruby) level
serilog (C#) Level / @l
slog (Rust) level
structlog (Python) level

Available Log Levels

Library VERBOSE TRACE DEBUG INFO NOTICE WARNING ERROR CRITICAL ALERT FATAL PANIC EMERGENCY
bunyan (JavaScript) N/A 10 20 30 N/A 40 50 N/A N/A 60 N/A N/A
katip (Haskell) N/A N/A Debug Info Notice Warning Error Critical Alert N/A N/A Emergency
logrus (Go) N/A N/A debug info N/A warning error N/A N/A fatal panic N/A
logstash HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED
ougai (Ruby) N/A 10 20 30 N/A 40 50 N/A N/A 60 N/A N/A
pygogo (Python) N/A N/A DEBUG INFO N/A WARNING ERROR CRITICAL N/A N/A N/A N/A
roarr (JavaScript) N/A 10 20 30 N/A 40 50 N/A N/A 60 N/A N/A
semantic logger (Ruby) N/A trace debug info N/A warn error N/A N/A fatal N/A N/A
serilog (C#) Verbose N/A Debug Information N/A Warning Error N/A N/A Fatal N/A N/A
slog (Rust) N/A TRACE DEBUG INFO N/A WARN ERROR CRITICAL N/A N/A N/A N/A
structlog (Python) N/A N/A debug info N/A warning error critical N/A N/A N/A N/A

System Context Data

Some libraries automatically add additional fields to all events (such as process ID and thread ID) and/or allow you to configure a few pre-determined global fields suchs the current environment and application.

Library Environment Application Namespace Package Machine Hostname Process ID Thread ID
bunyan (JavaScript) N/A N/A N/A N/A hostname pid N/A
katip (Haskell) env app (Array of strings) ns (Array of strings) N/A host pid thread
logrus (Go) N/A N/A N/A N/A N/A N/A N/A
logstash HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED HELP NEEDED
ougai (Ruby) N/A app N/A N/A hostname pid N/A
pygogo (Python) N/A N/A N/A N/A N/A N/A N/A
roarr (JavaScript) N/A context.application context.namespace context.package context.hostname N/A N/A
semantic logger (Ruby) environment application N/A N/A host pid thread
serilog (C#) N/A N/A N/A N/A N/A N/A N/A
slog (Rust) N/A N/A N/A N/A N/A N/A N/A
structlog (Python) N/A N/A N/A N/A N/A N/A N/A

HTTP Request Context Data

Logging of HTTP requests is very common. Let's figure out a common format for HTTP method, host, url, headers, remote IP, remote IP geo-lookup, response status code, response size, etc...

!!! TODO !!!

About

Mission: Agree on a common format for application structured logs

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published