Skip to content
lol, logging. We all do it poorly, best to laugh about it.
JavaScript Ruby
Latest commit 71ae004 Jul 7, 2012 @jordansissel clean landing.html

lol, logging.

Visit the site:

lol, logging


We all do it poorly, best to laugh about it, and perhaps find a better future.

This project aims to document logging stuff. I want to encourage the good, and derail the bad. I want to provide solid data that helps you make the right decisions about how/why/when you are logging and consuming logs.

I'm going to include bad things I do (and wish I did better) as well in this documentary.

Here are some of my ideas:

cultural battles around logging protocols


  • "I need human readable raw log data"
  • "We just dump random stuff over syslog!"
  • "We log key=value!"

"Standards" and other destructive forces

Why each of the following are complete bad, and why, and perhaps what we can try doing, in general, to fix things.

  • timestamps. ugh.
  • RFC3164, 5424, 5425. Why each are bad.
  • ArcSight's bad "Common Event Format"
  • Splunk's bad "Common Information Model"
  • LogStash's bad JSON schema (this would be a link if this was actually documented ☹)
  • Graylog2's bad JSON schema
  • CEE's bad, well, everything. 4 serialization formats, 4 conformance levels, 2+ transport mechanisms == 30+ combinations of bad.

explorations of bad logging.

  • mysql (binary log, slow query log, debug log; all are completely different formats)
  • more?

explorations of good logging.

Please tell me someone has an example of good logging in an application. We can't be all totally screwing this up across the world.


logging libraries


  • printf-style loggers like: ruby logger, python logging, etc.


  • log4j MDC/NDC, ruby-cabin, etc

types of logs

  • tracing (for the purposes of debugging)
  • accounting (for numerical applications like billing, metrics, etc)
  • transaction log (for the purposes of rollback and replay)

typical problems

  • Fat logs: "I have 300 gigs of logs, how can I make this useful?"
  • Fast logs: "I have 50,000 events logged per second, how can I make this useful?"
  • Lawn mowing: "I don't really use our logs much, but we have to use complex logrotate rules or otherwise we run out of disk and it takes down production"
  • Syntax vs domain: to answer domain questions (how many customers signed up?), you require syntax knowledge (how to parse apache logs)
  • Wrong audience: Giving the user stack traces instead of English.
Something went wrong with that request. Please try again.