Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
tree: 327679fd21
Fetching contributors…

Cannot retrieve contributors at this time

file 80 lines (53 sloc) 2.745 kb

lol, logging.

Visit the site:

lol, logging

Background

We all do it poorly, best to laugh about it, and perhaps find a better future.

This project aims to document logging stuff. I want to encourage the good, and derail the bad. I want to provide solid data that helps you make the right decisions about how/why/when you are logging and consuming logs.

I'm going to include bad things I do (and wish I did better) as well in this documentary.

Here are some of my ideas:

cultural battles around logging protocols

Sigh:

  • "I need human readable raw log data"
  • "We just dump random stuff over syslog!"
  • "We log key=value!"

"Standards" and other destructive forces

Why each of the following are complete bad, and why, and perhaps what we can try doing, in general, to fix things.

  • timestamps. ugh.
  • RFC3164, 5424, 5425. Why each are bad.
  • ArcSight's bad "Common Event Format"
  • Splunk's bad "Common Information Model"
  • LogStash's bad JSON schema (this would be a link if this was actually documented ☹)
  • Graylog2's bad JSON schema
  • CEE's bad, well, everything. 4 serialization formats, 4 conformance levels, 2+ transport mechanisms == 30+ combinations of bad.

explorations of bad logging.

  • mysql (binary log, slow query log, debug log; all are completely different formats)
  • more?

explorations of good logging.

Please tell me someone has an example of good logging in an application. We can't be all totally screwing this up across the world.

Anybody?

logging libraries

Sigh:

  • printf-style loggers like: ruby logger, python logging, etc.

Hurray:

  • log4j MDC/NDC, ruby-cabin, etc

types of logs

  • tracing (for the purposes of debugging)
  • accounting (for numerical applications like billing, metrics, etc)
  • transaction log (for the purposes of rollback and replay)

typical problems

  • Fat logs: "I have 300 gigs of logs, how can I make this useful?"
  • Fast logs: "I have 50,000 events logged per second, how can I make this useful?"
  • Lawn mowing: "I don't really use our logs much, but we have to use complex logrotate rules or otherwise we run out of disk and it takes down production"
  • Syntax vs domain: to answer domain questions (how many customers signed up?), you require syntax knowledge (how to parse apache logs)
  • Wrong audience: Giving the user stack traces instead of English.
Something went wrong with that request. Please try again.