Select and deploy a new log collector

[exarkun]

We had Fluentd for a while but it was problematic. Now we have nothing. Find something and start using it.

It should collect:

• Customer deployment logs to stdout and flog
• Infrastructure deployment logs (which all go to stdout)
• Kubernetes node logs (host journalctl probably)

It need not concern itself with metrics as Prometheus now fills that role.

[exarkun]

Fluentd (#603):

• 👎 Complicated, custom configuration languages (separate fluentd and fluent-bit languages
• 👎 Memory & CPU hungry
• 👎 Either broken in certain analysis cases or too complicated for me to figure out how to correctly tell it to do what I want
• 👎 Requires a sidecar process processing on-disk logs
• 👍 Popular (lots of other people asking questions about it)
• 👍 Open source
• 👍 Paid support available

[exarkun]

Graylog:

• 👎 Java
• 👎 Getting started guide produces non-working installation (possibly indicative of docs quality)
• 👎 Java
• 👎 Very sluggish UI
• 👎 Inherits setup and maintenance complexity of mongo and elasticsearch (Java!)
• 👎 Java
• 👍 Open source
• 👍 Somewhat popular (many people asking questions about it)
• 👍 Paid support available
• 👍 Built-in customizable UI

[exarkun]

Kafka:

[exarkun]

Logstash:

• 👎 Java & Ruby
• 👎 "Elastic Stack" docs are very sales/marketing focused (basically a distraction/trap, ignore them and read the technical docs)
• 👎 Funky configuration language (Well, Ruby... I guess?)
• 👎 Inherits setup and maintenance complexity of ElasticSearch (Java!)
• 👎 Java & Ruby!
• 👎 Apparently no Python implementation of the Beats protocol (but there are other input protocols so maybe it's actually okay)
• 👍 Relatively easy to get running and doing something after reading some docs
• 👍 Some support for Grafana (by way of Grafana ElasticSearch input source support)
• 👍 Widespread expertise combining with ElasticSearch for storage and Kibana for visualization
• 👍 Paid support available

Notes:

• Lumberjack protocol documentation