Store application logs in a central location

[nickclyde]

From the roadmap:

Centralized logging

• Thesis: should not need to ssh into instance (danger-danger)
• Scope: instance level, application level
• Current-state: Leverage Runit, built-in ability to log to a port
• Elements: app_id, instance_id
• UI needs: expose logs to UI (hold-off)
• References: Kibana, #engineering-chat
• [shorter-term] considerations: not designing out certain logging providers
• [longer-term] Considerations: PII scrubbing
• Gov-connection: PII (SSNs, TINs, Names, Emails, DOBs, etc.)

svlogd allows us to send application logs over a port. That's easy enough, but we need to decide what to send those logs into that allows us to expose the logs to the UI or allow the user to use their own log viewing application/service. Kibana seems more suited to data visualization than log parsing; however, Elastic does offer Logstash, which is billed as a centralized log processor that can output to Elasticsearch or a number of other destinations like Redis, S3 or even a HTTP endpoint or a websocket.

Logstash also offers a library of filters that can do things like "anonymize PII data and exclude sensitive fields completely", which we talked about above.

I think Logstash might be good for us, but I'm a little less certain on where the final datastore should be. @paulsmith @llimllib thoughts?

Logstash Github page

[llimllib]

Is this roughly the architecture you’re imagining?

(forgive my chickenscratch)

[nickclyde]

Yep, that looks about right!

[llimllib]

👍 that seems like the way to go to me

[malford]

FWIW I implemented a EFK (elasticsearch, fluentd, kibana) stack earlier this year at my previous role. It uses fluentd for log forwarding. Fluentd is ruby based and has a bit lower memory footprint and uses tags instead statements for forwarding. The supported endpoint options are about the same as logstash and seems to have good community support.

[llimllib]

Also rsyslog can ship logs directly to elasticsearch. Could we do that as a first cut, and cut out the middleman entirely?

[llimllib]

http://wiki.rsyslog.com/index.php/HOWTO:_rsyslog_%2B_elasticsearch

pros:

• less infrastructure (logstash requires a server and redis; fluentd requires a server)
• higher performance (no need to travel through a middleman)

cons:

• no ability to filter/alert/tag what have you at the logstash/fluentd level

what else would you add to these?

[nickclyde]

fluentd looks nice as well. Maybe though, it would be good to do as @llimllib mentioned and cut out the middleman for now until we have a concrete need for filtering?

[paulsmith]

Yep I support configuring svlogd to log to rsyslogd process and configuring rsyslogd to forward to ES -- that way we can log both the application and any other system logs on the ec2 instance we want.

…

On Thu, Aug 24, 2017 at 1:55 PM, Nick Clyde ***@***.***> wrote: fluentd looks nice as well. Maybe though, it would be good to do as @llimllib <https://github.com/llimllib> mentioned and cut out the middleman for now until we have a concrete need for filtering? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#132 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAEugffCGsRPduWK3kxRk9ulF18Gp5Tks5sbcc4gaJpZM4PBuO9> .

-- CTO, Ad Hoc https://adhocteam.us/