Server script for log shipping compliance

[netantho]

Create a python script that would be standalone web server listening for logs the same way as loginput does it.
It would display debugging info of logs received and warning if they are not compliant to the JSON standard.

[2xyo]

Why not use logstash for retrieving logs?
This solution has the advantage of offering many formats as input and it's easy to check if logs are compliant (with GROK).

[netantho]

@2xyo We prefer using heka internally to ship logs. Its lua sandbox + lpeg (kind of special regex implentation) can ship > 100,000 events/sec and allow nice scripting.

This task is rather to create a simple server with very minimal dependencies, ideally just python, that can be used by anyone who wants to send logs to MozDef (developer, IT, etc.) and wants something to emulate MozDef's receiver with debugging features.

[jeffbryner]

MozDef aims to be agnostic about log shippers, so logstash, nxlog, beaver, heka, rsyslog, fluentd, etc are all shippers we want to support. We use heka internally because it's a mozilla product, but logstash is supported out of the box by pointing logstash to a mozdef front-end http instance.

[2xyo]

Ok, perfect! Thank you for the clarification and for making me discover heka :)

[netantho]

We have heka config examples ;-)

[jeffbryner]

heka is deprecated, closing this in favor of ongoing fluentd work tracked in other issues.