New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log-driver for logstash #14949

Closed
Alexandre-io opened this Issue Jul 24, 2015 · 26 comments

Comments

Projects
None yet
@Alexandre-io

Alexandre-io commented Jul 24, 2015

Hi,

A plan to support logstash as a log-driver ?

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Jul 24, 2015

Logging drivers are provided by the community, by someone who has a need for it.
That said doesn't logstash support both journald and syslog?

@Alexandre-io

This comment has been minimized.

Alexandre-io commented Jul 24, 2015

Syslog works like a charm with logstash but the current driver have some security concerns like ssl (auth) and queuing, no ?

@Alexandre-io Alexandre-io changed the title from log-driver logstash ? to log-driver logstash-forwarder ? Jul 24, 2015

@cpuguy83 cpuguy83 changed the title from log-driver logstash-forwarder ? to Add log-driver for logstash-forwarder Jul 24, 2015

@cpuguy83 cpuguy83 changed the title from Add log-driver for logstash-forwarder to Add log-driver for logstash Jul 24, 2015

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Jul 24, 2015

I've updated the title for other people coming by who may want to implement this.

@rwincewicz

This comment has been minimized.

rwincewicz commented Aug 17, 2015

I've been looking at a logging driver that outputs to RabbitMQ as that's how I'm handling the logs from the rest of my services and applications. I've got logstash picking up the messages at the other end and putting them into elasticsearch. At the moment I'm not using SSL but that's next on my to-do list but it does satisfy your queuing requirement.
It's early days so it's rough around the edges but feel free to take a look: https://github.com/rwincewicz/docker/tree/add_amqp_logdriver/daemon/logger/amqp

@aishraj

This comment has been minimized.

aishraj commented Nov 22, 2015

#dibs

I'd to work on the above issue.

@runcom

This comment has been minimized.

Member

runcom commented Nov 22, 2015

I fear the next one will be elastisearch itself

@mkuzmin

This comment has been minimized.

mkuzmin commented Nov 22, 2015

Related issue: elastic/libbeat#37

@aishraj

This comment has been minimized.

aishraj commented Nov 22, 2015

Thanks for linking into the libbeat issue.
Based on the comments1 there, it looks like the Elastic community feels that with Docker moving to a non-embedded plugin approach via Rest API for plugin registration, it would not be a good timing to implement this. Please let me know if that is the case, and if so, the best way to get started with the new API based plugin approach.

[1] elastic/libbeat#37 (comment)

@bitsofinfo

This comment has been minimized.

bitsofinfo commented Dec 16, 2015

+1 using syslog is not a great match with multilined logs... logstash driver would be great, that writes to a logstash TCP input on the destination i.e. #18001

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Dec 16, 2015

~~~Also note this proposal; #18001~~~

edit: just noticed you mentioned exactly that one

@superdump

This comment has been minimized.

superdump commented Jan 10, 2016

TCP+TLS and being able to either write to logstash or directly to elasticsearch would be good.

@mrkschan

This comment has been minimized.

mrkschan commented Jan 15, 2016

@aishraj my comment on libbeat elastic/libbeat#37 (comment) does not represent the views from elastic nor docker nor dockerbeat.

I think we better ask if docker could improve the log driver to a non-embedded one. @cpuguy83 do you have any insight there?

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Jan 15, 2016

@bitsofinfo docker's loggers only deals with single lines at a time in any case, it doesn't matter what the driver is capable of.

@mrkschan I don't think the existing plugin API is a good one for logging.
It uses HTTP, which itself has tremendous overhead, it's not good for streaming, etc.
I think the way this will end up being pluggable (beyond support from golang for dynamically loading plugins) is #18001.

TLDR on #18001; Give docker a socket to connect to (udp, tcp, unix, doesn't matter) and docker will stream logs to it in it's native format that people can write adapters for.

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Jan 15, 2016

That said I'm not sure if the other maintainers want to also bring on a logstash driver or not.
My personal concern is vendoring more and more dependencies and bloating the code base.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 15, 2016

Yes I think we should try to work on supporting external plugins for logging if possible; agreed on the vendoring, and bloating the code base

@Alexandre-io

This comment has been minimized.

Alexandre-io commented Apr 29, 2016

Since the release 1.10.0 syslog over TCP+TLS were added. So my last concern is about the queuing.

@Alexandre-io Alexandre-io reopened this Apr 29, 2016

@rchicoli

This comment has been minimized.

rchicoli commented Aug 14, 2016

This is already supported by gelf log driver.
You just need make sure to use in logstash the plugin gelf as input

gelf {
          port => 12201
          type => "gelf"
          tags => "docker"
}
@danielmotaleite

This comment has been minimized.

danielmotaleite commented Nov 8, 2016

with gelf, if i stop logstash for a few minutes, it loses the messages during that time... queuing messages only seem to work with lost packets, not lost of connection

@alkuzad

This comment has been minimized.

alkuzad commented Nov 14, 2016

@rchicoli UDP is not a choice for server logs, it can be acceptable for statistics logs but not critical production logs when we have to ensure every event is logged. Right now I do not see any viable docker logger that can work without help of other processes than docker daemon itself.

@rchicoli

This comment has been minimized.

rchicoli commented Nov 14, 2016

@alkuzad I do agree. Btw I am willing to contribute to this, but the maintainers are not accepting any new log-drivers anymore (topic related #25694) and I don't really know how to rewrite this to an external plugin. I wish we had a couple of more examples or some documentation about writting external plugins. Otherwise I kind need some help to do so.

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Nov 14, 2016

Working on a plugin implementation for log drivers right now, hope to have a PoC by EOD.

@tyanko1

This comment has been minimized.

tyanko1 commented Dec 29, 2016

@cpuguy83 Hey, what is the status on the logstash plugin? Could really use this!!! Thanks!

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Jan 2, 2017

@tyanko1 Implementation is here: #28403

Still a few more things to deal with, but it works, I have a test logging plugin up on hub (See integration-cli test from the mentioned PR).
It supports both writing and reading, when the plugin specifies that it supports reading.

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Apr 10, 2017

Log plugin support is now merged and should be part of 17.05.
I do not think we'll be accepting any new built-in log drivers.

Closing this. A logstash driver should be implemented as a plugin at this point. If anyone wants to discuss this feel free to ping me on Slack, or come see me @ DockerCon!

@rchicoli

This comment has been minimized.

rchicoli commented Jan 3, 2018

here is a POC of docker log plugin for logstash docker-log-logstash. Although it is not much better than gelf at the moment.

This is just the first alpha release. There are some planning for adding caching, buffering, etc. Though it will take awhile until it is stable and ready for production, because I am doing this beside my real work.

Let me know, if you are still interested on it. Feel free to contribute also, if you are feeling like.

How to use:

Install

docker plugin install rchicoli/docker-log-logstash:0.5.1 --alias logstash

Logstash config

input {
    tcp {
            port => 5000
            codec => json
    }
}
output {
    stdout { codec => rubydebug }
}

Test

docker run --rm  -ti \
    --log-driver logstash \
    --log-opt logstash-url=127.0.0.1:5000 \
        alpine echo this is another logging driver

This is more an implementation of a TCP client, than Logstash.

Any ideas can be tracked on the project. For example, adding support to different input plugins, etc.

@rchicoli

This comment has been minimized.

rchicoli commented Jan 9, 2018

The current status of docker-log-logstash:0.0.7 plugin. Even though it is in alpha, this plugin is capable of:

  • reconnecting to logstash server, in case of lost connection
  • caching messages to the filesystem, while logstash is down
  • send cached log information to logstash, when it is online
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment