Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log-driver for logstash #14949

Closed
Alexandre-io opened this issue Jul 24, 2015 · 26 comments
Closed

Add log-driver for logstash #14949

Alexandre-io opened this issue Jul 24, 2015 · 26 comments

Comments

@Alexandre-io
Copy link

@Alexandre-io Alexandre-io commented Jul 24, 2015

Hi,

A plan to support logstash as a log-driver ?

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Jul 24, 2015

Logging drivers are provided by the community, by someone who has a need for it.
That said doesn't logstash support both journald and syslog?

@Alexandre-io
Copy link
Author

@Alexandre-io Alexandre-io commented Jul 24, 2015

Syslog works like a charm with logstash but the current driver have some security concerns like ssl (auth) and queuing, no ?

@Alexandre-io Alexandre-io changed the title log-driver logstash ? log-driver logstash-forwarder ? Jul 24, 2015
@cpuguy83 cpuguy83 changed the title log-driver logstash-forwarder ? Add log-driver for logstash-forwarder Jul 24, 2015
@cpuguy83 cpuguy83 changed the title Add log-driver for logstash-forwarder Add log-driver for logstash Jul 24, 2015
@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Jul 24, 2015

I've updated the title for other people coming by who may want to implement this.

@rwincewicz
Copy link

@rwincewicz rwincewicz commented Aug 17, 2015

I've been looking at a logging driver that outputs to RabbitMQ as that's how I'm handling the logs from the rest of my services and applications. I've got logstash picking up the messages at the other end and putting them into elasticsearch. At the moment I'm not using SSL but that's next on my to-do list but it does satisfy your queuing requirement.
It's early days so it's rough around the edges but feel free to take a look: https://github.com/rwincewicz/docker/tree/add_amqp_logdriver/daemon/logger/amqp

@aishraj
Copy link

@aishraj aishraj commented Nov 22, 2015

#dibs

I'd to work on the above issue.

@runcom
Copy link
Member

@runcom runcom commented Nov 22, 2015

I fear the next one will be elastisearch itself

@mkuzmin
Copy link

@mkuzmin mkuzmin commented Nov 22, 2015

@aishraj
Copy link

@aishraj aishraj commented Nov 22, 2015

Thanks for linking into the libbeat issue.
Based on the comments1 there, it looks like the Elastic community feels that with Docker moving to a non-embedded plugin approach via Rest API for plugin registration, it would not be a good timing to implement this. Please let me know if that is the case, and if so, the best way to get started with the new API based plugin approach.

[1] https://github.com/elastic/libbeat/issues/37#issuecomment-153944581

@bitsofinfo
Copy link

@bitsofinfo bitsofinfo commented Dec 16, 2015

+1 using syslog is not a great match with multilined logs... logstash driver would be great, that writes to a logstash TCP input on the destination i.e. #18001

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Dec 16, 2015


edit: just noticed you mentioned exactly that one

@superdump
Copy link

@superdump superdump commented Jan 10, 2016

TCP+TLS and being able to either write to logstash or directly to elasticsearch would be good.

@mrkschan
Copy link

@mrkschan mrkschan commented Jan 15, 2016

@aishraj my comment on libbeat https://github.com/elastic/libbeat/issues/37#issuecomment-153944581 does not represent the views from elastic nor docker nor dockerbeat.

I think we better ask if docker could improve the log driver to a non-embedded one. @cpuguy83 do you have any insight there?

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Jan 15, 2016

@bitsofinfo docker's loggers only deals with single lines at a time in any case, it doesn't matter what the driver is capable of.

@mrkschan I don't think the existing plugin API is a good one for logging.
It uses HTTP, which itself has tremendous overhead, it's not good for streaming, etc.
I think the way this will end up being pluggable (beyond support from golang for dynamically loading plugins) is #18001.

TLDR on #18001; Give docker a socket to connect to (udp, tcp, unix, doesn't matter) and docker will stream logs to it in it's native format that people can write adapters for.

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Jan 15, 2016

That said I'm not sure if the other maintainers want to also bring on a logstash driver or not.
My personal concern is vendoring more and more dependencies and bloating the code base.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jan 15, 2016

Yes I think we should try to work on supporting external plugins for logging if possible; agreed on the vendoring, and bloating the code base

@Alexandre-io
Copy link
Author

@Alexandre-io Alexandre-io commented Apr 29, 2016

Since the release 1.10.0 syslog over TCP+TLS were added. So my last concern is about the queuing.

@Alexandre-io Alexandre-io reopened this Apr 29, 2016
@rchicoli
Copy link

@rchicoli rchicoli commented Aug 14, 2016

This is already supported by gelf log driver.
You just need make sure to use in logstash the plugin gelf as input

gelf {
          port => 12201
          type => "gelf"
          tags => "docker"
}

@danielmotaleite
Copy link

@danielmotaleite danielmotaleite commented Nov 8, 2016

with gelf, if i stop logstash for a few minutes, it loses the messages during that time... queuing messages only seem to work with lost packets, not lost of connection

@alkuzad
Copy link

@alkuzad alkuzad commented Nov 14, 2016

@rchicoli UDP is not a choice for server logs, it can be acceptable for statistics logs but not critical production logs when we have to ensure every event is logged. Right now I do not see any viable docker logger that can work without help of other processes than docker daemon itself.

@rchicoli
Copy link

@rchicoli rchicoli commented Nov 14, 2016

@alkuzad I do agree. Btw I am willing to contribute to this, but the maintainers are not accepting any new log-drivers anymore (topic related #25694) and I don't really know how to rewrite this to an external plugin. I wish we had a couple of more examples or some documentation about writting external plugins. Otherwise I kind need some help to do so.

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Nov 14, 2016

Working on a plugin implementation for log drivers right now, hope to have a PoC by EOD.

@tyanko1
Copy link

@tyanko1 tyanko1 commented Dec 29, 2016

@cpuguy83 Hey, what is the status on the logstash plugin? Could really use this!!! Thanks!

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Jan 2, 2017

@tyanko1 Implementation is here: #28403

Still a few more things to deal with, but it works, I have a test logging plugin up on hub (See integration-cli test from the mentioned PR).
It supports both writing and reading, when the plugin specifies that it supports reading.

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Apr 10, 2017

Log plugin support is now merged and should be part of 17.05.
I do not think we'll be accepting any new built-in log drivers.

Closing this. A logstash driver should be implemented as a plugin at this point. If anyone wants to discuss this feel free to ping me on Slack, or come see me @ DockerCon!

@rchicoli
Copy link

@rchicoli rchicoli commented Jan 3, 2018

here is a POC of docker log plugin for logstash docker-log-logstash. Although it is not much better than gelf at the moment.

This is just the first alpha release. There are some planning for adding caching, buffering, etc. Though it will take awhile until it is stable and ready for production, because I am doing this beside my real work.

Let me know, if you are still interested on it. Feel free to contribute also, if you are feeling like.

How to use:

Install

docker plugin install rchicoli/docker-log-logstash:0.5.1 --alias logstash

Logstash config

input {
    tcp {
            port => 5000
            codec => json
    }
}
output {
    stdout { codec => rubydebug }
}

Test

docker run --rm  -ti \
    --log-driver logstash \
    --log-opt logstash-url=127.0.0.1:5000 \
        alpine echo this is another logging driver

This is more an implementation of a TCP client, than Logstash.

Any ideas can be tracked on the project. For example, adding support to different input plugins, etc.

@rchicoli
Copy link

@rchicoli rchicoli commented Jan 9, 2018

The current status of docker-log-logstash:0.0.7 plugin. Even though it is in alpha, this plugin is capable of:

  • reconnecting to logstash server, in case of lost connection
  • caching messages to the filesystem, while logstash is down
  • send cached log information to logstash, when it is online

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet