Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send log to multiple log drivers #17910

Closed
jimmidyson opened this issue Nov 11, 2015 · 61 comments
Closed

Send log to multiple log drivers #17910

jimmidyson opened this issue Nov 11, 2015 · 61 comments

Comments

@jimmidyson
Copy link

@jimmidyson jimmidyson commented Nov 11, 2015

I'd like to be able to demux logs to multiple log drivers. The use case for this is to log to disk with json-log driver so that docker logs (live streaming) works but also be able to send the logs to archive via e.g. fluentd log driver for longer term viewing/searching/filtering/etc.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 11, 2015

Perhaps viewing the fluentd logs is really the best option here (although supporting docker logs for other drivers than json-file and journald would be nice to have)

Using the json-file is really discouraged for production use, and can be quite resource-hungry for high-volume logging.

@jimmidyson
Copy link
Author

@jimmidyson jimmidyson commented Nov 11, 2015

Agreed. Fluentd doesn't do any storage itself, just routing to various storage/processing backends so not really an option unless the backend supports log streaming. In my case (& Kubernetes examples) that's Elasticsearch normally which doesn't support streaming.

I didn't know that docker logs worked with journald - that would be a much better way to go & have rsyslog or something configured to route logs on from there perhaps.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 11, 2015

Yup, it was added in #13707. Forgot which release that was (sorry 😄)

@GordonTheTurtle
Copy link

@GordonTheTurtle GordonTheTurtle commented Nov 12, 2015

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@renanvicente

@crosbymichael
Copy link
Contributor

@crosbymichael crosbymichael commented Nov 13, 2015

Would it be better to just log to something like syslog then have all the tools built around that forward to other locations? Its alot of overhead in the daemon to so something like this.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 15, 2015

I agree with @crosbymichael here; I'm not sure we want to add the extra complexity in the daemon for this.

@oncletom
Copy link

@oncletom oncletom commented Nov 17, 2015

Agreed with @crosbymichael, more portable as any tool can tail the logfile based on one or many container IDs.

@jimmidyson
Copy link
Author

@jimmidyson jimmidyson commented Nov 17, 2015

Can syslog driver still work with docker logs? I was under the impression that was only json-log & journald? I agree that your approach generally makes more sense though.

@oncletom
Copy link

@oncletom oncletom commented Nov 17, 2015

@jimmidyson not at the moment, but you can still do tail -f /var/log/syslog | grep --line-buffered <container-id> (or /var/log/messages on Fedora/CentOS etc.).

docker logs could wrap that for us to fully embrace logging strategies but this would be only a sugar on top of your system habits.

@jimmidyson
Copy link
Author

@jimmidyson jimmidyson commented Nov 17, 2015

The underlying issue I'm trying to solve is moving logs off of nodes as quickly as possible for archiving/filtering/searching/etc while retaining docker logs functionality for live viewing of logs. I guess some configuration around journald should do that for me but it would be good to support docker logs for all log drivers (fluentd comes to mind as one to support somehow for that).

@oncletom
Copy link

@oncletom oncletom commented Nov 17, 2015

@jimmidyson rsyslog is your friend too :-)

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Nov 18, 2015

@jimmidyson supporting docker logs on a logging backend is highly dependent on the service being called.
Logging to syslog for instance has no defined way to actually to and read those logs.

@djsly
Copy link

@djsly djsly commented Jan 11, 2016

We have the same requirements, currently logs are pushed to log stash using the gelf driver but for quick debugging (example : kube-ui / docker logs) we would like to keep the functionality of the json-log driver. log-opt for json-log could be keeping 1-2 days of logs while log stash provide us with a long term archiving solution.

@mihasK
Copy link

@mihasK mihasK commented Jan 22, 2016

Agreed with @djsly . I want to use gelf driver. I can set up logstash to log to a local file, and use tail -f for quick debugging, but I think there will be json-formatted log there, containing extra information like 'container_id', 'image_name', so it will harder to perceive information comparing with docker logs format.

@jotunskij
Copy link

@jotunskij jotunskij commented Mar 27, 2016

+1 for the same reasons as multiple people before me

@safanaj
Copy link

@safanaj safanaj commented Apr 6, 2016

+1

@jdunmore
Copy link

@jdunmore jdunmore commented Apr 6, 2016

I hate leaving a me too comment, exactly the same problem here for aws cloudwatch logs, and not being able to user docker logs nor the kube-ui.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Apr 6, 2016

Aren't there better tools for reading/parsing logs than docker logs?
On a prod system I would expect to be able to read all my logs in one place.

@jimmidyson
Copy link
Author

@jimmidyson jimmidyson commented Apr 6, 2016

@cpuguy83 For archiving logs there are loads of great tools. The problem is live streaming logs which, even in a single place, ultimately usually stream straight from docker logs. Both archiving & live streaming are important for prod systems.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Apr 6, 2016

@jimmidyson For live streaming you can do docker attach --no-stdin

@erindru
Copy link

@erindru erindru commented Apr 12, 2016

What about an "aggregate" logging driver thats sole purpose is to delegate to multiple other logging drivers? That way we could log to both json-log to retain docker logs functionality (and configure it to have a small max file size for live streaming) and also use something like GELF.

Is this a terrible idea?

@michaelajr
Copy link

@michaelajr michaelajr commented May 15, 2016

+1 for an aggregate diver delegating to multiple log drivers. That would give the most flexability. Did not know about "attach" for live-steam. Will try that. Thanks.

@uschtwill
Copy link

@uschtwill uschtwill commented May 31, 2016

Thanks for pointing that out, @cpuguy83, that's really all that's needed for me. Logs to Logstash via gelf, docker attach --no-stdin for development and debugging. Beautiful!

@mcandre
Copy link
Contributor

@mcandre mcandre commented Jun 30, 2016

+1

4 similar comments
@pulserdd
Copy link

@pulserdd pulserdd commented Jul 11, 2016

+1

@bwnyasse
Copy link

@bwnyasse bwnyasse commented Aug 5, 2016

+1

@averri
Copy link

@averri averri commented Aug 13, 2016

+1

@dmavrin
Copy link

@dmavrin dmavrin commented Aug 17, 2016

+1

@evmin
Copy link

@evmin evmin commented Jan 23, 2017

For what it's worth, it looks like docker logs now supports the journald logging driver. You can then use fluentd to stream from journald to wherever (e.g Elasticsearch), while still maintaining the benefits of docker logs.

Tried that. Fluentd can not read journald reliably as yet.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Apr 10, 2017

Is multiple drivers really needed or just the ability to be able to call docker logs when another driver is used?

By the way, logging plugins were just merged a multi-logger could be implemented there. I do not think we will add support for multiple drivers in the core, though we can look at a solution to enabling docker logs for all drivers.

@erindru
Copy link

@erindru erindru commented Apr 10, 2017

@cpuguy83 retaining the ability to use docker logs would solve my use case. Being still able to configure how many logs are retained locally would be crucial though, so you don't run the server out of space which is one of the main points of using a different logging driver in the first place

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Apr 10, 2017

@erindru I hope running out of space isn't the main case since we have rotation support 😄
Getting logs off an ephemeral machine though is pretty important.

@erindru
Copy link

@erindru erindru commented Apr 10, 2017

Sorry what I meant was, in order for docker logs to work the logs need to be local, right? So retaining the ability to configure the local log rotation in addition to configuring the specified logging driver would be crucial

@binman-docker
Copy link

@binman-docker binman-docker commented Apr 10, 2017

Multiple drivers would open up a lot of use cases and flexibility. I know we've talked about how complex it would be compared to a ring buffer or similar, but I think it's the "better" option long-term. Either would solve our specific use case though.

@simplesteph
Copy link

@simplesteph simplesteph commented Apr 10, 2017

Short term all I need is to do docker logs when my log is still shipping elsewhere!

@michaelajr
Copy link

@michaelajr michaelajr commented Apr 11, 2017

Exactly. having logs shipped off box is super important - but having docker logs available to quickly debug would be great.

@shane-axiom
Copy link

@shane-axiom shane-axiom commented Apr 11, 2017

+1 the ability to use local docker logs when another driver is used solves my use case, especially if true multiple drivers could be handled via a plugin

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Apr 11, 2017

@shane-axiom Even if it take some time to get the feature into Docker, a plugin can declare that it supports reading logs, so this can wholly be handled by a plugin.

@sudo-bmitch
Copy link

@sudo-bmitch sudo-bmitch commented Apr 11, 2017

I'm also looking for the ability to use docker logs with another plugin being used. Multiple plugin support or a meta logging plugin that sends to other logging plugins would be a nice to have, but not required for the environments I'm supporting right now.

@shurshun
Copy link

@shurshun shurshun commented Apr 12, 2017

+1 for the ability to use docker logs with another plugin being used.

@1arrow
Copy link

@1arrow 1arrow commented May 11, 2017

+1

@gerardjp
Copy link

@gerardjp gerardjp commented Aug 17, 2017

+1 for the ability to use docker logs with another plugin being used.

@sampleref
Copy link

@sampleref sampleref commented Aug 28, 2017

+1 for docker logs along with a logging driver

@weijiekoh
Copy link

@weijiekoh weijiekoh commented Sep 11, 2017

+1. This would be extremely useful.

@eugenegordeiev
Copy link

@eugenegordeiev eugenegordeiev commented Dec 15, 2017

+1. Its very difficult to debug without having docker logs available..

@Miserlou
Copy link

@Miserlou Miserlou commented Mar 14, 2018

This issue is cascading into downstream projects - for instance, since Docker is powering some Nomad tasks, there is no way to multiplex Nomad log streams with different drivers when using Docker-driven tasks. It'd be great if this feature got more attention.

@gbolo
Copy link

@gbolo gbolo commented Aug 4, 2018

Sending logs to a remote server using one of the logging drivers while still retaining the local logs would be a great option to have

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Aug 4, 2018

This is available in Docker EE, btw.

@sudo-bmitch
Copy link

@sudo-bmitch sudo-bmitch commented Aug 7, 2018

@cpuguy83 Is this planned for a Docker CE release soon, or will it remaining an EE only feature?

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Aug 8, 2018

@sudo-bmitch I can't answer that.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 3, 2020

The upcoming Docker 20.10 release will come with the feature described above ("dual logging"), which uses the local logging driver as a ring-buffer, which makes docker logs work when using a logging driver that does not have "read" support (for example, logging drivers that send logs to a remote logging aggregator).

This feature was previously available in Docker Enterprise, but has been open-sourced in #40543

I think that addresses the essence of this issue (being able to use docker logs, irregardless of the logging driver that's used for a container).

For clarity; there are no plans currently to allow specifying multiple arbitrary logging-drivers per containers. I don't think we want to add that complexity, and orthogonal to the feature request reported here.

I'm closing this ticket as I think this is resolved by the above.

@thaJeztah thaJeztah closed this Nov 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet