Defining log-driver and log-opt when specifying pod in RC and Pod

[ejemba]

We need to be able to define the following options when specifying the pod definition in RC and Pod

--log-driver= Logging driver for container
--log-opt=[] Log driver options

These options should be settable at container level and have been introduced with Docker 1.8.

Since docker client lib support both options as well adding those options to the pod definition is now doable.

[timothysc]

/cc @kubernetes/rh-cluster-infra

[ncdc]

Hmm, I think we'll probably want to be able to set this cluster-wide as a default, and then maybe allow specific pod definitions to override.

cc @sosiouxme @smarterclayton @liggitt @jwhonce @jcantrill @bparees @jwforres

[smarterclayton]

Can you describe how you would leverage this on a per container basis (use case)? We traditionally do not expose Docker specific options directly in containers unless they can be cleanly abstracted across runtimes. Knowing how you would want to use this will help justify it.

[sosiouxme]

Note that docker logs still only support json-file and journald drivers, though I imagine that list could expand.

Perhaps what users would actually want is a selection of defined log writing endpoints, not exposure to the logging driver details.

[ejemba]

@ncdc @smarterclayton I agree with both of you, after reconsidering our use case in internal, it turns out that

1. Our primary need is to protect our nodes. We send the logs to a log server but if it fails, logs fallback on docker internal logs. In such case, to prevent node saturation we need a cluster wide behaviour for docker log
2. Exposing specific docker options in the pod/Rc definitions is not a good idea as @smarterclayton suggested it. We also agree with an abstraction allowing definition of high level log behaviour if possible
3. Another option is making change on kubelet configuration files and code to handle such log behavior

[smarterclayton]

The changes to the salt templates to make this a default should not be
terribly difficult. It's really just proper daemon configuration (and
dealing with any changes to log aggregation via fluentd by virtue of
selecting a different source)

On Tue, Oct 13, 2015 at 10:55 AM, Epo Jemba notifications@github.com
wrote:

@ncdc https://github.com/ncdc @smarterclayton
https://github.com/smarterclayton I agree with both of you, after
reconsidering our use case in internal, it turns out that

1. Our primary need is to protect our nodes. We send the logs to a log
   server but if it fails, logs fallback on docker internal logs. In such
   case, to prevent node saturation we need a cluster wide behaviour for
   docker log
2. Exposing specific docker options in the pod/Rc definitions is not a
   good idea as @smarterclayton https://github.com/smarterclayton
   suggested it. We also agree with an abstraction allowing definition of high
   level log behaviour if possible
3. Another option is making change on kubelet configuration files and
   code to handle such log behavior

—
Reply to this email directly or view it on GitHub
#15478 (comment)
.

[halr9000]

👍

Note that there are now 9 logging drivers. What's the consensus on getting this one in?

[briangebala]

+1

[obeattie]

In case anyone isn't aware, you can define the default log driver on a per-node basis with a flag to the Docker daemon (--log-driver). In my environment, I set the driver to journald this way. I struggle to think of a use-case for overriding this on a per-container basis to be honest.

[timothysc]

Most clustering will not want their logs going "out-of-band", so what is the feature enablement that this would provide.

Also, from an ops perspective it looks like a loss of control. Currently we set the defaults and configure a logging stack to aggregate.

[jnardiello]

+1 on this.
Not being able to control how docker logging is handled implies that the only sane logging option is using the tools shipped with k8s, which is an incredible limitation.

@timothysc here our use-case. We have a complex dynamic infrastructure (~100 machines) with a lot of existing services running on them, with our own logstash to gather logs. Well, we are now trying to move our services, one by one, to k8s and to me there seems to be no clean way to integrate logging between our existing infrastructure and containers clustered on k8s.

K8S is extremely opinionated on how you gather logs. This might be great for whoever is starting from scratch on a simple infrastructure. For everyone else working on complex infrastructures which would not mind to dive deep and to implement a custom logging mechanism, there is simply not way to do it at the moment, which is quite frustrating.

Hopefully, it makes sense.

[smarterclayton]

So in your scenario logs are truly "per application", but you have to
ensure the underlying host supports those logs? That's the concern we're
discussing here - either we do cluster level, or node level, but if we do
pod level, then the scheduler would have to be aware of what log drivers
are present where. As much as possible we try to avoid that.

On Mon, May 23, 2016 at 10:50 AM, Jacopo Nardiello <notifications@github.com

wrote:

+1 on this.
Not being able to control how docker logging is handled implies that the
only sane logging option is using the tools shipped with k8s, which is an
incredible limitation.

@timothysc https://github.com/timothysc here our use-case. We have a
complex dynamic infrastructure (~100 machines) with a lot of existing
services running on them, with our own logstash to gather logs. Well, we
are now trying to move our services, one by one, to k8s and to me there
seems to be no clean way to integrate logging between our existing
infrastructure and containers clustered on k8s.

K8S is extremely opinionated on how you gather logs. This might be great
for whoever is starting from scratch on a simple infrastructure. For
everyone else working on complex infrastructures which would not mind to
dive deep and to implement a custom logging mechanism, there is simply not
way to do it at the moment, which is quite frustrating.

Hopefully, it makes sense.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15478 (comment)

[jnardiello]

@smarterclayton I do understand about your concerns and they are well placed. I'm not sure if the whole cluster has to be aware of the existence of pod-level logging, what I think we should do is giving the option to log pod stdout/stderr somewhere (a file based on their current pod name?) so that anyone willing to implement their custom solution, would have a persisted place where to get the content. This opens up a HUGE chapter though as logrotation is not trivial.

These are just my two cents, but we can't pretend that real-world complex scenarios just give up their existing logging infrastructure.

[smarterclayton]

Are you specifying custom log options per application? How many different
sets of log options would you have per cluster? If there are small sets of
config, an option would be to support an annotation on pods that is
correlated to node level config that offers a number of "standard log
options". I.e. at kubelet launch time define a "log mode X" (which defines
custom log options and driver), and the pod would specify "
pod.alpha.kubernetes.io/log.mode=X".

Yet another option would be that we expose a way to let deployers have the
opportunity to mutate the container definition immediately before we start
the container. That's harder today because we'd have to serialize the
docker def out to an intermediate format, execute it, and then run it
again, but potentially easier in the future.

Finally, we could expose key value pairs on the container interface that
are passed to the container engine directly, offer no API guarantees for
them, and ensure PodSecurityPolicy can regulate those options. That would
be the escape hatch for callers, but we wouldn't be able to provide any
guarantee those would continue to work across releases.

On Thu, May 26, 2016 at 5:34 AM, Jacopo Nardiello notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton I do understand about
your concerns and they are well placed. I'm not sure if the whole cluster
has to be aware of the existence of pod-level logging, what I think we
should do is giving the option to log pod stdout/stderr somewhere (a file
based on their current pod name?) so that anyone willing to implement their
custom solution, would have a persisted place where to get the content.
This opens up a HUGE chapter though as logrotation is not trivial.

These are just my two cents, but we can't pretend that real-world complex
scenarios just give up their existing logging infrastructure.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15478 (comment)

[ncdc]

@smarterclayton have you seen #24677 (comment)

[smarterclayton]

No, thanks. Moving discussion there.

On Thu, May 26, 2016 at 11:23 AM, Andy Goldstein notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton have you seen #24677
(comment)
#24677 (comment)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15478 (comment)

[pbthorste]

Hi there,
I think this is an important feature that should be considered for kubernetes.
Enabling the use of Docker's log driver can solve some non-trivial problems.

I would say that logging to disk is an anti-pattern. Logs are inherently "state", and should preferably not be saved to disk. Shipping the logs directly from a container to a repository solves many problems.

Setting the log driver would mean that the kubectl logs command can not show anything anymore.
While that feature is "nice to have" - the feature won't be needed when the logs are available from a different source.

Docker already has log drivers for google cloud (gcplogs) and Amazon (awslogs). While it is possible to set them on the Docker daemon itself, that has many drawbacks. By being able to set the two docker options:

--log-driver= Logging driver for container
--log-opt=[] Log driver options

It would be possible to send along labels (for gcplogs) or awslogs-group (for awslogs)
specific to a pod. That would make it easy to find the logs at the other end.

I have been reading up on how people are handling logs in kubernetes. Many seem to set up some elaborate scrapers that forward the logs to central systems. Being able to set the log driver will make that unnecessary - freeing up time to work on more interesting things :)

[daniilyar]

I can also add that some people, including me, want to perform docker logs rotation via '--log-opt max-size' option on JSON logging driver (which is native to docker) instead of setting up logrotate on the host. So, even exposing just the '--log-opt' option would be greatly appreciated

[barnettZQG]

I have modified the k8s, when creating container configuration LogConfig.

[defat]

+1
Using docker log driver for centralized log collection looks much simpler then creating symbolic links for log files, mounting them to a special fluentd container, tailing them and managing log rotation.

[et304383]

Use case for per container configuration: I want to log elsewhere or differently for containers I deploy and I don't care about (or want to change) the log driver for the standard containers necessary to run Kubernetes.

There you go. Please make this happen.

[xmik]

Another idea is, where all the containers still forward logs into the same endpoint, but you can at least set different fields values for different docker containers on your log server.

This would work for the gelf docker driver, if we could ensure docker containers created by Kubernetes are custom labelled. Meaning: some of a Pod fields could be forwarded as docker container labels. (Maybe this is already possible but I don't know how to achieve that).

Example without Kubernetes, only with docker daemon and gelf driver. Have docker daemon configured with: --log-driver=gelf --log-opt labels=env,label2 and create a docker container:

docker run -dti --label env=testing --label label2=some_value alpine:3.4 /bin/sh -c "while true; do date; sleep 2; done"

and another docker container:

docker run -dti --label env=production --label label2=some_value alpine:3.4 /bin/sh -c "while true; do date; sleep 2; done"

This way, on Graylog, you can differenciate between env=production and env=testing containers.

Currently I use such docker daemon options:

--log-driver=gelf --log-opt gelf-address=udp://graylog.example.com:12201 --log-opt tag=k8s-testing --log-opt labels=io.kubernetes.pod.namespace,io.kubernetes.container.name,io.kubernetes.pod.name

[k8s-ci-robot]

@yingsimai: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ejemba]

/reopen

[k8s-ci-robot]

@ejemba: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sudeep-ib]

This probably needs to be re-opened :)

[bithavoc]

/reopen

[k8s-ci-robot]

@bithavoc: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MoSunDay]

/reopen

[k8s-ci-robot]

@MoSunDay: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ejemba]

/reopen

[k8s-ci-robot]

@ejemba: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Janevski]

/reopen

So, it seems that Kubernetes is shit compared to Docker swarm.

[k8s-ci-robot]

@Janevski: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

So, it seems that Kubernetes is shit compared to Docker swarm.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bhautik0110]

Is that any update?

[ushuz]

Can we just tell the bot to leave this issue alone? cc @ejemba

[ejemba]

/reopen

@ushuz I don't know if it is possible to do this. I don't know if I have any right on it.

[k8s-ci-robot]

@ejemba: Reopened this issue.

In response to this:

/reopen

@ushuz I don't know if it is possible to do this. I don't know if I have any right on it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ushuz]

@ejemba In that case, maybe you can remove the lifecycle/rotten label with /remove-lifecycle rotten, to make the issue stay open for longer, thanks. Or maybe remove the needs-triage label?

#15478 (comment)

[ejemba]

/remove-lifecycle rotten

[thockin]

This issue is no longer relevant since we have CRI and not just docker. If we want to consider log params in CRI, we should open a new issue.