[kubelet.service] Add the mounting of /var/log. #258
Conversation
|
@jordanyaker I had fixed this previously: #164 I noticed today that apparently this change got lost when we moved to Do we need all of |
|
Additionally, |
|
@colemickens I'm not 100% about |
|
I was really doing this to try and get Deis working on Azure. deis/logger#50 specifically references |
| @@ -6,6 +6,7 @@ Documentation=https://github.com/kubernetes/kubernetes | |||
| ExecStartPre=/bin/mkdir -p /var/lib/kubelet | |||
| ExecStartPre=/bin/mount --bind /var/lib/kubelet /var/lib/kubelet | |||
| ExecStartPre=/bin/mount --make-shared /var/lib/kubelet | |||
| ExecStartPre=/bin/mkdir -p /var/log/containers | |||
There was a problem hiding this comment.
This isn't needed, docker will create dirs ahead of time for you:
$ docker run -it -v /home/cole/path/does/not/exist:/blah busybox
/ # exitThere was a problem hiding this comment.
Gotcha. I was unaware of that. I added and removed that line probably about half a dozen times during my troubleshooting.
| @@ -15,6 +16,7 @@ ExecStart=/usr/bin/docker run \ | |||
| -v /var/run:/var/run:rw \ | |||
| -v /var/lib/docker/:/var/lib/docker:rw \ | |||
| -v /var/lib/kubelet/:/var/lib/kubelet:shared \ | |||
| -v /var/log:/var/log:shared \ | |||
There was a problem hiding this comment.
Can you go ahead and change this to /var/log/containers:/var/log/containers:rw so that we're consistent with kube-deploy?
There was a problem hiding this comment.
Just saw your additional comment... And this comment from @chancez deis/logger#50 (comment)
@mikedanese What do you think? Sounds like we want to mount /var/log rather than /var/log/containers possibly.
There was a problem hiding this comment.
FYI referring to this: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image/td-agent.conf#L113
Those are the position files, which keep track of how far into a particular log file fluentd has ingested. If the container was deleted (ie for an upgrade) these files would be lost if they're not being stored on the host via a volume mount, resulting in fluentd re-ingesting logs that have already been ingested previously.
There was a problem hiding this comment.
Thanks @chancez for elaborating (it made me completely convinced). Can you confirm if is it okay to leave it mounted as shared?
There was a problem hiding this comment.
Yup, @chancez convinced me. If you remove the superfluous line above (and my current test deployment looks okay) then I'll merge it.
There was a problem hiding this comment.
Here's my question about that: does anything else create the /var/log/containers folder? If we're only sharing /var/log, doesn't there stand a chance that we will need to first create the containers folder? Or, is there another component that will check and create that?
There was a problem hiding this comment.
I think kubelet is smart enough to create that directory but I can't be sure.
There was a problem hiding this comment.
Yes, something is smart enough to make it. I merged this PR locally (but without the ExecStartPre line) and deployed a cluster. Looks like logs are flowing into ES/Kibana now as expected and /var/log/containers was created on the nodes.
There was a problem hiding this comment.
Good enough for me. I've pushed the changes with the removal of that line.
|
I've pushed up the |
|
Glad to be of assistance! |
The mounting of
/var/logenables necessary log sharing when Fluentd is in play.