New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement journalctl to see journald data within a docker container #10994

Closed
wants to merge 5 commits into
base: master
from

Conversation

Projects
None yet
9 participants
@rhatdan
Contributor

rhatdan commented Feb 24, 2015

The basic idea is to allow us to run systemd/journald within a container and have the journal data exposed to the host.

In order to do this, we had to implement three different things.

  1. Define a container_uuid environment variable in a systemd with a UUID. We do this using the container ID.(Patch 1)
  2. Create a directory on the host in /var/log/journald/CONTAINER.ID (Truncated to 32 chars), volume mount this directory into the container.
  3. Register the container with systemd using machinectl over systemd dbus.

With this patch you can do a
journalctl -M CONTAINER_NAME
And see the journal data within a systemd based container.

This has a secondary effect of allowing us to see and manipulate docker containers using machinectl. This will list all running containers, even if they are docker containers or systemd-nspawn containers. It will also list any VM's running on the machine.

@rhatdan rhatdan changed the title from This pull request implements journald within a docker container to Implement journalctl to see journald data within a docker container Feb 24, 2015

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Feb 24, 2015

Contributor

Replaces

#7685

Contributor

rhatdan commented Feb 24, 2015

Replaces

#7685

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Feb 24, 2015

Contributor

@rhatdan Can you provide image to test this patch?

Contributor

LK4D4 commented Feb 24, 2015

@rhatdan Can you provide image to test this patch?

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Feb 25, 2015

Contributor

You could build an image based on this Dockerfile to test.

#
# See the top level Makefile in https://github.com/docker/docker for usage.
#
FROM        fedora
MAINTAINER  Dan Walsh
ENV container docker
RUN yum -y update; yum -y install httpd; yum clean all; rm -f /etc/machine-id; systemctl enable httpd

EXPOSE 80
RUN 

CMD [ "/sbin/init" ]

Something like this should work, on a systemd based machine.

docker build -t test .
docker run -d test -n test
journalctl -M test

But even without the systemd within the journal you can see any container running using machinectl
machinectl list

And try out some of the other commands. machinectl kill does not seem to work for me, at least when I am running systemd as PID 1.

Contributor

rhatdan commented Feb 25, 2015

You could build an image based on this Dockerfile to test.

#
# See the top level Makefile in https://github.com/docker/docker for usage.
#
FROM        fedora
MAINTAINER  Dan Walsh
ENV container docker
RUN yum -y update; yum -y install httpd; yum clean all; rm -f /etc/machine-id; systemctl enable httpd

EXPOSE 80
RUN 

CMD [ "/sbin/init" ]

Something like this should work, on a systemd based machine.

docker build -t test .
docker run -d test -n test
journalctl -M test

But even without the systemd within the journal you can see any container running using machinectl
machinectl list

And try out some of the other commands. machinectl kill does not seem to work for me, at least when I am running systemd as PID 1.

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Feb 25, 2015

Contributor

BTW I have submitted a pull request to coreos for go-systemd.

coreos/go-systemd#75

Contributor

rhatdan commented Feb 25, 2015

BTW I have submitted a pull request to coreos for go-systemd.

coreos/go-systemd#75

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Feb 25, 2015

Contributor

@rhatdan thank you!

Contributor

LK4D4 commented Feb 25, 2015

@rhatdan thank you!

Show outdated Hide outdated docs/man/docker-run.1.md Outdated
@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Mar 17, 2015

Contributor

Seems like go-systemd upstream stuck :/

Contributor

LK4D4 commented Mar 17, 2015

Seems like go-systemd upstream stuck :/

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Mar 17, 2015

Contributor

I dropped the go-systemd part of the patch and went with a simpler dbus implementation. Eliminated a lot of code from the patch. Also fixed a resource leak.

This patch should be ready for review now.

@jfrazelle @crosbymichael @LK4D4

Contributor

rhatdan commented Mar 17, 2015

I dropped the go-systemd part of the patch and went with a simpler dbus implementation. Eliminated a lot of code from the patch. Also fixed a resource leak.

This patch should be ready for review now.

@jfrazelle @crosbymichael @LK4D4

@ronin13

This comment has been minimized.

Show comment
Hide comment
@ronin13

ronin13 Apr 1, 2015

This should certainly help. Currently I pass /dev/log as a volume (-v /dev/log:/dev/log) to make the container log to host's journald and then use journalctl to filter from that based on syslog tag used inside each container.

ronin13 commented Apr 1, 2015

This should certainly help. Currently I pass /dev/log as a volume (-v /dev/log:/dev/log) to make the container log to host's journald and then use journalctl to filter from that based on syslog tag used inside each container.

@@ -166,6 +166,9 @@ is the case the **--dns** flags is necessary for every run.
environment variables that are available for the process that will be launched
inside of the container.
The container_uuid is set automatically with a 32 character truncated

This comment has been minimized.

@jamtur01

jamtur01 Apr 1, 2015

Contributor

Should this be mentioned in the other run docs?

@jamtur01

jamtur01 Apr 1, 2015

Contributor

Should this be mentioned in the other run docs?

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 1, 2015

Contributor

@rhatdan Hmm, maybe I'm doing something wrong, but I get

journalctl -M test
Failed to open journal: Host is down

from your example. Could you recheck it pls?
This is probably because of -n is not found for me. Also I just run container with /sbin/init and there is no /etc/machine-id in it.

Contributor

LK4D4 commented Apr 1, 2015

@rhatdan Hmm, maybe I'm doing something wrong, but I get

journalctl -M test
Failed to open journal: Host is down

from your example. Could you recheck it pls?
This is probably because of -n is not found for me. Also I just run container with /sbin/init and there is no /etc/machine-id in it.

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Apr 7, 2015

Contributor

@LK4D4 Try it now. I added the /run patch to stop you from having to run --privileged.

These patches are working for me with

cat Dockerfile

FROM        fedora
MAINTAINER  Dan Walsh
ENV container docker
RUN yum -y update; yum -y install httpd; yum clean all; systemctl enable httpd

EXPOSE 80
RUN rm -f /etc/machine-id 

CMD [ "/sbin/init" ]

docker build -t systemd-httpd .
docker run -d --name dan -v /sys/fs/cgroup:/sys/fs/cgroup systemd-httpd
journalctl -M dan

Contributor

rhatdan commented Apr 7, 2015

@LK4D4 Try it now. I added the /run patch to stop you from having to run --privileged.

These patches are working for me with

cat Dockerfile

FROM        fedora
MAINTAINER  Dan Walsh
ENV container docker
RUN yum -y update; yum -y install httpd; yum clean all; systemctl enable httpd

EXPOSE 80
RUN rm -f /etc/machine-id 

CMD [ "/sbin/init" ]

docker build -t systemd-httpd .
docker run -d --name dan -v /sys/fs/cgroup:/sys/fs/cgroup systemd-httpd
journalctl -M dan

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 7, 2015

Contributor

Okay, it is failing on CI now :) Seems like /run patch did that.

Contributor

LK4D4 commented Apr 7, 2015

Okay, it is failing on CI now :) Seems like /run patch did that.

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Apr 8, 2015

Contributor

Yes, /run patch requires a tar be installed into the testing container. I can remove this patch and you would just need to run --privileged so systemd will create its own /run on tmpfs.

Contributor

rhatdan commented Apr 8, 2015

Yes, /run patch requires a tar be installed into the testing container. I can remove this patch and you would just need to run --privileged so systemd will create its own /run on tmpfs.

@dmcgowan

This comment has been minimized.

Show comment
Hide comment
@dmcgowan

dmcgowan Apr 21, 2015

Member

@rhatdan what logs should I expect to see in journalctl output?

I see systemd-machined[19777]: New machine dan. but then nothing from the container.

Running on Fedora 21

Member

dmcgowan commented Apr 21, 2015

@rhatdan what logs should I expect to see in journalctl output?

I see systemd-machined[19777]: New machine dan. but then nothing from the container.

Running on Fedora 21

av []byte
err error
)
if !SdBooted() {

This comment has been minimized.

@calavera

calavera Apr 21, 2015

Contributor

should we move this check outside of the library to be explicit about the fact that we don't run this when systemd is not running?

@calavera

calavera Apr 21, 2015

Contributor

should we move this check outside of the library to be explicit about the fact that we don't run this when systemd is not running?

This comment has been minimized.

@rhatdan

rhatdan Apr 22, 2015

Contributor

Either way, I am fine with it, just dirties up the code in the main line.

@rhatdan

rhatdan Apr 22, 2015

Contributor

Either way, I am fine with it, just dirties up the code in the main line.

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Apr 22, 2015

Contributor

@dmcgowan did you successfully get systemd/journald to run inside the container, if you do then you should see

journalctl -M NAME

give you the same data as

docker exec NAME journalct

Contributor

rhatdan commented Apr 22, 2015

@dmcgowan did you successfully get systemd/journald to run inside the container, if you do then you should see

journalctl -M NAME

give you the same data as

docker exec NAME journalct

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 22, 2015

Contributor

@rhatdan I see machine in machinectl list. But journalctl -M dan returns whole journal since boot for me and there is only:

systemd-machined[13207]: New machine dan.

there about machine. Also exec returns:

No journal files were found.

My host systemd is 219-r2. But I don't think it's an issue.

Contributor

LK4D4 commented Apr 22, 2015

@rhatdan I see machine in machinectl list. But journalctl -M dan returns whole journal since boot for me and there is only:

systemd-machined[13207]: New machine dan.

there about machine. Also exec returns:

No journal files were found.

My host systemd is 219-r2. But I don't think it's an issue.

@GordonTheTurtle GordonTheTurtle removed the dco/no label Apr 22, 2015

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Apr 22, 2015

Contributor

@LK4D4 Is there an /etc/machine-id in your base image?

Contributor

rhatdan commented Apr 22, 2015

@LK4D4 Is there an /etc/machine-id in your base image?

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 22, 2015

Contributor

@rhatdan Nope :/

Contributor

LK4D4 commented Apr 22, 2015

@rhatdan Nope :/

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Apr 22, 2015

Contributor

When you run the container does a /etc/machine-id get created along with a /var/log/journal/CONTAINER-ID?
Is journald running in the container

ps -ef | grep journald

Contributor

rhatdan commented Apr 22, 2015

When you run the container does a /etc/machine-id get created along with a /var/log/journal/CONTAINER-ID?
Is journald running in the container

ps -ef | grep journald

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 22, 2015

Contributor

@rhatdan Whoa. I ran your image with --privileged and it killed my X server and showed me tty greeting from fedora on all consoles apart from first. That was very scary.

Contributor

LK4D4 commented Apr 22, 2015

@rhatdan Whoa. I ran your image with --privileged and it killed my X server and showed me tty greeting from fedora on all consoles apart from first. That was very scary.

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 22, 2015

Contributor

@rhatdan Without privileged I see only /sbin/init in container.

Contributor

LK4D4 commented Apr 22, 2015

@rhatdan Without privileged I see only /sbin/init in container.

@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan Apr 22, 2015

Contributor

Well to get this stuff to fully work you need to volume mount in /run and /sys/fs/cgroup:ro

We have a separate patch that mounts /run as a tmpfs/ systemd will not run in a container properly without these two things, or in --privileged mode. Not sure why it killed X.

Contributor

rhatdan commented Apr 22, 2015

Well to get this stuff to fully work you need to volume mount in /run and /sys/fs/cgroup:ro

We have a separate patch that mounts /run as a tmpfs/ systemd will not run in a container properly without these two things, or in --privileged mode. Not sure why it killed X.

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 22, 2015

Contributor

So, actually we have problem. This can't work with current docker, apart from --privileged which I suspect I won't run again.
ping @crosbymichael your thoughts

Contributor

LK4D4 commented Apr 22, 2015

So, actually we have problem. This can't work with current docker, apart from --privileged which I suspect I won't run again.
ping @crosbymichael your thoughts

@GordonTheTurtle GordonTheTurtle removed the dco/no label Apr 23, 2015

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 May 5, 2015

Contributor

So, this PR is pretty old. We need to do something with it.
ping @docker/core-maintainers
Should we split preMount and postMount cmds to separate PR or what?

Contributor

LK4D4 commented May 5, 2015

So, this PR is pretty old. We need to do something with it.
ping @docker/core-maintainers
Should we split preMount and postMount cmds to separate PR or what?

rhatdan added some commits Apr 22, 2015

Set container_uuid environment
If you are running with systemd as init 1 and specify the container_uuid environment
variable, systemd will create /etc/machine-id with the correct data.

Then on the host you can setup journald to watch the container.

systemd only allows 32 chars in the UUID stored in /etc/machine-id

Then we can later apply a different patch to setup journald to watch
containers from the host and log all syslog/stdout/stderr data together
int the hosts journal.

The following link explains what systemd expects to be setup.

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/

This should replace

#3506

Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
Mount journal directory from the host into the container.
If you run journald within the container the host journalctl
will be able to display the content.

Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
Tar up contents of child directory onto tmpfs if mounted over
This patch will use the new PreMount and PostMount hooks to "tar"
up the contents of the base image on top of tmpfs mount points.

Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)

Conflicts:
	daemon/execdriver/native/create.go
Have docker register its machine with systemd
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
New implementation of /run support
    This mounts a /run tmpfs into the container, with the initial contents
    copies from the /run in the base image, unless MountRun is set to false
    in the HostConfig.

    Additionally BuildFlag is always set to true during a docker build, which i
    means any setup of /run in a Dockerfile is saved in the image to be copied
    into the final /run tmpfs when a container is started.

    Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)

Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)

Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
@rhatdan

This comment has been minimized.

Show comment
Hide comment
@rhatdan

rhatdan May 28, 2015

Contributor

Replacing this pull request with --systemd pull request

#13525

Contributor

rhatdan commented May 28, 2015

Replacing this pull request with --systemd pull request

#13525

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment