log driver - Interpolate fields into log tag #15384

phil-monroe · 2015-08-06T23:11:25Z

Substitute variables about the context of the container into the syslog tag. The following go templates are added:

{{.ID}} - The first 12 characters of the container id.
{{.FullID}} - The full id of the container.
{{.Name}} - The name of the container.
{{.ImageID}} - The first 12 characters of the image id used for the container.
{{.ImageFullID}} - The full id of the image used for the container.
{{.ImageName}} - The name of the image used for the container.

example:

docker run --log-driver=syslog --log-opt syslog-tag="{{.ImageName}}/{{.Name}}/{{.ID}}" --name foobar hello-world

output in syslog:

Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]: Hello from Docker.
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]: This message shows that your installation appears to be working correctly.
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]: To generate this message, Docker took the following steps:
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:  1. The Docker client contacted the Docker daemon.
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:  2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:     (Assuming it was not already locally available.)
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:  3. The Docker daemon created a new container from that image which runs the
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:     executable that produces the output you are currently reading.
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:  4. The Docker daemon streamed that output to the Docker client, which sent it
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:     to your terminal.
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]: To try something more ambitious, you can run an Ubuntu container with:
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:  $ docker run -it ubuntu bash
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]: For more examples and ideas, visit:
Aug  6 22:20:04 80fbe5afa9a9 docker/hello-world/foobar/c24f89499fce[952]:  http://docs.docker.com/userguide/

icecrime · 2015-08-07T15:51:59Z

Cool, seems like a good idea (and code LGTM). @LK4D4 WDYT?

phemmer · 2015-08-07T17:15:12Z

I think the idea is sound, but personally I would prefer go template formatting. We already have a few commands that use go templates, and I would prefer not to see another template format.

If the template had access to the ContainerJSON object, that'd be even better, as then the template would support the same fields as docker inspect. But this is probably too complex.

ghost · 2015-08-07T17:21:38Z

There is an opened pull request to address the same issue for the fluentd driver, and it uses go templates.

cpuguy83 · 2015-08-07T18:23:49Z

+1 for go-template if we do this.

phil-monroe · 2015-08-07T18:56:33Z

Updated to follow the go template convention and include documentation.

icecrime · 2015-08-07T21:36:04Z

@phemmer Good call thanks. Moving to code review.

thaJeztah · 2015-08-07T21:57:25Z

daemon/logger/syslog/syslog.go

@@ -167,3 +178,31 @@ func parseFacility(facility string) (syslog.Priority, error) {

 	return syslog.Priority(0), errors.New("invalid syslog facility")
 }
+
+func parseTag(ctx logger.Context) (string, error) {
+	tag := ctx.Config["syslog-tag"]


Would it make sense to move this (and setting the default (line 194)) outside of this function; I.e. pass a template and the "tag" and return the result. This would open up the possibility to re-use this for other log drivers at some point.

(Thinking out loud here :-))

So something like:

func parseTag(ctx logger.Context, tagTemplate string, defaultTagTemplate string) (string, error) { if tagTemplate == "" { tagTemplate = defaultTagTemplate } //... }

and would be used like so:

tag, err := parseTag(ctx, ctx.Config["syslog-tag"], "{{.ID}}")

Yes, something along that line. I'm not really a code reviewer here 😊, but I expect that other log-drivers may want to use templating as well, so having something more reusable would be great.

Perhaps some other people have suggestions here (@LK4D4 ?)

+1 to something reusable.

phil-monroe · 2015-08-10T18:01:16Z

Updated to make the code hopefully more reusable.

phil-monroe · 2015-08-10T21:02:52Z

Just pushed an update to namespace the helper method as to not clutter logger.go and add some unit specs.

phil-monroe · 2015-08-11T00:45:11Z

Do those build failures seem like they relate to this or should just be retried?

thaJeztah · 2015-08-13T08:58:12Z

@phil-monroe I triggered a rebuild; let's see 👍

phil-monroe · 2015-08-13T20:42:36Z

@thaJeztah seems like things passed.

@icecrime / @cpuguy83 / @phemmer / @eolamey - Any other comments for the code?

cpuguy83 · 2015-08-17T21:09:48Z

@phil-monroe Have you considered using the existing Context struct and adding methods to get the fields we want to support to that directly?

mariussturm · 2015-08-21T13:42:53Z

This should also be consistent for all logging drivers which support tags, not only the syslog-driver.

calavera · 2015-08-24T09:16:11Z

I agree with @mariussturm. @phil-monroe can you make changes in the other drivers too?

phil-monroe · 2015-08-25T02:53:49Z

Sorry for the delay, I've been on vacation. I'll get the changes in here soon.

@mariussturm, @calavera - Sure thing.

@cpuguy83 - For some reason I thought the logger.Context was crucial to the container and was worried that someone could exploit the function capabilities of go templates to do something dangerous. On second look, I don't think that there should be any real security concern from giving raw access to the
logger.Context. Do you (or anyone) think that there could be a security issue with that? I agree that helper methods on that struct would be better than a separate context if there aren't any concerns there. I'm probably being overly sensitive here as I'm new to go and the docker codebase.

phil-monroe · 2015-08-28T21:41:04Z

Updated the documentation. Let me know what you think.

thaJeztah · 2015-08-28T23:57:49Z

docs/reference/logging/overview.md

-The `syslog-tag` specifies a tag that identifies the container's syslog messages. By default,
-the system uses the first 12 characters of the container id. To override this behavior, specify
-a `syslog-tag` option
+See the [log tag option](/reference/logging/log_tags/) documentation on how to format log tags.


I think we should give slightly more context to this link. Something like this;

By default, Docker uses the first 12 characters of the container ID to tag log messages. Refer to the [log tag option documentation](/reference/logging/log_tags/) for customizing the log tag format.

thaJeztah · 2015-08-29T00:10:14Z

Added some suggestions for the docs; given that we're deprecating the driver specific "tag" options, can you add that to the deprecated features document. It should mention that the syslog-tag, gelf-tag and fluentd-tag log options are deprecated in docker 1.9, and will be removed in docker 1.11 (we try to keep deprecated options around for 2 releases)

Thanks @phil-monroe ! Great work altogether

ping @moxiegirl

thaJeztah · 2015-08-29T00:12:24Z

Oh, and ping @albers because the completion scripts will need to be updated ❤️

thaJeztah · 2015-08-29T00:38:47Z

Almost forgot; ping @sdurrheimer for zsh completions, once this is merged 👍

albers · 2015-08-29T06:22:20Z

I'll add this to bash completion. Thanks for pinging me, @thaJeztah.

moxiegirl · 2015-08-29T20:34:40Z

docs/reference/logging/log_tags.md

@@ -0,0 +1,39 @@
+<!--[metadata]>


@phil-monroe Thanks for the contribution! Nicely done. Can you add a link to this file in the reference/logging/index.md page? Some tweaks to this page, table works better for reference also I think better to show a simple use oftag first then use of the templates.

--> https://gist.github.com/moxiegirl/5d5fbd9bd282c3dce3bd

Log Tags

The tag log option specifies how to format a tag that identifies the
container's log messages. By default, the system uses the first 12 characters of
the container id. To override this behavior, specify a tag option:

docker run --log-driver=fluentd --log-opt fluentd-address=myhost.local:24224 --log-opt tag="mailer"

Docker supports some special template markup you can use when specifying a tag's value:

Markup Description

{{.ID}} The first 12 characters of the container id.

{{.FullID}} The full container id.

{{.Name}} The container name.

{{.ImageID}} The first 12 characters of the container's image id.

{{.ImageFullID}} The container's full image identifier.

{{.ImageName}} The name of the image used by the container.

For example, specifying a --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" value yields syslog log lines like:

Aug 7 18:33:19 HOSTNAME docker/hello-world/foobar/5790672ab6a0[9103]: Hello from Docker.

At startup time, the system sets the container_name field and {{.Name}} in
the tags. If you use docker rename to rename a container, the new name is not
reflected in the log messages. Instead, these messages continue to use the
original container name.

For advanced usage, the generated tag's use go
templates and the container's logging
context.

Note:The driver specific log options syslog-tag, fluentd-tag and
gelf-tag still work for backwards compatibility. However, going forward you
should standardize on using the generic tag log option instead.

cpuguy83 · 2015-09-15T16:17:29Z

Ping, any chance you can update the docs?

moxiegirl · 2015-09-15T17:00:51Z

@cpuguy83 if necessary, I can carry. Just an FYI

phil-monroe · 2015-09-16T16:31:34Z

Sorry for the delay! Crazy couple of weeks. I'll try and finish them off
today.

On Tuesday, September 15, 2015, moxiegirl notifications@github.com wrote:

@cpuguy83 https://github.com/cpuguy83 if necessary, I can carry. Just
an FYI

—
Reply to this email directly or view it on GitHub
#15384 (comment).

Phil Monroe

cpuguy83 · 2015-09-16T16:32:24Z

Thanks!

…g tag field Signed-off-by: Philip Monroe <phil@philmonroe.com>

phil-monroe · 2015-09-16T22:21:19Z

Documentation updated. Sorry for the radio silence!

moxiegirl · 2015-09-17T02:40:30Z

@phil-monroe Thanks! LGTM

…og-tag log driver - Interpolate fields into log tag

GordonTheTurtle added status/0-triage dco/no Automatically set by a bot when one of the commits lacks proper signature labels Aug 6, 2015

phil-monroe force-pushed the 15058-include-name-in-syslog-tag branch from d63e5fa to ac7fbcf Compare August 6, 2015 23:13

GordonTheTurtle removed the dco/no Automatically set by a bot when one of the commits lacks proper signature label Aug 6, 2015

phil-monroe force-pushed the 15058-include-name-in-syslog-tag branch from ac7fbcf to 4156f3f Compare August 6, 2015 23:24

icecrime added status/1-design-review and removed status/0-triage labels Aug 7, 2015

phil-monroe force-pushed the 15058-include-name-in-syslog-tag branch from 4156f3f to 0744010 Compare August 7, 2015 18:54

icecrime added status/2-code-review and removed status/1-design-review labels Aug 7, 2015

thaJeztah reviewed Aug 7, 2015
View reviewed changes

phil-monroe force-pushed the 15058-include-name-in-syslog-tag branch 2 times, most recently from c7ff302 to b0c4a05 Compare August 10, 2015 17:59

phil-monroe force-pushed the 15058-include-name-in-syslog-tag branch from b0c4a05 to ed2e260 Compare August 10, 2015 21:01

calavera added the impact/changelog label Aug 24, 2015

thaJeztah added status/3-docs-review and removed status/2-code-review labels Aug 28, 2015

thaJeztah reviewed Aug 28, 2015
View reviewed changes

thaJeztah added the impact/deprecation label Aug 29, 2015

sdurrheimer mentioned this pull request Aug 29, 2015

Zsh completion updates #15915

Merged

3 tasks

moxiegirl reviewed Aug 29, 2015
View reviewed changes

thaJeztah mentioned this pull request Aug 30, 2015

Use custom container name in logs docker/compose#1862

Merged

cpuguy83 mentioned this pull request Sep 1, 2015

Add extra fields based on label and env for gelf/fluentd/json-file/journald log drivers #15975

Closed

log driver - add ability to interpolate container context into the lo…

3be7146

…g tag field Signed-off-by: Philip Monroe <phil@philmonroe.com>

phil-monroe force-pushed the 15058-include-name-in-syslog-tag branch from 3b12b99 to 3be7146 Compare September 16, 2015 22:20

moxiegirl added status/4-merge and removed status/3-docs-review labels Sep 17, 2015

cpuguy83 added a commit that referenced this pull request Sep 17, 2015

Merge pull request #15384 from phil-monroe/15058-include-name-in-sysl…

4dfa996

…og-tag log driver - Interpolate fields into log tag

cpuguy83 merged commit 4dfa996 into moby:master Sep 17, 2015

albers mentioned this pull request Sep 18, 2015

bash completion: use generic tag log driver option #16400

Merged

thaJeztah mentioned this pull request Jan 7, 2016

aws cloudwatch logs awslogs-stream markup #19158

Closed

matthewdfuller mentioned this pull request Feb 18, 2016

Journald logging does not accept the "tag" or "syslog-tag" options #20443

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log driver - Interpolate fields into log tag #15384

log driver - Interpolate fields into log tag #15384

phil-monroe commented Aug 6, 2015

icecrime commented Aug 7, 2015

phemmer commented Aug 7, 2015

ghost commented Aug 7, 2015

cpuguy83 commented Aug 7, 2015

phil-monroe commented Aug 7, 2015

icecrime commented Aug 7, 2015

thaJeztah Aug 7, 2015

phil-monroe Aug 7, 2015

thaJeztah Aug 7, 2015

izaakschroeder Aug 8, 2015

phil-monroe commented Aug 10, 2015

phil-monroe commented Aug 10, 2015

phil-monroe commented Aug 11, 2015

thaJeztah commented Aug 13, 2015

phil-monroe commented Aug 13, 2015

cpuguy83 commented Aug 17, 2015

mariussturm commented Aug 21, 2015

calavera commented Aug 24, 2015

phil-monroe commented Aug 25, 2015

phil-monroe commented Aug 28, 2015

thaJeztah Aug 28, 2015

thaJeztah commented Aug 29, 2015

thaJeztah commented Aug 29, 2015

thaJeztah commented Aug 29, 2015

albers commented Aug 29, 2015

moxiegirl Aug 29, 2015

cpuguy83 commented Sep 15, 2015

moxiegirl commented Sep 15, 2015

phil-monroe commented Sep 16, 2015

cpuguy83 commented Sep 16, 2015

phil-monroe commented Sep 16, 2015

moxiegirl commented Sep 17, 2015

Markup	Description
`{{.ID}}`	The first 12 characters of the container id.
`{{.FullID}}`	The full container id.
`{{.Name}}`	The container name.
`{{.ImageID}}`	The first 12 characters of the container's image id.
`{{.ImageFullID}}`	The container's full image identifier.
`{{.ImageName}}`	The name of the image used by the container.

log driver - Interpolate fields into log tag #15384

log driver - Interpolate fields into log tag #15384

Conversation

phil-monroe commented Aug 6, 2015

icecrime commented Aug 7, 2015

phemmer commented Aug 7, 2015

ghost commented Aug 7, 2015

cpuguy83 commented Aug 7, 2015

phil-monroe commented Aug 7, 2015

icecrime commented Aug 7, 2015

thaJeztah Aug 7, 2015

Choose a reason for hiding this comment

phil-monroe Aug 7, 2015

Choose a reason for hiding this comment

thaJeztah Aug 7, 2015

Choose a reason for hiding this comment

izaakschroeder Aug 8, 2015

Choose a reason for hiding this comment

phil-monroe commented Aug 10, 2015

phil-monroe commented Aug 10, 2015

phil-monroe commented Aug 11, 2015

thaJeztah commented Aug 13, 2015

phil-monroe commented Aug 13, 2015

cpuguy83 commented Aug 17, 2015

mariussturm commented Aug 21, 2015

calavera commented Aug 24, 2015

phil-monroe commented Aug 25, 2015

phil-monroe commented Aug 28, 2015

thaJeztah Aug 28, 2015

Choose a reason for hiding this comment

thaJeztah commented Aug 29, 2015

thaJeztah commented Aug 29, 2015

thaJeztah commented Aug 29, 2015

albers commented Aug 29, 2015

moxiegirl Aug 29, 2015

Choose a reason for hiding this comment

Log Tags

cpuguy83 commented Sep 15, 2015

moxiegirl commented Sep 15, 2015

phil-monroe commented Sep 16, 2015

cpuguy83 commented Sep 16, 2015

phil-monroe commented Sep 16, 2015

moxiegirl commented Sep 17, 2015