Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "local" log driver #37092

Merged
merged 4 commits into from Aug 20, 2018

Conversation

@cpuguy83
Copy link
Contributor

cpuguy83 commented May 17, 2018

This adds the "local" log driver described in #33475.

@@ -81,9 +85,13 @@ func decodeFunc(rdr io.Reader) func() (*logger.Message, error) {
if err == io.ErrUnexpectedEOF {
reader := io.MultiReader(dec.Buffered(), rdr)
dec = json.NewDecoder(reader)
retries++
continue

This comment has been minimized.

@anusha-ragunathan

anusha-ragunathan May 17, 2018

Contributor

Why is retries not incremented? Both here and L73?

This comment has been minimized.

@cpuguy83

cpuguy83 May 17, 2018

Author Contributor

These were erroneous. The for loop already increments retries.

func TestTailFiles(t *testing.T) {
s1 := strings.NewReader("Hello.\nMy name is Inigo Montoya.\n")
s2 := strings.NewReader("I'm serious.\nDon't call me Shirley!\n")
s3 := strings.NewReader("Roads?\nWhere we're going we don't need roads.\n")

This comment has been minimized.

@anusha-ragunathan

anusha-ragunathan May 17, 2018

Contributor

We should add s4 with Why does it say paper jam when there is no paper jam?.

@anusha-ragunathan

This comment has been minimized.

Copy link
Contributor

anusha-ragunathan commented May 17, 2018

Overall LGTM.

Needs a followup doc PR

@@ -107,7 +108,10 @@ func BenchmarkJSONFileLoggerLog(b *testing.B) {
ContainerID: "a7317399f3f857173c6179d44823594f8294678dea9999662e5c625b5a1c7657",
LogPath: tmp.Join("container.log"),
Config: map[string]string{
"labels": "first,second",
"labels": "first,second",
"max-file": "10",

This comment has been minimized.

@thaJeztah

thaJeztah May 26, 2018

Member

IIUC, the intent of this logging driver is to be an "optimised, general-purpose logging driver" (which could become the default).

The storage mechanisms for this driver (currently, file-based, file-format-x) is an implementation detail. Because of that, should we abstract these options? i.e., have options for:

  • "maximum amount of logs to keep" (the driver can decide to use multiple files to optimize)
  • "maximum age of logs to keep" (in light of GDPR #36853)
  • compression; good one: is there a significant overhead? are there situations where you want to disable this?

This comment has been minimized.

@cpuguy83

cpuguy83 Jul 6, 2018

Author Contributor

We discussed this on the maintainers meeting and decided to keep using the same options that are available for jsonfile. Others can be added later.

@@ -0,0 +1,9 @@
// Package local provides a logger implementation that stores logs on disk.
//
// Log messages are encoded as protobufs with a header and footer for each message.

This comment has been minimized.

@thaJeztah

thaJeztah May 26, 2018

Member

Q: If I understand correctly, we want the storage-format to be an implementation detail; if we ever decide to switch to/add a new storage format, would there be a need to store the format in the container's configuration?

This comment has been minimized.

@cpuguy83

cpuguy83 May 30, 2018

Author Contributor

This is covered a bit in the propsoal issue.
Basically we'd store the format details in the header of the log file.

I started to go ahead and add support in logfile for setting headers, but it's a significant change, and technically not needed for the first format since we can detect that no format is specified and therefore it's the first format.

This comment has been minimized.

@thaJeztah

thaJeztah May 30, 2018

Member

technically not needed for the first format since we can detect that no format is specified and therefore it's the first format.

Makes sense, and yes, that would work

@cpuguy83

This comment has been minimized.

Copy link
Contributor Author

cpuguy83 commented Jul 6, 2018

Moved this to code review per discussion with @thaJeztah on #maintainers in Slack.

@vdemeester
Copy link
Member

vdemeester left a comment

SGTM 🐫

if err := os.MkdirAll(logDir, 0700); err != nil {
return nil, errdefs.System(errors.Wrap(err, "error creating local logs dir"))
}
info.LogPath = filepath.Join(logDir, "container.log")

This comment has been minimized.

@vdemeester

vdemeester Jul 9, 2018

Member

why don't we set container.LogPath here too ? 🤔

This comment has been minimized.

@cpuguy83

cpuguy83 Jul 9, 2018

Author Contributor

We don't want to expose this to users.

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

Can you add a comment about that? Just in case someone decides to add it, and doesn't know about this history.

@thaJeztah
Copy link
Member

thaJeztah left a comment

Finally able to give this another look; found some issues in tests, and some possible issues around max-file validation/checks


"bufio"

"github.com/gotestyourself/gotestyourself/assert"

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

This needs to be updated to gotest.tools

@@ -309,33 +321,45 @@ func (w *LogFile) ReadLogs(config logger.ReadConfig, watcher *logger.LogWatcher)
}

if config.Tail != 0 {
// TODO(@cpuguy83): Instead of opening every file, only get the files which

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

We can make a tracking-issue for this after this is merged


"github.com/docker/docker/daemon/logger"
"github.com/docker/docker/pkg/tailfile"
"github.com/gotestyourself/gotestyourself/assert"

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

This should now use gotest.tools

if err := os.MkdirAll(logDir, 0700); err != nil {
return nil, errdefs.System(errors.Wrap(err, "error creating local logs dir"))
}
info.LogPath = filepath.Join(logDir, "container.log")

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

Can you add a comment about that? Just in case someone decides to add it, and doesn't know about this history.

return errors.New("max size should be a positive number")
}
if cfg.MaxFileCount < 0 {
return errors.New("max file count cannot be less than 0")

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

Can it be less than one?

Ok; testing; --log-opt max-file=0 is equivalent to --log-opt max-file=1;

CID=$(docker run -dit --rm --log-driver=local --log-opt max-file=0 --log-opt max-size=100k busybox /bin/sh -c 's="aasdasdad"; while true; do echo $s; done')
watch ls -lah /var/lib/docker/containers/${CID}/local-logs/

Shows that it's maxing out at 100k, then resets the file;

Every 2.0s: ls -lah /var/lib/docker/containers/57656d42e81d529cf5c2ddd097460d1bbd64530ea0690b529fbbf5b65dccfbb7/local-logs/

drwx------ 2 root root 4.0K Jul 11 12:42 .
drwx------ 5 root root 4.0K Jul 11 12:42 ..
-rw-r----- 1 root root    0 Jul 11 12:44 container.log

Every 2.0s: ls -lah /var/lib/docker/containers/57656d42e81d529cf5c2ddd097460d1bbd64530ea0690b529fbbf5b65dccfbb7/local-logs/

drwx------ 2 root root 4.0K Jul 11 12:42 .
drwx------ 5 root root 4.0K Jul 11 12:42 ..
-rw-r----- 1 root root  90K Jul 11 12:44 container.log


Every 2.0s: ls -lah /var/lib/docker/containers/57656d42e81d529cf5c2ddd097460d1bbd64530ea0690b529fbbf5b65dccfbb7/local-logs/

drwx------ 2 root root 4.0K Jul 11 12:42 .
drwx------ 5 root root 4.0K Jul 11 12:42 ..
-rw-r----- 1 root root    0 Jul 11 12:44 container.log

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

Some further testing; if no max-file is provided, the default is 4 (5?);

Every 2.0s: ls -lah /var/lib/docker/containers/dfc64e17d8fa30ea9d5328d35a28cbe38b3852d69719f12af182a1418c5ce746/local-logs/                                                            117a4e8e575a: Wed Jul 11 12:56:21 2018

total 112K
drwx------ 2 root root 4.0K Jul 11 12:56 .
drwx------ 5 root root 4.0K Jul 11 12:55 ..
-rw-r----- 1 root root  54K Jul 11 12:56 container.log
-rw-r----- 1 root root 9.0K Jul 11 12:56 container.log.1.gz
-rw-r----- 1 root root 9.0K Jul 11 12:56 container.log.2.gz
-rw-r----- 1 root root 8.8K Jul 11 12:56 container.log.3.gz
-rw-r----- 1 root root 9.1K Jul 11 12:56 container.log.4.gz

But inspecting the container won't reflect that; should we show the default?

docker inspect $CID --format '{{json .HostConfig.LogConfig}}'
{"Type":"local","Config":{"max-size":"100k"}}

This comment has been minimized.

@cpuguy83

cpuguy83 Aug 11, 2018

Author Contributor

We could... maybe adding a DefaultConfig() function to the logger interface?

This comment has been minimized.

@cpuguy83

cpuguy83 Aug 17, 2018

Author Contributor

Actually I think it would be better to not do this for now at least.
I'd prefer to not change the hostconfig on the container.

}

if !cfg.DisableCompression {
if cfg.MaxFileCount == 1 {

This comment has been minimized.

@thaJeztah

thaJeztah Jul 11, 2018

Member

Per the above; MaxFileCount == 1 looks to do exactly the same as MaxFileCount == 0, so this should probably be;

if cfg.MaxFileCount <= 1 {
@kolyshkin

This comment has been minimized.

Copy link
Contributor

kolyshkin commented Jul 26, 2018

SGTM

cpuguy83 added some commits May 17, 2018

Add partial log metadata to log driver proto
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Add func to get an io.Reader for tail operations
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Decouple logfile from tailfile.
This makes it so consumers of `LogFile` should pass in how to get an
io.Reader to the requested number of lines to tail.

This is also much more efficient when tailing a large number of lines.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>

@cpuguy83 cpuguy83 force-pushed the cpuguy83:local_logger branch from 877ca64 to 1d1c205 Aug 11, 2018

@andrewhsu

This comment has been minimized.

Copy link
Contributor

andrewhsu commented Aug 19, 2018

Previous z build failure seems infrastructure network related:

00:00:51.155 docker: Error response from daemon: Get https://registry-1.docker.io/v2/library/docker-s390x/manifests/0e10011: net/http: TLS handshake timeout.

I kicked the z builds again, but if it can't get a decent run, I propose we ignore z results. We've been having intermittent z infrastructure issues the last few days.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Aug 20, 2018

Agreed; z failure is definitely not related; everything else is green, so let's get this merged

@thaJeztah thaJeztah merged commit e0ad6d0 into moby:master Aug 20, 2018

5 of 6 checks passed

z Jenkins build Docker-PRs-s390x 10772 has failed
Details
dco-signed All commits are signed
experimental Jenkins build Docker-PRs-experimental 41690 has succeeded
Details
janky Jenkins build Docker-PRs 50447 has succeeded
Details
powerpc Jenkins build Docker-PRs-powerpc 10883 has succeeded
Details
windowsRS1 Jenkins build Docker-PRs-WoW-RS1 21754 has succeeded
Details

@thaJeztah thaJeztah removed this from backlog in maintainers-session Aug 20, 2018

@cpuguy83 cpuguy83 deleted the cpuguy83:local_logger branch Aug 20, 2018

@yosifkit

This comment has been minimized.

Copy link
Contributor

yosifkit commented Aug 21, 2018

Just to be clear for users that read the description in #33475 and think that more is done than is currently complete. It is correct that this PR only addresses the adding of the local log driver and does not yet add the "dual-logging" described there?

Maybe add a checklist over there to show when things are merged and which docker release is targeted for changing the default from jsonfile?

@cpuguy83

This comment has been minimized.

Copy link
Contributor Author

cpuguy83 commented Aug 21, 2018

@yosifkit That is correct, this is only the logger.

@cpuguy83 cpuguy83 referenced this pull request Aug 21, 2018

Open

[Proposal] Enhance local logging / remote logging #33475

1 of 2 tasks complete
@mterron

This comment has been minimized.

Copy link

mterron commented Aug 30, 2018

How do we enable this driver?

@cpuguy83

This comment has been minimized.

Copy link
Contributor Author

cpuguy83 commented Aug 31, 2018

@mterron it's not in a release yet, but it would be "--log-driver=local"

@crazy-max crazy-max referenced this pull request Nov 12, 2018

Closed

Traefik #8

albers added a commit to albers/docker-cli that referenced this pull request Dec 2, 2018

Add bash completion for "local" log driver
Ref: moby/moby#37092

Also adds log-opt `compress` to json-file log driver because this was
also added in the referenced PR.

Signed-off-by: Harald Albers <github@albersweb.de>

thaJeztah added a commit to thaJeztah/cli that referenced this pull request Dec 7, 2018

Add bash completion for "local" log driver
Ref: moby/moby#37092

Also adds log-opt `compress` to json-file log driver because this was
also added in the referenced PR.

Signed-off-by: Harald Albers <github@albersweb.de>
(cherry picked from commit c59038b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

docker-jenkins pushed a commit to docker/docker-ce that referenced this pull request Dec 7, 2018

Add bash completion for "local" log driver
Ref: moby/moby#37092

Also adds log-opt `compress` to json-file log driver because this was
also added in the referenced PR.

Signed-off-by: Harald Albers <github@albersweb.de>
Upstream-commit: c59038b15c4fd2a796665925dec1cbb5993d4501
Component: cli

docker-jenkins pushed a commit to docker/docker-ce that referenced this pull request Dec 8, 2018

Add bash completion for "local" log driver
Ref: moby/moby#37092

Also adds log-opt `compress` to json-file log driver because this was
also added in the referenced PR.

Signed-off-by: Harald Albers <github@albersweb.de>
(cherry picked from commit c59038b15c4fd2a796665925dec1cbb5993d4501)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: e05745b4a55bb9a7d5e5d01eb614949ab1d8e897
Component: cli

wherka pushed a commit to wherka/docker-ce that referenced this pull request Dec 18, 2018

Add bash completion for "local" log driver
Ref: moby/moby#37092

Also adds log-opt `compress` to json-file log driver because this was
also added in the referenced PR.

Signed-off-by: Harald Albers <github@albersweb.de>
Upstream-commit: c59038b15c4fd2a796665925dec1cbb5993d4501
Component: cli
Upstream-commit: 0e20f85
Component: cli
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Feb 20, 2019

This driver shipped with Docker CE 18.09 and up, for those that were still waiting for this to arrive 👍 (I think it was already included in Docker Enterprise (EE) 18.03)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.