Add rollover log driver, and --log-driver-opts flag #11485

Merged
merged 1 commit into from Jul 17, 2015

Conversation

Projects
None yet
@wlan0
Contributor

wlan0 commented Mar 19, 2015

Closes #8911

This adds log rollover capability to docker. I have added a new driver called rollover for this.

There are two options that can be specified along with this option. In order to specify these options, I added a new flag called --log-driver-opts. That would allow one to specify options like this

docker run --logging-driver rollover --log-driver-opts max_size=1k --log-driver-opts file_count=10

daemon/logger/rollover/rollover.go
+ return err
+}
+
+func writeLog(l *RolloverLogger) (int64, error) {

This comment has been minimized.

@cpuguy83

cpuguy83 Mar 19, 2015

Contributor

This probably needs to be protected by a mutex, no?

@cpuguy83

cpuguy83 Mar 19, 2015

Contributor

This probably needs to be protected by a mutex, no?

This comment has been minimized.

@ibuildthecloud

ibuildthecloud Mar 19, 2015

Contributor

The impression I got was that calls to Log() are serialized. @LK4D4 can you confirm that?

@ibuildthecloud

ibuildthecloud Mar 19, 2015

Contributor

The impression I got was that calls to Log() are serialized. @LK4D4 can you confirm that?

This comment has been minimized.

@LK4D4

LK4D4 Mar 19, 2015

Contributor

Uhm, serialized?

@LK4D4

LK4D4 Mar 19, 2015

Contributor

Uhm, serialized?

This comment has been minimized.

@wlan0

wlan0 Mar 19, 2015

Contributor

I think he meant to ask if the calls to Log are being made in parallel. They are not, right
? My understanding was that each container has a copier that calls Log(). Therefore, we shouldn't worry about race conditions.

@wlan0

wlan0 Mar 19, 2015

Contributor

I think he meant to ask if the calls to Log are being made in parallel. They are not, right
? My understanding was that each container has a copier that calls Log(). Therefore, we shouldn't worry about race conditions.

This comment has been minimized.

@LK4D4

LK4D4 Mar 19, 2015

Contributor

They are. You need to make logger threadsafe, because for now copier writes stdout and stderr concurrently.
Look #11472 :)

@LK4D4

LK4D4 Mar 19, 2015

Contributor

They are. You need to make logger threadsafe, because for now copier writes stdout and stderr concurrently.
Look #11472 :)

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Mar 19, 2015

Contributor

@wlan0 Gofmt check failed.
We need design review on new flag, but I sorta like it, not tried to use though.
ping @crosbymichael @tiborvass @jfrazelle

Contributor

LK4D4 commented Mar 19, 2015

@wlan0 Gofmt check failed.
We need design review on new flag, but I sorta like it, not tried to use though.
ping @crosbymichael @tiborvass @jfrazelle

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Mar 19, 2015

Contributor

@LK4D4 fixed the gofmt issue.

Contributor

wlan0 commented Mar 19, 2015

@LK4D4 fixed the gofmt issue.

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Mar 19, 2015

Contributor

@wlan0 Haha, tests don't compile :)

Contributor

LK4D4 commented Mar 19, 2015

@wlan0 Haha, tests don't compile :)

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Mar 19, 2015

Contributor

@LK4D4 All tests pass now.. There were some unused packages earlier

Contributor

wlan0 commented Mar 19, 2015

@LK4D4 All tests pass now.. There were some unused packages earlier

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Mar 19, 2015

Collaborator

@wlan0 @LK4D4 I'd much rather have --log-opt like we have --storage-opt.

Also, I think we should not make it a new driver, but instead make the default json-file driver have options with this new flag. So rollover would be done like this: docker run --log-opt max_file=10 --log-opt max_size=1k.

Proposed defaults:

  • max_size = 0 (no limit)
  • max_file = 1

What do you guys think of this? If we can agree, we could move on to code review.

Collaborator

tiborvass commented Mar 19, 2015

@wlan0 @LK4D4 I'd much rather have --log-opt like we have --storage-opt.

Also, I think we should not make it a new driver, but instead make the default json-file driver have options with this new flag. So rollover would be done like this: docker run --log-opt max_file=10 --log-opt max_size=1k.

Proposed defaults:

  • max_size = 0 (no limit)
  • max_file = 1

What do you guys think of this? If we can agree, we could move on to code review.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Mar 19, 2015

Contributor

+1 I like @tiborvass's suggestion.

Contributor

cpuguy83 commented Mar 19, 2015

+1 I like @tiborvass's suggestion.

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Mar 19, 2015

Contributor

@tiborvass I'm okay with moving that logic to default driver.

Contributor

LK4D4 commented Mar 19, 2015

@tiborvass I'm okay with moving that logic to default driver.

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Mar 19, 2015

Contributor

@tiborvass I also agree it should be in the default, we did it as a separate only because we didn't know if you guys would want us screwing with the default logging driver. For example, just keep the default stupid simple because you know it works.

Contributor

ibuildthecloud commented Mar 19, 2015

@tiborvass I also agree it should be in the default, we did it as a separate only because we didn't know if you guys would want us screwing with the default logging driver. For example, just keep the default stupid simple because you know it works.

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Mar 19, 2015

Collaborator

@ibuildthecloud except this PR would introduce log options, and that's a whole new world :)

EDIT: https://www.youtube.com/watch?v=-kl4hJ4j48s

Collaborator

tiborvass commented Mar 19, 2015

@ibuildthecloud except this PR would introduce log options, and that's a whole new world :)

EDIT: https://www.youtube.com/watch?v=-kl4hJ4j48s

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Mar 19, 2015

Contributor

Okay. To sum up.

  • Change --log-driver-opts to --log-opts
  • Remove "rollover" driver and put logic in json-file
  • max_size = 0 should mean unlimited
  • Defaults should be max_size=0, max_file=1

Question:

  • Should it be max-size or max_size, which is more consistent with existing Docker?
  • docker logs should read from all historical logs or just the latest?
Contributor

ibuildthecloud commented Mar 19, 2015

Okay. To sum up.

  • Change --log-driver-opts to --log-opts
  • Remove "rollover" driver and put logic in json-file
  • max_size = 0 should mean unlimited
  • Defaults should be max_size=0, max_file=1

Question:

  • Should it be max-size or max_size, which is more consistent with existing Docker?
  • docker logs should read from all historical logs or just the latest?
@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Mar 19, 2015

Collaborator

@ibuildthecloud

flag would be singular: --log-opt.

Let's make it max-size to make it consistent with -restart on-failure[:max-retry]. Also I generally prefer - rather than _.

If docker logs outputs only the latest file, the default value of --tail would need to change to latest or something like that. I'm not sure what's the best solution for that, I'm still thinking.

Collaborator

tiborvass commented Mar 19, 2015

@ibuildthecloud

flag would be singular: --log-opt.

Let's make it max-size to make it consistent with -restart on-failure[:max-retry]. Also I generally prefer - rather than _.

If docker logs outputs only the latest file, the default value of --tail would need to change to latest or something like that. I'm not sure what's the best solution for that, I'm still thinking.

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Mar 19, 2015

Contributor

@LK4D4 @tiborvass updated the code according to comments.

  • Now, the flag is --log-opt
  • the options use - instead of _, so max-file and max-size are the options
  • the options have been added to the json-file log driver
  • the options have default values max-file=1 and max-size=0 (infinite)
  • updated docs

example

docker run --log-driver json-file --log-opt max-size=1k --log-opt max-file=10 -d redis

Contributor

wlan0 commented Mar 19, 2015

@LK4D4 @tiborvass updated the code according to comments.

  • Now, the flag is --log-opt
  • the options use - instead of _, so max-file and max-size are the options
  • the options have been added to the json-file log driver
  • the options have default values max-file=1 and max-size=0 (infinite)
  • updated docs

example

docker run --log-driver json-file --log-opt max-size=1k --log-opt max-file=10 -d redis

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Mar 20, 2015

Collaborator

@wlan0 in your example --log-driver json-file is not necessary since it's the default.

I agree with everything you pointed out. Don't forget to update the API as well.

What's left to decide is what to do with docker logs... :S @LK4D4 @jfrazelle @icecrime

EDIT: also, what happens when it reaches maximum? How are the log files named? Do you go back to "file1" or do you remove file1, and write to file11 ?

Collaborator

tiborvass commented Mar 20, 2015

@wlan0 in your example --log-driver json-file is not necessary since it's the default.

I agree with everything you pointed out. Don't forget to update the API as well.

What's left to decide is what to do with docker logs... :S @LK4D4 @jfrazelle @icecrime

EDIT: also, what happens when it reaches maximum? How are the log files named? Do you go back to "file1" or do you remove file1, and write to file11 ?

@pires

This comment has been minimized.

Show comment
Hide comment

pires commented Mar 21, 2015

Great!

@mahnunchik

This comment has been minimized.

Show comment
Hide comment

👍

@gtonic

This comment has been minimized.

Show comment
Hide comment

gtonic commented Mar 21, 2015

+1

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Mar 23, 2015

Contributor

@tiborvass Once it reaches maximum, I go back to file1

Contributor

wlan0 commented Mar 23, 2015

@tiborvass Once it reaches maximum, I go back to file1

@ibuildthecloud

This comment has been minimized.

Show comment
Hide comment
@ibuildthecloud

ibuildthecloud Mar 23, 2015

Contributor

@wlan0 What do the filename look like? When a container is destroyed, do all the log files get deleted?

Contributor

ibuildthecloud commented Mar 23, 2015

@wlan0 What do the filename look like? When a container is destroyed, do all the log files get deleted?

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Mar 23, 2015

Contributor

The logs are always written to the default log file - /var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log

Then the logs are rotated amongst
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-1
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-2
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log....
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-$(n-1)

where n = the maximum number of log files

I start by writing to the default log file. Then once this file reaches (max_size/n), I remove the last log file (/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-$(n-1)) and switch the contents of file-$(n-2) to file-$(n-1), and file-$(n-3) to file-$(n-2) and so on...

i.e. The higher the number on the log file name, the older the log is.

When a container is destroyed, the log files are deleted as well

Contributor

wlan0 commented Mar 23, 2015

The logs are always written to the default log file - /var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log

Then the logs are rotated amongst
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-1
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-2
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log....
/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-$(n-1)

where n = the maximum number of log files

I start by writing to the default log file. Then once this file reaches (max_size/n), I remove the last log file (/var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log-$(n-1)) and switch the contents of file-$(n-2) to file-$(n-1), and file-$(n-3) to file-$(n-2) and so on...

i.e. The higher the number on the log file name, the older the log is.

When a container is destroyed, the log files are deleted as well

@defunctzombie

This comment has been minimized.

Show comment
Hide comment
@defunctzombie

defunctzombie Mar 23, 2015

Any reason why the default is unlimited? Just thinking about situations where a user has not configured docker and their systems will fill up with logs.

Any reason why the default is unlimited? Just thinking about situations where a user has not configured docker and their systems will fill up with logs.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Mar 24, 2015

Contributor

@defunctzombie because that's how it currently works.

Contributor

cpuguy83 commented Mar 24, 2015

@defunctzombie because that's how it currently works.

@icecrime icecrime removed the dco/yes label Mar 30, 2015

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Apr 1, 2015

Contributor

@tiborvass for the docker logs command, we could add a new flag called index (i)

The index flag represents the file number that you want to tail. i would be a valid option only for the rollover log driver. it would have a default value of 0, meaning you tail the latest logs. `i' is valid in the ranges [0, n)
,where n = maximum number of log files

this way, a user can tail whichever one of the log files they want to.

Contributor

wlan0 commented Apr 1, 2015

@tiborvass for the docker logs command, we could add a new flag called index (i)

The index flag represents the file number that you want to tail. i would be a valid option only for the rollover log driver. it would have a default value of 0, meaning you tail the latest logs. `i' is valid in the ranges [0, n)
,where n = maximum number of log files

this way, a user can tail whichever one of the log files they want to.

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Apr 1, 2015

Contributor

If any kind of driver specific options are to be added for logging, these need to be specified in a dynamic way without adding a ton of flags which are driver dependent.

This would be OK:

docker logs --driver-options "option1=value1,option2=value2" container-id

NOT OK:

docker logs --some-driver-option 6 --some-driver-another-option foo container-id
Contributor

unclejack commented Apr 1, 2015

If any kind of driver specific options are to be added for logging, these need to be specified in a dynamic way without adding a ton of flags which are driver dependent.

This would be OK:

docker logs --driver-options "option1=value1,option2=value2" container-id

NOT OK:

docker logs --some-driver-option 6 --some-driver-another-option foo container-id
@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Apr 1, 2015

Collaborator

@unclejack we already have --storage-opt. So the format is --log-opt k1=v1 --log-opt k2=v2

Collaborator

tiborvass commented Apr 1, 2015

@unclejack we already have --storage-opt. So the format is --log-opt k1=v1 --log-opt k2=v2

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Apr 1, 2015

Collaborator

@wlan0 like I said earlier, there would not be a rollover driver, it would be part of the default json-file driver.

Everyone, are we all okay naming the different files *.log then *.log-1, then *.log-2 ? Shouldn't we do *.log then *.log.1 then *.log.2 instead?

@wlan0 if you want to add a new --index flag to docker logs, it has to have a meaning for ALL drivers, and I'm afraid it does not. So we need a better solution. Basically each driver would get a request saying "gimme the last --tail logs". By default --tail=all. Should all be interpreted differently by different drivers? I feel like there's something wrong with docker logs. Maybe docker logs only makes sense for with the default driver :S Ping @LK4D4 @icecrime @crosbymichael

Collaborator

tiborvass commented Apr 1, 2015

@wlan0 like I said earlier, there would not be a rollover driver, it would be part of the default json-file driver.

Everyone, are we all okay naming the different files *.log then *.log-1, then *.log-2 ? Shouldn't we do *.log then *.log.1 then *.log.2 instead?

@wlan0 if you want to add a new --index flag to docker logs, it has to have a meaning for ALL drivers, and I'm afraid it does not. So we need a better solution. Basically each driver would get a request saying "gimme the last --tail logs". By default --tail=all. Should all be interpreted differently by different drivers? I feel like there's something wrong with docker logs. Maybe docker logs only makes sense for with the default driver :S Ping @LK4D4 @icecrime @crosbymichael

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 1, 2015

Contributor

I think that developing docker logs for each driver is PITA and doesn't worth it.
But this driver will be default, so for now we can end with tailing last file and document it.

Contributor

LK4D4 commented Apr 1, 2015

I think that developing docker logs for each driver is PITA and doesn't worth it.
But this driver will be default, so for now we can end with tailing last file and document it.

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Apr 1, 2015

Collaborator

@LK4D4 then how would you tail ALL files ?

Collaborator

tiborvass commented Apr 1, 2015

@LK4D4 then how would you tail ALL files ?

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 1, 2015

Contributor

@tiborvass I would just document that you can't tail ALL files now. I'm not sure that there is person in world who want so big tails(I'm sure they exist, but they want it only to make me sad).

Contributor

LK4D4 commented Apr 1, 2015

@tiborvass I would just document that you can't tail ALL files now. I'm not sure that there is person in world who want so big tails(I'm sure they exist, but they want it only to make me sad).

@tiborvass

This comment has been minimized.

Show comment
Hide comment
@tiborvass

tiborvass Apr 1, 2015

Collaborator

@LK4D4 how about having --tail last by default, and still keep --tail all ?

Collaborator

tiborvass commented Apr 1, 2015

@LK4D4 how about having --tail last by default, and still keep --tail all ?

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Apr 1, 2015

Contributor

@tiborvass Do we have --tail=all already implemented to keep it?

Contributor

LK4D4 commented Apr 1, 2015

@tiborvass Do we have --tail=all already implemented to keep it?

daemon/logger/jsonfilelog/jsonfilelog.go
+ return nil
+}
+
+func (l *JSONFileLogger) GetReaderByIndex(index int) (io.Reader, error) {

This comment has been minimized.

@LK4D4

LK4D4 Jul 1, 2015

Contributor

Omg, I don't like this so much. But I think we have no choice now. GetReader was superbad idea in the first place.
I don't know, maybe use type assertion instead of bringing this to interface?

@LK4D4

LK4D4 Jul 1, 2015

Contributor

Omg, I don't like this so much. But I think we have no choice now. GetReader was superbad idea in the first place.
I don't know, maybe use type assertion instead of bringing this to interface?

This comment has been minimized.

@cpuguy83

cpuguy83 Jul 1, 2015

Contributor

I didn't like it either...but couldn't think of anything better.
I don't like asserting to a specific type for this either.

Maybe we split the reader interface out of the main logger interface, since not everything will be implementing reader anyway.

Then in daemon/logs.go we can do type assertion to a reader, if that fails we return ErrNotSupported (or whatever the error is)

@cpuguy83

cpuguy83 Jul 1, 2015

Contributor

I didn't like it either...but couldn't think of anything better.
I don't like asserting to a specific type for this either.

Maybe we split the reader interface out of the main logger interface, since not everything will be implementing reader anyway.

Then in daemon/logs.go we can do type assertion to a reader, if that fails we return ErrNotSupported (or whatever the error is)

This comment has been minimized.

@wlan0

wlan0 Jul 1, 2015

Contributor

I agree with cpuguy83. Doing a type assertion would just move the problem around. I don't like this either, but couldn't think of a better solution that doesn't involve extensive change.

@wlan0

wlan0 Jul 1, 2015

Contributor

I agree with cpuguy83. Doing a type assertion would just move the problem around. I don't like this either, but couldn't think of a better solution that doesn't involve extensive change.

This comment has been minimized.

@LK4D4

LK4D4 Jul 1, 2015

Contributor

GetReader is abomination which must die anyway. It's very sad, that now we have so much changes which rely on it. That's why I thought it was better to have type assertion, so no-one will try to adapt awful json-file logs retrieval.

@LK4D4

LK4D4 Jul 1, 2015

Contributor

GetReader is abomination which must die anyway. It's very sad, that now we have so much changes which rely on it. That's why I thought it was better to have type assertion, so no-one will try to adapt awful json-file logs retrieval.

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 1, 2015

Contributor

@cpuguy83 @LK4D4 I separated logger and reader interfaces.

Contributor

wlan0 commented Jul 1, 2015

@cpuguy83 @LK4D4 I separated logger and reader interfaces.

add support for maximum log size, and max number of log files
Signed-off-by: wlan0 <sidharthamn@gmail.com>
@jessfraz

This comment has been minimized.

Show comment
Hide comment
@jessfraz

jessfraz Jul 13, 2015

Contributor

oh man needs rebase then I promise we should be able to get this in, hopefully, no promises, well kinda... ;)

Contributor

jessfraz commented Jul 13, 2015

oh man needs rebase then I promise we should be able to get this in, hopefully, no promises, well kinda... ;)

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 13, 2015

Contributor

not a problem :) Gimme till the end of the day.

Contributor

wlan0 commented Jul 13, 2015

not a problem :) Gimme till the end of the day.

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 13, 2015

Contributor

@jfrazelle the rebase was fortunately easier than last time. :) It's ready.

Contributor

wlan0 commented Jul 13, 2015

@jfrazelle the rebase was fortunately easier than last time. :) It's ready.

@jessfraz

This comment has been minimized.

Show comment
Hide comment
@jessfraz

jessfraz Jul 14, 2015

Contributor

ping @LK4D4 for last review :)

Contributor

jessfraz commented Jul 14, 2015

ping @LK4D4 for last review :)

@calavera

This comment has been minimized.

Show comment
Hide comment
@calavera

calavera Jul 15, 2015

Contributor

code LGMT. Moving to docs review.

Contributor

calavera commented Jul 15, 2015

code LGMT. Moving to docs review.

docs/reference/logging/index.md
@@ -17,7 +17,7 @@ container's logging driver. The following options are supported:
| `none` | Disables any logging for the container. `docker logs` won't be available with this driver. |
|-------------|-------------------------------------------------------------------------------------------------------------------------------|
-| `json-file` | Default logging driver for Docker. Writes JSON messages to file. No logging options are supported for this driver. |
+| `json-file` | Default logging driver for Docker. Writes JSON messages to file. two logging options are supported for this driver. |

This comment has been minimized.

@calavera

calavera Jul 15, 2015

Contributor

s/two/Two/, I think there is also an extra white space before the beginning of the sentence that we can remove.

@calavera

calavera Jul 15, 2015

Contributor

s/two/Two/, I think there is also an extra white space before the beginning of the sentence that we can remove.

This comment has been minimized.

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

Default Docker logging driver. Writes JSON messages to a file. This driver supports two options.

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

Default Docker logging driver. Writes JSON messages to a file. This driver supports two options.

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 15, 2015

Contributor

@calavera Fixed the doc!

Contributor

wlan0 commented Jul 15, 2015

@calavera Fixed the doc!

docs/reference/commandline/logs.md
@@ -19,6 +19,7 @@ weight=1
--since="" Show logs since timestamp
-t, --timestamps=false Show timestamps
--tail="all" Number of lines to show from the end of the logs
+ --log-opt="" Specify options to the logger, Currently supported only by json-file log driver

This comment has been minimized.

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

The entire command has a note that says it only works with the json-file log driver --- so you don't need to repeat that on the flag.

"Specify logger options. " is tighter

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

The entire command has a note that says it only works with the json-file log driver --- so you don't need to repeat that on the flag.

"Specify logger options. " is tighter

docs/reference/commandline/logs.md
@@ -29,7 +30,7 @@ The `docker logs --follow` command will continue streaming the new output from
the container's `STDOUT` and `STDERR`.
Passing a negative number or a non-integer to `--tail` is invalid and the
-value is set to `all` in that case. This behavior may change in the future.
+value is set to `latest` in that case. This behavior may change in the future.

This comment has been minimized.

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

Omit "This behavior may change in the future" as the facts of that statement are inherent.

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

Omit "This behavior may change in the future" as the facts of that statement are inherent.

docs/reference/logging/index.md
+ --log-opt max-file=[0-9+]
+
+`max-size` sets the size of each of the log files that are rolled over
+`max-file` sets the maximum number of log files that are rolled over amongst

This comment has been minimized.

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

missing something after "amongst" --- also --- what is the point of the max size? and 9+ does that mean unlimited? Why aren't we use date or something --- roll over files that are aged past 9 days and x size...

---> Something like

The json-file options

The json-file logging driver supports the following options:

--log-opt max-size=[0-9+][k/m/g/K/M/G]
--log-opt max-file=[0-9+]

max-size Logs that reach this size are rolled over. You can set the size in kilobytes (k), megbytes (m), or gigbytes (g).

@moxiegirl

moxiegirl Jul 15, 2015

Contributor

missing something after "amongst" --- also --- what is the point of the max size? and 9+ does that mean unlimited? Why aren't we use date or something --- roll over files that are aged past 9 days and x size...

---> Something like

The json-file options

The json-file logging driver supports the following options:

--log-opt max-size=[0-9+][k/m/g/K/M/G]
--log-opt max-file=[0-9+]

max-size Logs that reach this size are rolled over. You can set the size in kilobytes (k), megbytes (m), or gigbytes (g).

This comment has been minimized.

@wlan0

wlan0 Jul 15, 2015

Contributor

@moxiegirl For max-size I am trying to denote that one or more digits should be present before the unit field. If max-size is not set, then it is assumed to be unlimited.

The default value for max-size parameter is unlimited (denoted by not setting that parameter). How about something like this?

The json-file options

The following logging options are supported for the json-file logging driver:

   --log-opt max-size=(unlimited)
   --log-opt max-file=1

max-size Logs that reach this size are rolled over. You can set the size in kilobytes(k), megabytes(m), or gigabytes(g). eg --log-opt max-size=50m
max-file Maximum number of files that a log is rolled over before being discarded. eg --log-opt max-file=100

@wlan0

wlan0 Jul 15, 2015

Contributor

@moxiegirl For max-size I am trying to denote that one or more digits should be present before the unit field. If max-size is not set, then it is assumed to be unlimited.

The default value for max-size parameter is unlimited (denoted by not setting that parameter). How about something like this?

The json-file options

The following logging options are supported for the json-file logging driver:

   --log-opt max-size=(unlimited)
   --log-opt max-file=1

max-size Logs that reach this size are rolled over. You can set the size in kilobytes(k), megabytes(m), or gigabytes(g). eg --log-opt max-size=50m
max-file Maximum number of files that a log is rolled over before being discarded. eg --log-opt max-file=100

@moxiegirl

This comment has been minimized.

Show comment
Hide comment
@moxiegirl

moxiegirl Jul 15, 2015

Contributor

@wlan0 Thanks for the contribution. Please check your docs output with a make docs from the docs dir.

Contributor

moxiegirl commented Jul 15, 2015

@wlan0 Thanks for the contribution. Please check your docs output with a make docs from the docs dir.

docs/reference/logging/index.md
@@ -17,7 +17,7 @@ container's logging driver. The following options are supported:
| `none` | Disables any logging for the container. `docker logs` won't be available with this driver. |
|-------------|-------------------------------------------------------------------------------------------------------------------------------|
-| `json-file` | Default logging driver for Docker. Writes JSON messages to file. No logging options are supported for this driver. |
+| `json-file` | Default logging driver for Docker. Writes JSON messages to file. Two logging options are supported for this driver. |

This comment has been minimized.

@thaJeztah

thaJeztah Jul 15, 2015

Member

I think you can remove "Two logging options are supported for this driver.". We don't mention this for the other drivers and it is explained below, so doesn't add much.

@thaJeztah

thaJeztah Jul 15, 2015

Member

I think you can remove "Two logging options are supported for this driver.". We don't mention this for the other drivers and it is explained below, so doesn't add much.

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 15, 2015

Contributor

@moxiegirl @thaJeztah I've updated the docs based on your comments.

Contributor

wlan0 commented Jul 15, 2015

@moxiegirl @thaJeztah I've updated the docs based on your comments.

docs/reference/logging/index.md
+The following logging options are supported for the `json-file` logging driver:
+
+ --log-opt max-size=(unlimited)
+ --log-opt max-file=1

This comment has been minimized.

@thaJeztah

thaJeztah Jul 15, 2015

Member

This isn't rendered as code-block. Perhaps the indentation is 3 in stead of 4 spaces? You can test the docs by running make docs, which will build the and serve the docs in a container so that you can preview them in your webbrowser.

@thaJeztah

thaJeztah Jul 15, 2015

Member

This isn't rendered as code-block. Perhaps the indentation is 3 in stead of 4 spaces? You can test the docs by running make docs, which will build the and serve the docs in a container so that you can preview them in your webbrowser.

This comment has been minimized.

@thaJeztah

thaJeztah Jul 15, 2015

Member

Thinking of the max-size option; if the default is "unlimited", then specifying max-files without specifying max-size is a no-op? I.e., the max-size is never reached, so there will never be a roll-over?

@thaJeztah

thaJeztah Jul 15, 2015

Member

Thinking of the max-size option; if the default is "unlimited", then specifying max-files without specifying max-size is a no-op? I.e., the max-size is never reached, so there will never be a roll-over?

This comment has been minimized.

@wlan0

wlan0 Jul 15, 2015

Contributor

yes. That is correct. max-file is meaningless without max-size. It won't be honored even if you set it. If you want me to describe it differently, please let me know.

@wlan0

wlan0 Jul 15, 2015

Contributor

yes. That is correct. max-file is meaningless without max-size. It won't be honored even if you set it. If you want me to describe it differently, please let me know.

@@ -25,6 +25,16 @@ container's logging driver. The following options are supported:
The `docker logs`command is available only for the `json-file` logging driver.
+### The json-file options
+
+The following logging options are supported for the `json-file` logging driver:

This comment has been minimized.

@thaJeztah

thaJeztah Jul 15, 2015

Member

Can you change this line to the example that @moxiegirl gave in #11485 (comment) ?

@thaJeztah

thaJeztah Jul 15, 2015

Member

Can you change this line to the example that @moxiegirl gave in #11485 (comment) ?

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 16, 2015

Contributor
Contributor

wlan0 commented Jul 16, 2015

@draghuram

This comment has been minimized.

Show comment
Hide comment
@draghuram

draghuram Jul 16, 2015

Contributor

Hi,

Thanks for the PR as I have no doubt it adds a very useful feature to Docker. Here are some comments about the doc changes.

  • docs/reference/logging/index.md

    • As @thaJeztah mentioned, I think it is worthwhile to clearly document that max-file is meaningless without max-size. At the same time, we should also mention the default value for max-size (which is unlimited).
    • There is a missing space in "The docker logscommand" though it is not this PR's doing.
  • docs/reference/commandline/logs.md

    • It is not clear to me why log-opt option is added to docker logs command. What exactly can I pass with this option? I may be missing something here but the options max-size and max-file make sense only for docker run command.
    • If I run docker logs without any options, does it combine all the log files or just return the latest? I ask because the command seems to default to "latest" when an invalid value is passed to --tail option. Similarly, does --since option apply across all the log files?
  • docs/reference/commandline/run.md

    It would be beneficial if an example is provided in this file covering the new log options.

Contributor

draghuram commented Jul 16, 2015

Hi,

Thanks for the PR as I have no doubt it adds a very useful feature to Docker. Here are some comments about the doc changes.

  • docs/reference/logging/index.md

    • As @thaJeztah mentioned, I think it is worthwhile to clearly document that max-file is meaningless without max-size. At the same time, we should also mention the default value for max-size (which is unlimited).
    • There is a missing space in "The docker logscommand" though it is not this PR's doing.
  • docs/reference/commandline/logs.md

    • It is not clear to me why log-opt option is added to docker logs command. What exactly can I pass with this option? I may be missing something here but the options max-size and max-file make sense only for docker run command.
    • If I run docker logs without any options, does it combine all the log files or just return the latest? I ask because the command seems to default to "latest" when an invalid value is passed to --tail option. Similarly, does --since option apply across all the log files?
  • docs/reference/commandline/run.md

    It would be beneficial if an example is provided in this file covering the new log options.

@moxiegirl

This comment has been minimized.

Show comment
Hide comment
@moxiegirl

moxiegirl Jul 16, 2015

Contributor

@draghuram makes very good points. @wlan0 can you clarify in the docs these points please? Thank you for being so cooperative on making changes. User would appreciate this extra clarification.

Contributor

moxiegirl commented Jul 16, 2015

@draghuram makes very good points. @wlan0 can you clarify in the docs these points please? Thank you for being so cooperative on making changes. User would appreciate this extra clarification.

@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 16, 2015

Contributor

@moxiegirl @draghuram Thanks! Absolutely not a problem. I would like to be as clear as possible while describing these options as well.

Firstly, I've updated the docs keeping in mind all of your comments. Secondly, to answer your questions -

  1. --log-opt doesn't make sense in logs command. Good catch. Removed this from the logs.md file in the docs.
  2. If you run docker logs without options, it only provides the latest logs. Added this to the docs.

I've updated the docs according to the rest of your comments.

Contributor

wlan0 commented Jul 16, 2015

@moxiegirl @draghuram Thanks! Absolutely not a problem. I would like to be as clear as possible while describing these options as well.

Firstly, I've updated the docs keeping in mind all of your comments. Secondly, to answer your questions -

  1. --log-opt doesn't make sense in logs command. Good catch. Removed this from the logs.md file in the docs.
  2. If you run docker logs without options, it only provides the latest logs. Added this to the docs.

I've updated the docs according to the rest of your comments.

+
+`max-file` specifies the maximum number of files that a log is rolled over before being discarded. eg `--log-opt max-file=100`. If `max-size` is not set, then `max-file` is not honored.
+
+If `max-size` and `max-file` are set, `docker logs` only returns the log lines from the newest log file.

This comment has been minimized.

@draghuram

draghuram Jul 17, 2015

Contributor

I think this information really belongs in logs.md. In fact, I would like logs.md to provide some description of rollover feature and how that impacts the behavior of options. For example, --tail description says that when an invalid value is passed, the value is set to latest. However, you can't tell what latest means just by looking at the man page.

@draghuram

draghuram Jul 17, 2015

Contributor

I think this information really belongs in logs.md. In fact, I would like logs.md to provide some description of rollover feature and how that impacts the behavior of options. For example, --tail description says that when an invalid value is passed, the value is set to latest. However, you can't tell what latest means just by looking at the man page.

This comment has been minimized.

@draghuram

draghuram Jul 17, 2015

Contributor

Just to be clear, my comment above only applies to the line "If max-size and max-file are set, docker logs only returns the log lines from the newest log file.".

@draghuram

draghuram Jul 17, 2015

Contributor

Just to be clear, my comment above only applies to the line "If max-size and max-file are set, docker logs only returns the log lines from the newest log file.".

This comment has been minimized.

@moxiegirl

moxiegirl Jul 17, 2015

Contributor

@draghuram I think you have a point about the logs.md file but it is beyond the scope of this PR. I created an issue here: #14711 and assigned it to myself.

@moxiegirl

moxiegirl Jul 17, 2015

Contributor

@draghuram I think you have a point about the logs.md file but it is beyond the scope of this PR. I created an issue here: #14711 and assigned it to myself.

@draghuram

This comment has been minimized.

Show comment
Hide comment
@draghuram

draghuram Jul 17, 2015

Contributor

@wlan0 Thanks for the changes. In addition to my comment above, I would really like to see an example in run.md but I am ok with the PR being merged without one.

Contributor

draghuram commented Jul 17, 2015

@wlan0 Thanks for the changes. In addition to my comment above, I would really like to see an example in run.md but I am ok with the PR being merged without one.

@moxiegirl

This comment has been minimized.

Show comment
Hide comment
@moxiegirl

moxiegirl Jul 17, 2015

Contributor

LGTM thanks @wlan0

Contributor

moxiegirl commented Jul 17, 2015

LGTM thanks @wlan0

@moxiegirl

This comment has been minimized.

Show comment
Hide comment
@moxiegirl

moxiegirl Jul 17, 2015

Contributor

@thaJeztah for the merge.

Contributor

moxiegirl commented Jul 17, 2015

@thaJeztah for the merge.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 17, 2015

Member

LGTM! Thanks so much for being so patient and cooperative, @wlan0. Apologies that it took so long :-)

Member

thaJeztah commented Jul 17, 2015

LGTM! Thanks so much for being so patient and cooperative, @wlan0. Apologies that it took so long :-)

thaJeztah added a commit that referenced this pull request Jul 17, 2015

Merge pull request #11485 from wlan0/rollover_log
Add rollover log driver, and --log-driver-opts flag

@thaJeztah thaJeztah merged commit 415f744 into moby:master Jul 17, 2015

4 checks passed

docker/dco-signed All commits signed
Details
experimental Jenkins build Docker-PRs-experimental 2946 has succeeded
Details
janky Jenkins build Docker-PRs 11505 has succeeded
Details
windows Jenkins build Windows-PRs 8126 has succeeded
Details
@wlan0

This comment has been minimized.

Show comment
Hide comment
@wlan0

wlan0 Jul 17, 2015

Contributor

yaay finally!!! thanks all... :)

Contributor

wlan0 commented Jul 17, 2015

yaay finally!!! thanks all... :)

@cdancy

This comment has been minimized.

Show comment
Hide comment

cdancy commented Jul 17, 2015

+1!!!

@wlan0 wlan0 deleted the wlan0:rollover_log branch Jul 18, 2015

@artpolikarpov artpolikarpov referenced this pull request in arunoda/meteor-up-legacy Nov 21, 2015

Open

Suddenly broke uploading packages (mupx) #762

@henfri

This comment has been minimized.

Show comment
Hide comment
@henfri

henfri Feb 17, 2016

Hello,

thanks for this feature!
Is it possible to use it with docker-compose?
How do I specify/call it in the compose.yml

Greetings,
Hendrik

henfri commented Feb 17, 2016

Hello,

thanks for this feature!
Is it possible to use it with docker-compose?
How do I specify/call it in the compose.yml

Greetings,
Hendrik

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 17, 2016

Member

@henfri please don't comment on closed/merged PR's for support questions; the issue tracker is meant for tracking bugs, and reviewing pull requests; your question is better asked;

Member

thaJeztah commented Feb 17, 2016

@henfri please don't comment on closed/merged PR's for support questions; the issue tracker is meant for tracking bugs, and reviewing pull requests; your question is better asked;

@genggjh genggjh referenced this pull request in hyperledger-archives/fabric Jun 16, 2016

Closed

chaincode logging fills up system drive #709

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment