service logs #28089

aluzzardi · 2016-11-05T02:57:27Z

This PR adds docker service logs support by wiring up the SwarmKit logs plumbing.

This includes:

Container Adapter & Controller: To push up the logs from the agent using the internal engine APIs
API integration: Exposing logs using a similar API than the regular docker logs
CLI integration: docker service logs

I want to validate the API before documenting and adding integration tests, so this PR does not include them and is not in a mergeable state.

/cc @aaronlehmann @stevvooe

aaronlehmann · 2016-11-05T17:44:26Z

cli/command/service/logs.go

+	if err != nil {
+		return 0, err
+	}
+	if n != len(output) {


Write must return a non-nil error if it returns n < len(p)

If lw.w.Write follows this requirement, this check should not be necessary.

aaronlehmann · 2016-11-05T17:45:01Z

cli/command/service/logs.go

+		return 0, err
+	}
+	if n != len(output) {
+		return 0, err


As noted above, this if doesn't seem necessary. But if it's kept, it should return n and a non-nil error.

aaronlehmann · 2016-11-05T17:49:34Z

client/service_logs.go

+	}
+	query.Set("tail", options.Tail)
+
+	resp, err := cli.get(ctx, "/services/"+serviceID+"/logs", query, nil)


Wondering if we should validate client-side that user input only contains valid characters. Passing in a service ID with a ? in it will cause some pretty weird behavior. But this should probably be done in a different PR.

aaronlehmann · 2016-11-05T17:58:04Z

daemon/cluster/cluster.go

+				msg.Context.NodeID,
+				msg.Context.ServiceID,
+				msg.Context.TaskID,
+			)), data...)


I would reorder these appends to append rather than prepend. This is more intuitive and also slightly more efficient (prepending has to copy the old contents, appending might not if cap(x) > len(x)).

Actually, it might be more efficient to just flush to the stream, rather than assembling the entire data buffer.

Followed the same pattern as docker logs (reverse append) - I can change that

stevvooe · 2016-11-08T22:21:45Z

api/server/router/swarm/cluster_routes.go

+
+	serviceName := vars["id"]
+	logsConfig := &backend.ContainerLogsConfig{
+		ContainerLogsOptions: basictypes.ContainerLogsOptions{


Do we rename this to LogsOptions?

In a follow up

Can we add a TODO somewhere?

aluzzardi · 2016-11-08T23:24:51Z

@aaronlehmann @stevvooe Actually, the biggest thing I need to solve is the format.

Here's docker logs:

[optional timestamp] [optional details] [line]

Details (docker logs --details) are key values (k1=v1,k2=v2) which I believe are set by the log driver?

In service logs, we have a context (node_id, task_id, etc).

Should we support "details" in service logs?
Should we inject the context into the details or keep it separate?

Having the context in details makes the service logs protocol 100% compliant with docker logs, although it gets a little messier for parsing: depending whether --timestamp and or --details was passed or not, we have to split the log stream differently. Details contains the context (which we need to resolve to actual service names) and the actual details (which we need to passthrough).

Thoughts?

/cc @tiborvass @icecrime @vieux

Original:
[optional timestamp] [optional details] [line]

Service logs proposals:
1) [context] [optional timestamp] [optional details] [line]
2) [context] [optional timestamp] [line]
3) [optional timestamp] [optional details + context] [line]
4) [optional timestamp] [optional context] [line]

vieux · 2016-11-09T00:03:02Z

@aluzzardi I would start by the timestamp, so I vote for 3) or 4) details is nice to have but can come later.

stevvooe · 2016-11-09T00:03:50Z

[optional timestamp] [optional details + context] [line]

I vote for 3.

aluzzardi · 2016-11-09T00:14:07Z

Which one though:

[optional(details) + context]
OR
[optional(details + context)]

@dnephin @aanand Any input? The output of service logs basically looks a lot like compose logs.

@stevvooe @vieux The weird part is that context has a special meaning. The CLI will parse out the context to resolve it (map service ID to names and so on)

stevvooe · 2016-11-09T00:19:34Z

[optional(details) + context]

I'm in favor of this, since you need the context to disambiguate the streams.

vieux · 2016-11-09T01:44:31Z

@aluzzardi #28088 was merged

dongluochen · 2016-11-08T22:40:49Z

api/server/router/swarm/backend.go

@@ -17,6 +19,7 @@ type Backend interface {
 	CreateService(types.ServiceSpec, string) (string, error)
 	UpdateService(string, uint64, types.ServiceSpec, string, string) error
 	RemoveService(string) error
+	ServiceLogs(context.Context, string, *backend.ContainerLogsConfig, chan struct{}) error


It'd be better to follow Verb + Noun naming convention. ServiceLogs -> GetServiceLogs?

Staying consistent with ContainerLogs

It'd be better to follow Verb + Noun naming convention.

This is not a convention in Go.

dongluochen · 2016-11-09T07:14:58Z

cli/command/service/logs.go

+}
+
+func (lw *logWriter) Write(buf []byte) (int, error) {
+	parts := bytes.SplitN(buf, []byte(" "), 2)


Does this mean context cannot have any space?

dongluochen · 2016-11-09T07:58:49Z

cli/command/service/logs.go

+	noResolve  bool
+	follow     bool
+	since      string
+	timestamps bool


Does logs from different nodes ordered by timestamps by default? As an option?

No. The resource requirements for doing so are too intensive. In practice, they tend to come in with the correct ordering. To guarantee it, you have to buffer all logs, sort them and merge on client-side. Empty signals also need to be sent to preserve ordering.

aluzzardi · 2016-11-09T18:35:46Z

[optional(details) + context]

I'm in favor of this, since you need the context to disambiguate the streams.

Keep in mind that this means:

Logs format is incompatible with service logs (because there's an extra column even if you don't provide --details)
CLI parsing would require to: Extract the "thing" in the middle, parse out the relevant part (node_id and so on) to resolve and display it, and if --details was provided, take what's remaining of that "thing" (minus node_id and so on) and display it

Are we cool with that, @stevvooe @aaronlehmann @vieux @icecrime ?

dnephin · 2016-11-09T19:56:59Z

I think the logs are going to be pretty confusing without the task id. I would opt for always including a task id by default (maybe a flag to disable it).

aluzzardi · 2016-11-10T01:57:57Z

PTAL @stevvooe @vieux @stevvooe

There are a few things far from perfect (e.g. the reference docs are mostly copy paste from container logs) - could you please distinguish your comments between fix now and fix later? I'll make a follow-up with all less important fixes

aluzzardi · 2016-11-10T01:58:24Z

Stop gaps: non-follow is not supported, details is not supported

vieux · 2016-11-10T02:13:45Z

@aluzzardi I get

$ docker service logs pbf1duhxowx0
Error response from daemon: Bad parameters: Only follow mode is currently supported

stevvooe · 2016-11-10T02:13:59Z

cli/command/service/logs.go

+		context[parts[0]] = parts[1]
+	}
+
+	taskID, ok := context["task_id"]


We should use the convention of task.id we've used in labels. It might also be a good idea to namespace these com.docker.swarm.

aluzzardi · 2016-11-10T02:15:48Z

@vieux Yeah, unfortunately, only follow mode (-f) is supported

vieux · 2016-11-10T02:17:38Z

my bad @aluzzardi I read non-follow is supported, details is not supported

vieux · 2016-11-10T02:18:45Z

neat 👍

$ docker service create --replicas 3 redis
pbf1duhxowx0z66ma4y2oq1yb

$ docker service logs -f pbf1duhxowx0
jovial_bhabha.2.nsj65viveh8t@dev    | 1:C 10 Nov 02:13:28.502 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
jovial_bhabha.3.0emnvq7gr6iu@dev    | 1:C 10 Nov 02:13:28.435 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
jovial_bhabha.1.ex9jdel9krtn@dev    | 1:C 10 Nov 02:13:28.594 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
jovial_bhabha.1.ex9jdel9krtn@dev    |                 _._
jovial_bhabha.2.nsj65viveh8t@dev    |                 _._
jovial_bhabha.3.0emnvq7gr6iu@dev    |                 _._
jovial_bhabha.3.0emnvq7gr6iu@dev    |            _.-``__ ''-._
jovial_bhabha.1.ex9jdel9krtn@dev    |            _.-``__ ''-._
jovial_bhabha.2.nsj65viveh8t@dev    |            _.-``__ ''-._
jovial_bhabha.3.0emnvq7gr6iu@dev    |       _.-``    `.  `_.  ''-._           Redis 3.2.5 (00000000/0) 64 bit
jovial_bhabha.2.nsj65viveh8t@dev    |       _.-``    `.  `_.  ''-._           Redis 3.2.5 (00000000/0) 64 bit
jovial_bhabha.1.ex9jdel9krtn@dev    |       _.-``    `.  `_.  ''-._           Redis 3.2.5 (00000000/0) 64 bit
jovial_bhabha.1.ex9jdel9krtn@dev    |   .-`` .-```.  ```\/    _.,_ ''-._
jovial_bhabha.3.0emnvq7gr6iu@dev    |   .-`` .-```.  ```\/    _.,_ ''-._
jovial_bhabha.2.nsj65viveh8t@dev    |   .-`` .-```.  ```\/    _.,_ ''-._
jovial_bhabha.2.nsj65viveh8t@dev    |  (    '      ,       .-`  | `,    )     Running in standalone mode
jovial_bhabha.1.ex9jdel9krtn@dev    |  (    '      ,       .-`  | `,    )     Running in standalone mode
jovial_bhabha.3.0emnvq7gr6iu@dev    |  (    '      ,       .-`  | `,    )     Running in standalone mode
jovial_bhabha.3.0emnvq7gr6iu@dev    |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
jovial_bhabha.2.nsj65viveh8t@dev    |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
jovial_bhabha.1.ex9jdel9krtn@dev    |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
jovial_bhabha.2.nsj65viveh8t@dev    |  |    `-._   `._    /     _.-'    |     PID: 1
jovial_bhabha.3.0emnvq7gr6iu@dev    |  |    `-._   `._    /     _.-'    |     PID: 1
jovial_bhabha.3.0emnvq7gr6iu@dev    |   `-._    `-._  `-./  _.-'    _.-'
jovial_bhabha.1.ex9jdel9krtn@dev    |  |    `-._   `._    /     _.-'    |     PID: 1
jovial_bhabha.2.nsj65viveh8t@dev    |   `-._    `-._  `-./  _.-'    _.-'
jovial_bhabha.2.nsj65viveh8t@dev    |  |`-._`-._    `-.__.-'    _.-'_.-'|
jovial_bhabha.3.0emnvq7gr6iu@dev    |  |`-._`-._    `-.__.-'    _.-'_.-'|
jovial_bhabha.1.ex9jdel9krtn@dev    |   `-._    `-._  `-./  _.-'    _.-'
jovial_bhabha.1.ex9jdel9krtn@dev    |  |`-._`-._    `-.__.-'    _.-'_.-'|
jovial_bhabha.2.nsj65viveh8t@dev    |  |    `-._`-._        _.-'_.-'    |           http://redis.io
jovial_bhabha.3.0emnvq7gr6iu@dev    |  |    `-._`-._        _.-'_.-'    |           http://redis.io
jovial_bhabha.3.0emnvq7gr6iu@dev    |   `-._    `-._`-.__.-'_.-'    _.-'
jovial_bhabha.1.ex9jdel9krtn@dev    |  |    `-._`-._        _.-'_.-'    |           http://redis.io
jovial_bhabha.2.nsj65viveh8t@dev    |   `-._    `-._`-.__.-'_.-'    _.-'
jovial_bhabha.2.nsj65viveh8t@dev    |  |`-._`-._    `-.__.-'    _.-'_.-'|
jovial_bhabha.3.0emnvq7gr6iu@dev    |  |`-._`-._    `-.__.-'    _.-'_.-'|
jovial_bhabha.1.ex9jdel9krtn@dev    |   `-._    `-._`-.__.-'_.-'    _.-'
jovial_bhabha.2.nsj65viveh8t@dev    |  |    `-._`-._        _.-'_.-'    |
jovial_bhabha.2.nsj65viveh8t@dev    |   `-._    `-._`-.__.-'_.-'    _.-'
jovial_bhabha.3.0emnvq7gr6iu@dev    |  |    `-._`-._        _.-'_.-'    |
jovial_bhabha.1.ex9jdel9krtn@dev    |  |`-._`-._    `-.__.-'    _.-'_.-'|
jovial_bhabha.1.ex9jdel9krtn@dev    |  |    `-._`-._        _.-'_.-'    |
jovial_bhabha.2.nsj65viveh8t@dev    |       `-._    `-.__.-'    _.-'
jovial_bhabha.3.0emnvq7gr6iu@dev    |   `-._    `-._`-.__.-'_.-'    _.-'
jovial_bhabha.3.0emnvq7gr6iu@dev    |       `-._    `-.__.-'    _.-'
jovial_bhabha.1.ex9jdel9krtn@dev    |   `-._    `-._`-.__.-'_.-'    _.-'
jovial_bhabha.2.nsj65viveh8t@dev    |           `-._        _.-'
jovial_bhabha.2.nsj65viveh8t@dev    |               `-.__.-'
jovial_bhabha.3.0emnvq7gr6iu@dev    |           `-._        _.-'
jovial_bhabha.1.ex9jdel9krtn@dev    |       `-._    `-.__.-'    _.-'
jovial_bhabha.1.ex9jdel9krtn@dev    |           `-._        _.-'
jovial_bhabha.2.nsj65viveh8t@dev    |
jovial_bhabha.3.0emnvq7gr6iu@dev    |               `-.__.-'
jovial_bhabha.3.0emnvq7gr6iu@dev    |
jovial_bhabha.1.ex9jdel9krtn@dev    |               `-.__.-'
jovial_bhabha.2.nsj65viveh8t@dev    | 1:M 10 Nov 02:13:28.508 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
jovial_bhabha.3.0emnvq7gr6iu@dev    | 1:M 10 Nov 02:13:28.438 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
jovial_bhabha.1.ex9jdel9krtn@dev    |
jovial_bhabha.2.nsj65viveh8t@dev    | 1:M 10 Nov 02:13:28.508 # Server started, Redis version 3.2.5
jovial_bhabha.2.nsj65viveh8t@dev    | 1:M 10 Nov 02:13:28.508 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
jovial_bhabha.3.0emnvq7gr6iu@dev    | 1:M 10 Nov 02:13:28.439 # Server started, Redis version 3.2.5
jovial_bhabha.1.ex9jdel9krtn@dev    | 1:M 10 Nov 02:13:28.597 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
jovial_bhabha.1.ex9jdel9krtn@dev    | 1:M 10 Nov 02:13:28.597 # Server started, Redis version 3.2.5
jovial_bhabha.2.nsj65viveh8t@dev    | 1:M 10 Nov 02:13:28.508 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
jovial_bhabha.3.0emnvq7gr6iu@dev    | 1:M 10 Nov 02:13:28.439 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
jovial_bhabha.2.nsj65viveh8t@dev    | 1:M 10 Nov 02:13:28.508 * The server is now ready to accept connections on port 6379
jovial_bhabha.1.ex9jdel9krtn@dev    | 1:M 10 Nov 02:13:28.597 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
jovial_bhabha.1.ex9jdel9krtn@dev    | 1:M 10 Nov 02:13:28.597 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
jovial_bhabha.3.0emnvq7gr6iu@dev    | 1:M 10 Nov 02:13:28.439 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
jovial_bhabha.3.0emnvq7gr6iu@dev    | 1:M 10 Nov 02:13:28.439 * The server is now ready to accept connections on port 6379
jovial_bhabha.1.ex9jdel9krtn@dev    | 1:M 10 Nov 02:13:28.597 * The server is now ready to accept connections on port 6379

stevvooe · 2016-11-10T02:27:38Z

LGTM

I think we can tweak the output quite a bit but let's do that in a follow up.

tonistiigi · 2016-11-10T18:38:43Z

daemon/cluster/cluster.go

@@ -1121,6 +1130,82 @@ func (c *Cluster) RemoveService(input string) error {
 	return nil
 }

+// ServiceLogs collects service logs and writes them back to `config.OutStream`
+func (c *Cluster) ServiceLogs(ctx context.Context, input string, config *backend.ContainerLogsConfig, started chan struct{}) error {
+	if !c.isActiveManager() {


Need to get a readlock for this. You can release it before the long running operations.

vieux · 2016-11-10T18:53:10Z

It would be nice to get it for 1.13, but since it's still missing a few feature and after some discussion among the engine maintainers, we think it's best if you can put this in experimental only @al

Plumbed the executor to the container logs backend. Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>

aluzzardi · 2016-11-10T21:48:05Z

Moved to experimental

@vieux PTAL

vieux · 2016-11-10T22:03:28Z

LGTM

bvis · 2016-11-25T02:28:12Z

Hi,

I tried this feature in my staging swarm cluster and it didn't worked as expected. I reproduced the error here: http://play-with-docker.com/p/f0020045-d8e8-4bf0-a86c-55f3a25cd4dd

I left some messages related with this in the swam slack room.

If you want me to create an issue just tell it to me.

aaronlehmann · 2016-11-28T19:19:28Z

@bvis: I wasn't able to load your link. I get the error message "session not found".

Opening an issue with some details sounds like a good next step.

bvis · 2016-11-29T02:18:10Z

@aaronlehmann Here you are: #28915

service logs

aluzzardi added the status/1-design-review label Nov 5, 2016

GordonTheTurtle added the status/0-triage label Nov 5, 2016

aaronlehmann removed the status/0-triage label Nov 5, 2016

aaronlehmann reviewed Nov 5, 2016

View reviewed changes

aluzzardi added rebuild/experimental and removed rebuild/experimental labels Nov 7, 2016

aluzzardi added this to the 1.13.0 milestone Nov 8, 2016

aluzzardi added the area/swarm label Nov 8, 2016

stevvooe reviewed Nov 8, 2016

View reviewed changes

dongluochen reviewed Nov 9, 2016

View reviewed changes

aluzzardi changed the title ~~[wip] service logs~~ service logs Nov 10, 2016

stevvooe reviewed Nov 10, 2016

View reviewed changes

tonistiigi reviewed Nov 10, 2016

View reviewed changes

aluzzardi added 4 commits November 10, 2016 12:02

cluster executor: Logs retrieving support.

0ec6865

Plumbed the executor to the container logs backend. Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>

api: Service Logs support

819d015

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>

cli: docker service logs support

c7995fd

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>

service logs reference documentation

97e4bff

Signed-off-by: Andrea Luzzardi <aluzzardi@gmail.com>

vieux added status/4-merge and removed status/1-design-review labels Nov 10, 2016

thaJeztah added impact/api impact/changelog impact/cli kind/experimental docs/revisit labels Nov 10, 2016

aluzzardi merged commit 5f9fe54 into moby:master Nov 10, 2016

aluzzardi deleted the service-logs branch November 10, 2016 23:13

thaJeztah mentioned this pull request Nov 15, 2016

[1.13.0-rc1] Not possible to get DNS name for service tasks through "service ps" #28430

Closed

bvis mentioned this pull request Nov 22, 2016

swarm services logs portainer/portainer#334

Closed

vdemeester removed the docs/revisit label Nov 23, 2016

This was referenced Nov 23, 2016

Implement "docker logs" for services #24812

Closed

docs: document that docker logs is not supported for services #24813

Closed

albers mentioned this pull request Jan 27, 2017

Add bash completion for docker service logs #30510

Merged

sdurrheimer mentioned this pull request Feb 12, 2017

Add zsh completion for 'docker service logs' command #30941

Merged

dnephin pushed a commit to dnephin/docker that referenced this pull request Apr 17, 2017

Merge pull request moby#28089 from aluzzardi/service-logs

79e1fd0

service logs

service logs #28089

service logs #28089

Conversation

aluzzardi commented Nov 5, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronlehmann Nov 5, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aluzzardi commented Nov 8, 2016 • edited Loading

vieux commented Nov 9, 2016

stevvooe commented Nov 9, 2016

aluzzardi commented Nov 9, 2016 • edited Loading

stevvooe commented Nov 9, 2016

vieux commented Nov 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aluzzardi commented Nov 9, 2016

dnephin commented Nov 9, 2016

aluzzardi commented Nov 10, 2016

aluzzardi commented Nov 10, 2016

vieux commented Nov 10, 2016

Choose a reason for hiding this comment

aluzzardi commented Nov 10, 2016

vieux commented Nov 10, 2016

vieux commented Nov 10, 2016

stevvooe commented Nov 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vieux commented Nov 10, 2016

aluzzardi commented Nov 10, 2016

vieux commented Nov 10, 2016

bvis commented Nov 25, 2016

aaronlehmann commented Nov 28, 2016

bvis commented Nov 29, 2016

aluzzardi commented Nov 5, 2016 •

edited

Loading

aaronlehmann Nov 5, 2016 •

edited

Loading

aluzzardi commented Nov 8, 2016 •

edited

Loading

aluzzardi commented Nov 9, 2016 •

edited

Loading