docker events command doesn't work with swarm

[garagatyi]

I tried to run swarm in accordance to docs at https://docs.docker.com/swarm/install-w-machine/.
Docker events doesn't return anything. Here is how I check it:

~$ docker -H tcp://192.168.99.101:3376 info
Containers: 12
Images: 5
Storage Driver: 
Role: primary
Strategy: spread
Filters: affinity, health, constraint, port, dependency
Nodes: 2
 swarm-agent-00: 192.168.99.102:2376
  └ Containers: 6
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.022 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.0.9-boot2docker, operatingsystem=Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015, provider=virtualbox, storagedriver=aufs
 swarm-master: 192.168.99.101:2376
  └ Containers: 6
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.022 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.0.9-boot2docker, operatingsystem=Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015, provider=virtualbox, storagedriver=aufs
Execution Driver: 
Kernel Version: 
Operating System: 
CPUs: 2
Total Memory: 2.043 GiB
Name: 729aa5032753
ID: 
Http Proxy: 
Https Proxy: 
No Proxy: 
$ time docker -H tcp://192.168.99.101:2376 events --since '2015-09-06T12:15' --until '2015-09-07T12:15' --filter 'event=start'
2015-09-08T13:19:54.000000000+03:00 729aa5032753c8181a0d3d12fa0d0a3f4017893da450655fa7daa732d70840d8: (from swarm:latest) start

2015-09-08T13:19:54.000000000+03:00 794093de3c6f2e266edce60e1a2f5ebf99dc8727e3755caed2c812760a145b7b: (from swarm:latest) start

2015-09-08T14:07:52.000000000+03:00 740df9cde5d730c0ea3db21fe1864da5ea1b5c4e8e3c31eb5fb8c8a120a8bdf5: (from ubuntu) start

2015-09-08T14:08:01.000000000+03:00 53d8cba4876f924c026029a19e9c3bca2641eefee6734b6ce82f0dc88c5f5731: (from ubuntu) start

2015-09-08T14:13:18.000000000+03:00 78590a882efec1a5d79db4f9e11706f374023220fd3d4ec2cc6cbd1bc5f4a037: (from ubuntu) start

2015-09-08T14:29:38.000000000+03:00 cb0dfbd8a42b37559397faa149b956887fc2cde9ed60faf620bc77fa846e5c32: (from hello-world) start


real    0m0.178s
user    0m0.152s
sys 0m0.008s
~$ time docker -H tcp://192.168.99.101:3376 events --since '2015-09-06T12:15' --until '2015-09-07T12:15' --filter 'event=start'
^C
real    0m17.496s
user    0m0.152s
sys 0m0.004s
~$ 

Do you have any thoughts?

[abronan]

Hi @garagatyi, thanks for reporting. It just seems like events from swarm only gets the event stream and does not support the request parameters: since, until and filter. We need to add those so that events is fully supported.

[MHBauer]

Looking at the code, it's set up to stream the events as they come in. Getting events in the past, using --since would mean going and talking to every client, and then collating them. Or the events could be cached from each daemon, as they only keep the most recent 64 events (see here), but you'd have to ask for all the events up front when connecting.

--until should be pretty easy. Ignore events until a specific time.

I do not have any ideas for --filter.

[vieux]

@MHBauer can you try to implement until and then since in a separate PR ?

Thanks!

[MHBauer]

Yes, I will give until a shot.

[vieux]

Thank you

[vieux]

@MHBauer can you look at since :)

[MHBauer]

@vieux I can try. How do you think I should approach the problem? Since (ha!) we have to go back in time to get previously existing events, it doesn't fit with the way the events are currently set up, which works by streaming the events live from the engines.

[aluzzardi]

Nah, I don't think we can do since with the current architecture - there's no way to go reliably back in time, unless we store events, but I don't think we want to get in that business.

[vieux]

Right, we don't want to store events but I think the engine stores a few events (maybe up to 64) so we could try to support since and call since on the underlying engines.

Not sure it's useful though

[aluzzardi]

Oh yeah, if there's a limit, then we could store those even in memory in our events struct

[doronl]

Hi ,

I opened issue #1504 which relates to this subject... is this scheduled to be handled in next version? do you need more input?

Thanks,
Doron

[MHBauer]

Sorry, I missed the conversation. I'll take a look at the events struct.

[garagatyi]

@aluzzardi @vieux I don't think 64 events pool will work when a lot of containers are working in cluster. You showed 50K of containers at DockerCon. In this case 64 events worth nothing.

[doronl]

@garagatyi as far as I understand its 64 events per host (node) and not per container, in DockerCon there were 1000 hosts (an extreme case), is storing 64K events in cache too much memory wise? (again this is not a common case)… BTW I don’t know the memory footprint of a single event…
Another option is that the Swarm manager will just loop all nodes, query them and stream the results back to the client as it progresses on the loop, the down side is that if there are many clients querying the events it could be a performance issue (NumberOfCalls = NumberOfNodes*NumberOfClients).
A 3rd option is that there will be support for a single secondary storage that all events data will be forward to and clients may query it directly or the secondary storage could be an embedded part of the Swarm image and clients may use the Regular Remote API as today since the Swarm Manager will forward the query to the secondary storage, it will have to be cyclic storage in some manner to avoid running out of disk space.
I agree it’s not a trivial request but IMO this is an important use case, cluster management wise, to have a single point of access to query all events.

Options summary (as I see it) are:

1. In memory cache
2. loop + streaming
3. Secondary storage
   a. External (user managed)
   b. Embedded (self-managed, transparent)

[garagatyi]

@doronl Yes, I misunderstand @vieux, thank you.

is storing 64K events in cache too much memory wise?

I believe it's too much. But maybe @vieux or @aluzzardi have another opinion.
AFAIK swarm doesn't stream events when clients don't ask for that, so just storing on streaming wouldn't work.

loop + streaming

It can be improved by loop storage, so once client ask for events Swarm ask for events all nodes and save it in loop storage (on disk or memory) . When another client ask for events Swarm returns events from this storage.
But in cases with caching (on disk or in memory) there are several corner cases.
What if new nodes were added after events from other nodes were cached in storage? If client asks Swarm in this case Swarm should ask for events on new nodes.
What if there are pauses between session of events streaming and not all events were cached. If client asks with --since Swarm should recall events that was fired between streaming sessions. And so on.
So looks like this solution is too complicated (sorry for the long comment, just thinking loudly).

So I think that the easiest solution to implement will be loop + streaming without any cache.

[doronl]

Why was this closed? did you implement the missing ‘since’ events functionality in the swarm API? as I mentioned in #1504

Thanks

[vieux]

Sorry @doronl we just merged a PR related to events but you are right it doesn't fix the --since issue.

Every engine stores 64 events in memory but it's true that swarm could store more, I'm going to try to PoC something.

[doronl]

So is the PR opened somewhere? or should #1504 be reopened? … just to keep track of things

Thanks

[garagatyi]

I think that this issue should be closed. The purpose of it is to find explanation why origin command doesn't show events. Now we know that this is because of --since option.
I, definitely,would like to have working --since with swarm, but it probably should be requested in separate issue.
@vieux @doronl WDYT?

[funkytaco]

Any updates on this?

[nishanttotla]

Closing because docker events is supported by Swarm.