Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

events: close channel explicitly #1254

Merged
merged 1 commit into from Jul 27, 2017
Merged

Conversation

stevvooe
Copy link
Member

@stevvooe stevvooe commented Jul 27, 2017

Signed-off-by: Stephen J Day stephen.day@docker.com

Signed-off-by: Stephen J Day <stephen.day@docker.com>
@codecov-io
Copy link

Codecov Report

Merging #1254 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1254   +/-   ##
=======================================
  Coverage   28.02%   28.02%           
=======================================
  Files          29       29           
  Lines        3033     3033           
=======================================
  Hits          850      850           
  Misses       2033     2033           
  Partials      150      150

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4349ef...a1f7509. Read the comment docs.

@estesp
Copy link
Member

estesp commented Jul 27, 2017

At some point, with this patch applied to master, my debug containerd log starts to fill with:

DEBU[0129] eventqueue: dropped event                     error="events: sink closed" event=&Envelope{Timestamp:2017-07-27 21:39:24.675133705 +0000 UTC,Namespace:,Topic:/containers/delete,Event:&google_protobuf1.Any{TypeUrl:types.containerd.io/containerd.services.events.v1.ContainerDelete,Value:[10 11 98 98 45 99 116 114 45 48 45 50 49],},} sink=map[C:0xc420bd7f20 closed:0xc420bd7f80]

After a few thousand lines of that, I end up back in a hung state :(

@stevvooe
Copy link
Member Author

@estesp Is the client hanging or containerd?

@estesp
Copy link
Member

estesp commented Jul 27, 2017

Just for posterity for this PR discussion thread (given we discussed on Slack): it is the client hanging because gRPC APIs on the server-side are blocked on something. I can create new containers via a secondary client, so containerd itself is not truly hung. But, the client which created the containers is waiting on deletes/waits to complete/return from the gRPC API, so is effectively hung as the server-side goroutines handling those APIs are not returning.

This PR does change the goroutine trace on the server-side. There are no more goroutines stuck in semacquire in the Queue.Close() code, so seems this fix is valid no matter what.

@dmcgowan
Copy link
Member

LGTM

@dmcgowan dmcgowan merged commit 4e299a9 into containerd:master Jul 27, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants