Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't process container event if host is removed #1445

Closed
cjellick opened this issue Jul 9, 2015 · 3 comments
Closed

Don't process container event if host is removed #1445

cjellick opened this issue Jul 9, 2015 · 3 comments
Assignees
@sangeethah
Labels
area/native-docker kind/bug Issues that are defects reported by users or that we know have reached a real release
Milestone
Milestone 7/29/2015

Comments

@cjellick
Copy link

cjellick commented Jul 9, 2015

@ibuildthecloud and @cloudnautique observed a large number (20k+) of unprocessed ContainEventCreates in the load test environment. The number of events was likely a normal side effect of the things Bill was doing while setting up the environment. But now that the environment is stable, those 20k processes are still stuck because of a bug that cause ContainerEventCreates to hang if the associated host is removed.

The fix is to not process the event if the associated host is removed.
@cjellick cjellick self-assigned this Jul 9, 2015
@cjellick cjellick added this to the Milestone 7/15/2015 milestone Jul 9, 2015
cjellick pushed a commit to cjellick/cattle that referenced this issue Jul 9, 2015
Dont process containerEvent if host is removed.
3b52060 
Addresses rancher/rancher#1445.
If a host is removed and we try to process a container event, that
event will hang forever and a huge queue will build up for it in
processInstances. To avoid this, immediately exit if host is removed.
@cjellick cjellick mentioned this issue Jul 9, 2015
Merged
@cjellick
Copy link
Author

cjellick commented Jul 9, 2015

Fix isn't merged yet, but will be soon, So, @sangeethah I can't think of a good automated and reproducable validation test that we could write for this. However, our load test environment has roughly 25 thousands such containerEvents, so it will be a good test bed for this fix. Think that will suffice?

@cjellick cjellick added the kind/bug Issues that are defects reported by users or that we know have reached a real release label Jul 9, 2015
@sangeethah
Copy link
Contributor

Since this issue was reported from load test environment , it would be sufficient if we can validate this issue in the same set up.

@cjellick cjellick assigned sangeethah and unassigned cjellick Jul 28, 2015
@cloudnautique
Copy link
Contributor

Verified in loadtest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sangeethah sangeethah

Labels
area/native-docker kind/bug Issues that are defects reported by users or that we know have reached a real release
Projects
None yet
Milestone
Milestone 7/29/2015
Development

No branches or pull requests
4 participants
@cloudnautique @cjellick @sangeethah @will-chan