New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

thread checker, case where docker container has already stopped #717

Merged
merged 3 commits into from Oct 8, 2015

Conversation

Projects
None yet
2 participants
@ssalinas
Member

ssalinas commented Oct 5, 2015

@tpetr found our docker threads issue. In the case where we inspect the container, but it has already stopped, the pid returns as 0. This is due to the docker daemon's response to inspect, not the client's processing. The wrapper checks every few seconds for the container to be running, so a badly timed thread check will throw up the error we saw with /proc/0/cgroup

So, I've added a check that if the pid is 0, we check that it is running. If it isn't running, we return 0. No need for throwing sentry errors when we are about to shut down anyways and nothing is really wrong

Show outdated Hide outdated ...a/com/hubspot/singularity/executor/SingularityExecutorThreadChecker.java
dockerPid = Optional.of(dockerClient.inspectContainer(containerName).state().pid());
int possiblePid = dockerClient.inspectContainer(containerName).state().pid();
if (possiblePid == 0) {
LOG.warn(String.format("Container %s has pid 0. Running: (%s). Will not try to get threads", containerName, dockerClient.inspectContainer(containerName).state().running()));

This comment has been minimized.

@tpetr

tpetr Oct 5, 2015

Member

i'd suggest "Container %s has pid %s (running: %s). Defaulting to 0 threads running."

@tpetr

tpetr Oct 5, 2015

Member

i'd suggest "Container %s has pid %s (running: %s). Defaulting to 0 threads running."

@ssalinas

This comment has been minimized.

Show comment
Hide comment
@ssalinas

ssalinas Oct 5, 2015

Member

@tpetr updated the message

Member

ssalinas commented Oct 5, 2015

@tpetr updated the message

@ssalinas ssalinas added the hs_staging label Oct 5, 2015

@tpetr tpetr added the hs_qa label Oct 5, 2015

@ssalinas ssalinas added the hs_stable label Oct 7, 2015

@tpetr tpetr added this to the 0.4.6 milestone Oct 7, 2015

tpetr added a commit that referenced this pull request Oct 8, 2015

Merge pull request #717 from HubSpot/docker_threads
thread checker, case where docker container has already stopped

@tpetr tpetr merged commit 280ef1b into master Oct 8, 2015

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@tpetr tpetr deleted the docker_threads branch Oct 8, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment