Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
time-box the docker client to avoid ever getting stuck #773
It seems that we can get into an odd state when contacting the docker daemon. Even with the appropriate timeouts (default apache client in this case) set, we can still get into a situation where we try to contact the docker daemon then hang there forever. While the root problem lies in the docker daemon, it shouldn't stop the executor from being able to shut down properly.
This optionally creates a time limit for all calls using the docker daemon to avoid things hanging and causing an executor process that will simply wait forever doing nothing.
spotify docker client is just using apache http client underneath, default timeouts for connect and read get set to 5 and 30 seconds. I think the current bug with the docker daemon is that it keeps the connection open in such a way that the timeouts do no get hit, thus the usage of TimeLimiter here instead (docker cli calls and calls to docker daemon from any source/client hang as well)
The bug is a rare case and the purpose of the PR is more to limit our executor from being pinned by it. If we can't launch something because docker is in a bad state, call it failed and move on.