ConnectTimeout in django_docker_cleanup

[hackdna]

• Specific code commit: c96b8f5
• Version of the web browser and OS: N/A
• Environment where the error occurred: AWS

Steps to reproduce

Launch a stack without a Docker host

Observed behavior

Celery error log:

2018-09-11 17:09:20,242 ERROR    celery.worker.job:282 log() - Task tool_manager.tasks.django_docker_cleanup[3f01b4b1-0d9f-4103-9069-de4b38f8bc6b] raised unexpected: ConnectTimeout(MaxRetryError('None: Max retries exceeded with url: /v1.35/containers/json?all=1&limit=-1&trunc_cmd=0&size=0 (Caused by None)',),)
Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/srv/refinery-platform/refinery/tool_manager/tasks.py", line 12, in django_docker_cleanup
    settings.DJANGO_DOCKER_ENGINE_SECONDS_INACTIVE
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/docker_utils.py", line 123, in purge_inactive
    self._purge(seconds=seconds)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/docker_utils.py", line 107, in _purge
    for container in self.list({'label': label} if label else {}):
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/docker_utils.py", line 86, in list
    return self._containers_manager.list(filters)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/container_managers/docker_engine.py", line 182, in list
    return self._containers_client.list(all=True, filters=filters)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/models/containers.py", line 899, in list
    since=since)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/api/container.py", line 205, in containers
    res = self._result(self._get(u, params=params), True)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/api/client.py", line 198, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/adapters.py", line 419, in send
    raise ConnectTimeout(e, request=request)
ConnectTimeout: None: Max retries exceeded with url: /v1.35/containers/json?all=1&limit=-1&trunc_cmd=0&size=0 (Caused by None)
2018-09-11 17:09:50,263 ERROR    celery.worker.job:282 log() - Task tool_manager.tasks.django_docker_cleanup[1a7c8558-ed22-43e4-a5f7-e77303a64fb3] raised unexpected: ConnectTimeout(MaxRetryError('None: Max retries exceeded with url: /v1.35/containers/json?all=1&limit=-1&trunc_cmd=0&size=0 (Caused by None)',),)
Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/srv/refinery-platform/refinery/tool_manager/tasks.py", line 12, in django_docker_cleanup
    settings.DJANGO_DOCKER_ENGINE_SECONDS_INACTIVE
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/docker_utils.py", line 123, in purge_inactive
    self._purge(seconds=seconds)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/docker_utils.py", line 107, in _purge
    for container in self.list({'label': label} if label else {}):
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/docker_utils.py", line 86, in list
    return self._containers_manager.list(filters)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/django_docker_engine/container_managers/docker_engine.py", line 182, in list
    return self._containers_client.list(all=True, filters=filters)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/models/containers.py", line 899, in list
    since=since)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/api/container.py", line 205, in containers
    res = self._result(self._get(u, params=params), True)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/docker/api/client.py", line 198, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/ubuntu/.virtualenvs/refinery-platform/local/lib/python2.7/site-packages/requests/adapters.py", line 419, in send
    raise ConnectTimeout(e, request=request)
ConnectTimeout: None: Max retries exceeded with url: /v1.35/containers/json?all=1&limit=-1&trunc_cmd=0&size=0 (Caused by None)

Expected behavior

Error handled, no tracebacks

[scottx611x]

@hackdna Do we ever deploy without the docker host? If being able to run VisualizationTools when a Refinery instance is deployed is still a story that we want to tell then I'm not sure if this is a scenario that we should be worrying about at this point in time.

[hackdna]

Docker host could crash, lose network connection or be terminated by AWS.

[jkmarx]

We don't have this task anymore.

[scottx611x]

PR: c313a98 removed this task