Cannot run flows if docker agent is started with the network option

[ManuelMourato25]

Description

Every time I try to run a flow using a docker agent with the network option specified, like so:

# Custom bridge network
prefect agent start docker -l docker-execs-test --show-flow-logs --network test-network
# Network used by Prefect services
prefect agent start docker -l docker-execs-test --show-flow-logs --network prefect-server
# Host network
prefect agent start docker -l docker-execs-test --show-flow-logs --network host

I get the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_outp ut
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fb645a37cf8>, 'Connection to host.docker.internal timed out. (connect timeout=30)')

Reproduction

1. In order to reproduce, create any flow and store it in a docker storage like so:

# Define a flow and save it in a test_flow variable
....
task_sequence = [task1,task2]
test_flow.chain(*task_sequence)

storage = Docker(image_name="flows-storage", image_tag="v1.0",local_image=True)
storage.build()
test_flow.storage=storage
test_flow.register(build=False,labels=['docker-execs-test'])

The image is created with no issues.

2. Then start a docker agent like so:

prefect agent start docker -l docker-execs-test --show-flow-logs --network host

3. Go to the UI and run the registered flow

4. You should see the behaviour mentioned above.

Environment

{
"config_overrides": {},
"env_vars": [
"PREFECT__CLOUD__AGENT__LEVEL"
],
"system_information": {
"platform": "Linux-4.15.0-91-generic-x86_64-with-debian-buster-sid",
"prefect_version": "0.10.7",
"python_version": "3.6.10"
}
}

[joshmeek]

Thanks for opening @ManuelMourato25!

[joshmeek]

Hey @ManuelMourato25 is this still an issue for you?

[ericwong3]

@joshmeek apparently the issue is still here, experienced similar issue today.

Same issue, when an agent is started with prefect agent docker start --network host, and a flow is executed on that agent, the flow will fail and the flow log will say requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.18.61', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8b253f8f50>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

[zanieb]

@ericwong3 Are you running Prefect Server? If so, are you also running it in that network? This looks like a different issue than above, here it can't connect to the backend API.

[ericwong3]

Yes I am running prefect server. The server and agent is started separately. The server is started with prefect server start*, and the client is started on the same host with prefect agent docker start --network host --api http://192.168.18.61:4200.

192.168.18.61 is the docker host's IP, so a container is running with bridge or host network should both be able to connect. Except it only starts without --network host, and removing it does make the issue disappear.

* I did prefect server config --use-volume > docker-compse.yml, changed pg_data mount location, and did docker-compose up, guess that is no different from doing prefect server start in relation to this issue.

[zanieb]

I'm going to track a fix to this in #4840 — the issue is that the "host" network requires a change in container network mode as well.