Docker Agent: Passing custom parameters when creating docker.APIClient

[hiendang]

Current behavior

prefect docker agent is using default parameters to create an instance of docker.APIClient. This makes it impossible to customize the docker client. Things such as to increase the timeout of docker client cannot be done.

https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/docker/agent.py#L186

Proposed behavior

We should have a way to pass custom parameters to when creating an instance of docker.APIClient.
Either as a dictionary or as a json config file

Example

When the somehow get overloaded, docker daemon failed to create a container within 60 seconds, docker.APIClient will then return a timeout error similar to docker/compose#3927. In my case increase the timeout from 60 to 120 seconds would mitigate this problem.

[jcrist]

Would exposing only the timeout be sufficient. Exposing the full kwargs options through the CLI is tricky to support in an ergonomic way, exposing individual parameters is much simpler.

I'd write this as

$ prefect agent docker start --docker-client-timeout 120

[hiendang]

yes I agreed, it would be sufficient in this case. Passing full kwargs would be messy and maybe only timeout parameter is needed at the end

[zanieb]

It would also be nice to be able to set credstore_env but I can't remember where the thread was where a user needed it.

[jcrist]

It would also be nice to be able to set credstore_env but I can't remember where the thread was where a user needed it.

I'm going to leave that one out for now, if a user needs it they can elevate it to an issue later.