Expose shm-size for Docker Agent

[kvnkho]

Current behavior

Reported by Nelson Griffits on Slack. The Docker image pulls and runs fine, but it uses PyTorch and he runs into some shared memory issues for the dataloader. Normally using Docker, people have solved this by passing arguments to increase the shared memory size to the docker run command.

This takes the form docker run -it --shm-size=256m

This is something that has to happen on the DockerRun or Agent (runtime). It seems like that we would ideally want to get it on the RunConfig so that it would be changed from flow to flow, but it is easier to expose this on the Agent at the moment.

Proposed behavior

The shm-size needs to be passed to Docker SDK's create_host_config. This then gets passed to create_container.

See the create_container call here

And the Docker SDK create_host_config takes in the shm_size here

Proposed behavior is that the agent takes is shm_size and it gets passed to create_host_config. Maybe we can even take the opportunity to expose more stuff?

This might need to be exposed on the CLI also.

Example

This would allow people to change the shared memory for containers run.

[ngriffiths13]

Thanks @kvnkho for creating this issue. I am happy to help do any work needed to get this into place.

[ngriffiths13]

I also agree that it would be ideal to pass the shared memory arguments to DockerRun instead of the agent. This is the first time I have not had enough memory for something, so it seems that it would be unnecessary to add shared memory to any flow that a Docker Agent ran.

[zanieb]

@ngriffiths13 If you want to add a host_config setting to the DockerRun that is merged (overwriting on key collision) with the default at

prefect/src/prefect/agent/docker/agent.py

Line 398 in d61fa6a

host_config = {

I'm happy to review it. I'd rather expose something generic than add obscure things like shm-size directly to the DockerRun object.

[ngriffiths13]

This issue will hopefully be fixed by this PR: #4777