Running docker-compose up with lots of services makes it stall

[gerritdrost]

Description of the issue

Running docker-compose up on a docker-compose.yml with lots of services makes it stall. This seems to be related to the threading code in compose/parallel.py. The number of threads allowed to run simultaneously is not limited by default. If my analysis is correct, somehow, when too many threads are started, they enter a state in which they will never get the STOP-event in parallel.parallel_execute_iter, which is generated by parallel.feed_queue.

I managed to create a stable workaround in gerritdrost/compose, by limiting the number of concurrent threads.

I haven't been able to figure out what actually causes the problems. With correctly implemented threading code, the code should just work without these limits set.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.5, build unknown

Output of docker version

Docker version 19.03.9-ce, build 9d988398e7

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

version: '3.4'
services:
  service_000:
    command: sleep 30
    image: alpine:latest

  [...]

  service_299:
    command: sleep 30
    image: alpine:latest

Steps to reproduce the issue

As most concurrency problems, it is difficult to deterministically reproduce the problem. This is how I was able to reproduce it somewhat reliably:

1. Use this script to generate a docker-compose file python generate.py 100 > docker-compose.yml
2. docker-compose up -d
3. If it hangs: you just reproduced it. If it doesn't hang, docker-compose down and retry. Up the number of services if necessary. I used 300.

Observed result

docker-compose will hang while creating/starting the services. The only way to get out is Ctrl-C.

Expected result

All services start.

Additional information

Happens for me on Arch Linux but also for colleagues on various versions of Ubuntu Desktop.

[octaviancorlade]

The default limit on parallel operations seems to be 64.
In the example with 300 services docker-compose up adds 300 parallel calls to service.execute_convergence_plan to the queue. Each one of these also add at least one Service._execute_convergence_create.<locals>.create_and_start to the same queue, but that's already full so nothing else is executed.

COMPOSE_PARALLEL_LIMIT=301 docker-compose up -d (or more) should make it work

[asampal]

#7500 might be related.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.