Run build jobs from queue in parallel on a system with large number of cores

[davidcoulson]

Describe the enhancement

Allow the dashboard (local or remote) to build a specified number of devices in parallel - Especially for remote build servers it would be great to specify 4,6,8 builds on that host in parallel rather than having to spin up multiple instances of the container and farm the builds out to all of them.

Use cases

This would reduce the amount of time it takes to build a large number of devices where the system has lots of CPU capacity to run multiple builds at the same time.

Visual examples

No response

Anything else?

No response

[ssieb]

There is no benefit to parallel compiling on a single system. The compile will use all the cores you give it.

[ssieb]

I think the uploads are only done from the main dashboard, but I don't see any reason they couldn't be in parallel. That would be a separate feature request. It's generally not necessary though. Why is that one taking so long?

[davidcoulson]

I think the uploads are only done from the main dashboard, but I don't see any reason they couldn't be in parallel. That would be a separate feature request. It's generally not necessary though. Why is that one taking so long?

Crappy wifi - 2.4g in my house is a disaster. some take 10s, some take 300.

[ssieb]

You should be able to run multiple builder instances on the same system if they are on different ports.

[davidcoulson]

You should be able to run multiple builder instances on the same system if they are on different ports.

Yep I'm running four instances of builder on the same box.

[ssieb]

ok, I thought that's what wasn't working for you. But now I re-read it again and you wanted a single builder to run multiple parallel compiles. That would add a lot of complication for a very niche case.

[bdraco]

Closing the loop on this one, since it ended up covering a few different things:

Parallel uploads were split out to #3781 and are implemented as of 1.6.6b1. The upload lane now runs up to 3 flashes concurrently, so fast OTAs no longer serialize behind a device on slow wifi (esphome/device-builder#2192).

Parallel builds across build servers already works today: the dashboard dispatches waiting compiles to every free paired server concurrently, so adding more servers scales your build throughput right now.

Multiple concurrent builds on a single host doesn't buy what you'd hope. As ssieb said, a single compile already uses all the cores you give it, so overlapping builds on the same machine mostly split the same CPU rather than finishing sooner, while multiplying memory and disk pressure.

Auto-authenticating a build container (skipping the pairing flow) is a good idea but really its own feature with its own trust and security tradeoffs. Could you open a separate discussion for it so it doesn't get buried here? That way it can be tracked and discussed on its own merits.

With the uploads shipped and multi-server builds already in place, I'm going to close this one out. Thanks for the detailed feedback.

[bdraco]

Reopened based on https://github.com/orgs/esphome/discussions/3779#discussioncomment-17685654 since the use case is for systems with 128 cores (where the compiler actually use more than ~20 cpus at once)

[bdraco]

Concurrent compile jobs within a single builder are unlikely to happen: every framework in the toolchain (PlatformIO included) would need to support running side by side, and there are locking issues and design challenges that make that unrealistic. We tried it in our CI and it led to random failures and hard-to-diagnose problems because the builds step on each other.

The best path right now is to run multiple remote headless builders in containers on the same host. Each container runs in isolation, which sidesteps the per-framework design challenges, and the dashboard dispatches queued compiles to paired builders concurrently, so pairing 4 builders on one box gets you 4 parallel builds without managing the queue yourself.

[davidcoulson]

Is there an environment variable or parameter that enables --remote-build-only via docker-compose? Rather than replacing the whole command line :)

[bdraco]

esphome/device-builder#2515 will raise it to 8 at a time once the changes in esphome are done

[bdraco]

esphome/device-builder#2518 reached 9 slots with some more optimizations