jobs/build: cut image building parallelism in half #519
Conversation
|
Hmm. Let's be a bit smarter here and put the ones that are long running together. Let me look at the run history and make a suggestion. |
I'm not sure it's worth trying to optimize this further vs e.g. pooling as suggested. The step itself only takes ~5 minutes which is tiny in comparison to a full pipeline run (~1h30m). It'd make adding new platforms trickier too. |
Yeah OK. I vaguely remembered the DigitalOcean one taking more than 10 minutes at one point in the past and one other one being long too and was going to try to put them in the same parallel bucket. Looking at the most recent runs it seems like the times are really too bad. |
We're seeing issues where the pipeline sometimes hits EAGAIN on `fork`: coreos/fedora-coreos-tracker#1186 This may have started happening more frequently after the recent addition of VirtualBox. Anyway, it's kinda silly to build that many images in parallel. There's diminishing returns and eventually a performance cost. So let's do it in two rounds instead. (Ideally instead it'd be a kind of task pool, but let's not try to implement that in Groovy.)
jlebon
force-pushed
the
pr/unparallelize-imgs
branch
from
88db43f
to
67a3273
Compare
|
Thanks for working on this. |
We're seeing issues where the pipeline sometimes hits EAGAIN on
fork:coreos/fedora-coreos-tracker#1186
This may have started happening more frequently after the recent
addition of VirtualBox.
Anyway, it's kinda silly to build that many images in parallel. There's
diminishing returns and eventually a performance cost. So let's do it in
two rounds instead. (Ideally instead it'd be a kind of task pool, but
let's not try to implement that in Groovy.)