Stagger image builds instead of building all of them at once

[benmandrew]

Currently we build the images on a weekly schedule, rebuilding all of them at once, which creates a large spike of demand on the cluster. It may be preferable to spread the builds out over the week.

We may wish to preserve the building-all-at-once behaviour for startup of the service.

[shonfeder]

It may be preferable to spread the builds out over the week

Could you elaborate a bit on the advantages of spreading them out? I can imagine some, but it would help us prioritize this issue to spell out the expected benefits :)

[mtelvers]

We often see failures when pushing to Docker Hub which appear to be related to multiple simultaneous pushes. Perhaps there is a rate or bandwidth limit? The failures need to be resubmitted manually. See #211

[shonfeder]

We often see failures when pushing to Docker Hub which appear to be related to multiple simultaneous pushes.

Ah ha! That is a compelling motive for sure. But also automating the retries (perhaps after a delay?) also seems like it should be helpful per se, and maybe that should be tackled first?

[shonfeder]

Rather than introducing scheduling changes to stagger the builds, @mtelvers suggested today that we could just build each OS in sequence (and the version and architecture matrix under each OS will then fan out and take a good amount of our build capacity for most OSs).

This will reduce the pressure on the system and instead carry out the builds in a slower (likely steadier) stream.

I'm provisionally in favor of this approach, as it is a simpler way of achieving the desired aim here, and seems to have nice properties overall.