Find a way to resume or re-run pending builds

[KrauseFx]

Right now, when a build runs, and the CI server either gets killed or restarted, a Build will stay stuck in pending for the rest of time of the universe. We gotta build something to see if the worker is actually still running, and if not, mark the build as failed.

e.g. https://github.com/KrauseFx/ci-config/commit/6167cdac97f96fc34d1277683652539701cbc638#diff-2d77cf50a00bf53694d062b016527aa0R3

[Sherlouk]

I wonder if this is relevant to this but another idea I had was the potential of, when retrying a build, if it could skip actions that have already happened successfully and somehow reuse the artifacts?

As we know Xcode tests have a tendency to randomly fail, but if I've already successfully built the app (using build-for-testing) then I shouldn't have to redo all of that to just retry the test action?

(Hope I explained that well enough!)

[taquitos]

I think that makes sense, but we have to be super judicious about what artifacts are reusable. Having a list of things that are safe to reuse might be the best way to go about this.

[KrauseFx]

Yeah, agreed here, it's very tricky to do it right, better to have a clean build and know for sure that things are working 👍