Add a timeout to every CI step to halt hung builds #906
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If a step in build hangs or takes an unusually long time, previously CI would let it continue, occupying machines forever. In lieu of a global timeout (buildkite/feedback#170, https://forum.buildkite.community/t/pipeline-timeouts/722), we can manually apply a timeout to every step, as a last resort to catch slow/hung builds. This uses the
timeout_in_minutes
optional attribute:Our steps currently range from 30 seconds to 8 minutes, so 30 minutes should be a safe "something serious is wrong" time-out.
See: #905