Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abort unsupported build steps #727

Merged
merged 4 commits into from Mar 27, 2020
Merged

Abort unsupported build steps #727

merged 4 commits into from Mar 27, 2020

Conversation

@edolstra
Copy link
Member

edolstra commented Mar 26, 2020

If we don't see machine that supports a build step for max_unsupported_time seconds, the step is aborted. The default is 0, which is appropriate for Hydra installations that don't provision missing machines dynamically.

edolstra added 2 commits Mar 26, 2020
If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.
@grahamc

This comment has been minimized.

Copy link
Member

grahamc commented Mar 26, 2020

Seems like a good idea. I wonder how we can setup alerting on hydra.n.o to alert us before they get aborted. We might have all the metrics we need already.

edolstra added 2 commits Mar 26, 2020
@edolstra

This comment has been minimized.

Copy link
Member Author

edolstra commented Mar 26, 2020

I've added a metric to keep track of the current number of unsupported runnable steps.

@grahamc

This comment has been minimized.

Copy link
Member

grahamc commented Mar 26, 2020

Looking ...

@grahamc

This comment has been minimized.

Copy link
Member

grahamc commented Mar 26, 2020

I wonder what it would take to export this information at the prometheus endpoint as well

sub prometheus :Local Args(0) {
though a method which got all of the gauge information by default would be nicest.

@grahamc

This comment has been minimized.

Copy link
Member

grahamc commented Mar 26, 2020

(btw I don't consider this blocking, we can re-export it elsewhere like most of these metrics)

@edolstra edolstra merged commit a7540b1 into flake Mar 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.