Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal] Failure avoidance for Current_docker.pull #181

Open
kit-ty-kate opened this issue Apr 26, 2020 · 1 comment
Open

[Proposal] Failure avoidance for Current_docker.pull #181

kit-ty-kate opened this issue Apr 26, 2020 · 1 comment
Labels
enhancement New feature or request

Comments

@kit-ty-kate
Copy link
Contributor

kit-ty-kate commented Apr 26, 2020

Currently when something fails during a Current_docker.pull its output will stay a failure until the next scheduled pull. For this particular command we probably don't want that. Here is my proposal to improve this case:

  1. If a pull fails, then just take its previous value.
  2. If there was no previous value, try again a minute later, then every hour if it still fails.
  3. Unless the service just started and the failed previous value is to be taken from the disk cache, in which case, try right away, then every hour if it still fails.
@talex5
Copy link
Contributor

talex5 commented May 7, 2020

More generally, we need an option to latch the old successful result of a step when a new build fails. We'll also need a way to report a metric for such hidden failures.

Retrying failed jobs on a schedule is also important, but should probably be a separate issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants