-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve retry workflow for release automation #606
Comments
Some approaches I tried out along the way: Self-rerunAdd an approval ("manual intervention") step, and if approved, have the pipeline run itself again with the same parameters it was run with. I got it roughly working: microsoft/go-infra@46d549f. However, I don't think it actually fits very well:
URL that runs a retry buildThere are a few issues and I didn't end up figuring this out:
Our own serviceWe could set up our own service (e.g. Azure functions) that has its own authentication to AzDO and has the authority to queue builds. This would be very flexible and could even help with retry scenarios that need something more complex than a straightforward retry. This can be taken a step further: we could have a service do our release instead of having a set of AzDO pipelines doing it. (At least, it could only call out to AzDO for steps that involve manipulating artifacts.) Reusing AzDO does have some benefits: it's relatively easy to see what release actions are going on and diagnose issues, and the security scope is fairly clear. It would take some time to get there with our own system. |
Retrying a build requires copying a value from the build log and pasting it into the correct field in the "Run build" form. The dev needs to remember which field to paste the value into while navigating the AzDO UI to get to the right place. Not a huge problem, and a mistake would normally result in a failed build at worst (because the values aren't interchangeable), but it's worth improving.
Ideally, the build would generate a link that the dev can click to run a new build with the retry parameter filled in. This may not be possible with a simple link: the AzDO API expects POST, the build parameters are passed in the request body, and I suspect AzDO has cross-site limitations that could also block us from simple workarounds like posting a static HTML page to blob storage.
An alternative could be a
releasego
command that the dev runs on their machine to run the build. This adds some complexity and potentially secret management to the dev environment.A simple mitigation would be to include the retry step number inside the value the dev copies. For example, instead of copying
abc12342
and remembering it goes into field3
, the dev would copy3 abc12342
and when pasting the value, it's obvious if the number doesn't match the prompt, and it's easy to ctrl-z and paste into the right one if it doesn't match. The build process then ensures the3
is present in the value and fails immediately upon mismatch, so if the dev misses the number mismatch and hits "Run build" anyway, the build can provide very quick feedback. (Rather than infinitely polling for a value that will never exist, for example.)The text was updated successfully, but these errors were encountered: