Skip to content

Fix "concurrent update in progress" errors during canary deploys#4825

Merged
dangra merged 4 commits intomasterfrom
rm/canary-stopped
Apr 14, 2026
Merged

Fix "concurrent update in progress" errors during canary deploys#4825
dangra merged 4 commits intomasterfrom
rm/canary-stopped

Conversation

@rianmcguirefly
Copy link
Copy Markdown
Member

@rianmcguirefly rianmcguirefly commented Apr 14, 2026

Change Summary

What and Why:

Avoid possibility of failure during a canary deploy if the machine that we run the canary update on is currently stopped.

Example error:
Failed to update machines: failed to update machine <machine id>: failed to update VM <machine id>: aborted: machine is replacing: concurrent update in progress Retrying...

The machines API blocks concurrent updates to a machine, so we need to ensure that doesn't happen.

How:

Fix all the things

  • When updating a non-started machine, we now wait for the machine update to fully complete (eg. machine transitions from replacing back to stopped) before continuing
  • Avoid updating the canary machine twice
  • Prefer a started machine for the canary where possible

@dangra dangra merged commit eaa1e0f into master Apr 14, 2026
35 of 37 checks passed
@dangra dangra deleted the rm/canary-stopped branch April 14, 2026 16:39
@dangra
Copy link
Copy Markdown
Member

dangra commented Apr 14, 2026

@rianmcguirefly I merged this after it passed all tests so it is included in today's auto release. thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants