Skip to content

fix(warm): tolerate concurrent instance consumption - #295

Merged
rldyourmnd merged 4 commits into
mainfrom
fix/warm-consumption-race
Aug 31, 2026
Merged

fix(warm): tolerate concurrent instance consumption#295
rldyourmnd merged 4 commits into
mainfrom
fix/warm-consumption-race

Conversation

@rldyourmnd

@rldyourmnd rldyourmnd commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Outcome

This closes two live warm-capacity races:

  1. A warm-pool reconcile no longer fails when concurrent job consumption or pressure retirement removes a newly created instance before network/readiness observation completes. It reports consumed_during_create; the next bounded timer pass converges any remaining deficit.
  2. A claimed warm instance now carries both identities explicitly: the persistent Incus provider instance name and the one-job GitHub runner name. The job-start hook authenticates the runner name while the broker/journal remains keyed by the provider instance. Previously it compared RUNNER_NAME with warm-standard-*, exited before contacting the broker, and GitHub showed only Set up runner failure.

Resource pressure remains a structured non-error deferral, and real Incus/control-plane failures remain errors.

Live evidence

  • gha-warm-pool@nddev-priority-standard.service reached 2/2, then a peak-load reconcile failed after 103 seconds with Instance not found while waiting for a newly created instance network.
  • Server Almaty CodeQL jobs 99341447534 and 99341447679 used claimed warm instances and failed their pre-job hook in 44 ms. Sanitized worker diagnostics showed RUNNER_NAME=nddev-* while the schema-2 assignment was keyed by warm-standard-*; the cache broker received no request.

Release

  • provider: v0.1.5-nddev.91
  • exact source: ac2736f9b9801e16fc675b4dea0fca821f545672
  • reproducible SHA-256: 2ad8f48470c1d979b12c8b5f244391348a454a5478ea9943212ed7431438ea57
  • two byte-identical builds

Verification

make verify passes after both fixes, including race tests, vet, and full build.

@rldyourmnd
rldyourmnd merged commit 4237987 into main Aug 31, 2026
10 checks passed
@rldyourmnd
rldyourmnd deleted the fix/warm-consumption-race branch August 31, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant