fix(orchestrator): stamp the queue on path sets and path-build links - #551
Merged
Conversation
## Summary
### Why?
Speculation never funded a path, so no build was ever dispatched and the submitqueue e2e hung until Bazel killed it at 300s, having spilled ~11MB of retry logs.
Two producers build an entity without naming its queue, and both of their stores are bound to one queue and reject a write that disagrees:
- `speculate/dispatch.go` created a head's first `SpeculationPathSet` as `{Head: batch.ID}`, so `Create` failed with `queue "" does not match the store's bound queue`. That is the first write of every newly funded head, so no head was ever funded.
- `build/build.go` created the `PathBuild` link as `{PathID, Attempt, BuildID}`, so the link write failed the same way. Without the link a path is never observed moving to `building`, so it stays `pending` and every later run re-dispatches it.
Both stores gained their queue-bound guard when their tables were re-keyed to lead with the queue (#543 for `speculation_path_set`, #502 for `path_build`); the producers were not updated to match.
Unit tests missed it because they use the gomock stores, which do not enforce the binding, and the storage contract suite missed it because it builds its own fixtures with the queue set. Only the e2e exercises a real producer against a real queue-bound store.
### What?
Stamps the queue at both construction sites: the path set takes the run's queue, the link takes its batch's.
Turns both into regression tests rather than leaving them to the e2e: the build controller's link expectations are exact struct matches that now require the queue, and the speculate run test asserts the created set names it. Each fails against the unfixed code.
## Test Plan
✅ `go test ./submitqueue/... ./service/...`
✅ Verified each new assertion fails when its fix is reverted
✅ `make lint` / `make check-gazelle`
mnoah1
approved these changes
Aug 7, 2026
behinddwalls
temporarily deployed
to
stack-rebase
August 7, 2026 21:21 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
Speculation never funded a path, so no build was ever dispatched and the submitqueue e2e hung until Bazel killed it at 300s, having spilled ~11MB of retry logs.
Two producers build an entity without naming its queue, and both of their stores are bound to one queue and reject a write that disagrees:
speculate/dispatch.gocreated a head's firstSpeculationPathSetas{Head: batch.ID}, soCreatefailed withqueue "" does not match the store's bound queue. That is the first write of every newly funded head, so no head was ever funded.build/build.gocreated thePathBuildlink as{PathID, Attempt, BuildID}, so the link write failed the same way. Without the link a path is never observed moving tobuilding, so it stayspendingand every later run re-dispatches it.Both stores gained their queue-bound guard when their tables were re-keyed to lead with the queue (#543 for
speculation_path_set, #502 forpath_build); the producers were not updated to match.Unit tests missed it because they use the gomock stores, which do not enforce the binding, and the storage contract suite missed it because it builds its own fixtures with the queue set. Only the e2e exercises a real producer against a real queue-bound store.
What?
Stamps the queue at both construction sites: the path set takes the run's queue, the link takes its batch's.
Turns both into regression tests rather than leaving them to the e2e: the build controller's link expectations are exact struct matches that now require the queue, and the speculate run test asserts the created set names it. Each fails against the unfixed code.
Test Plan
✅
go test ./submitqueue/... ./service/...✅ Verified each new assertion fails when its fix is reverted
✅
make lint/make check-gazelle