Skip to content

fix(orchestrator): stamp the queue on path sets and path-build links - #551

Merged
behinddwalls merged 1 commit into
mainfrom
fix/queue-stamp-writes
Aug 7, 2026
Merged

fix(orchestrator): stamp the queue on path sets and path-build links#551
behinddwalls merged 1 commit into
mainfrom
fix/queue-stamp-writes

Conversation

@behinddwalls

Copy link
Copy Markdown
Collaborator

Summary

Why?

Speculation never funded a path, so no build was ever dispatched and the submitqueue e2e hung until Bazel killed it at 300s, having spilled ~11MB of retry logs.

Two producers build an entity without naming its queue, and both of their stores are bound to one queue and reject a write that disagrees:

  • speculate/dispatch.go created a head's first SpeculationPathSet as {Head: batch.ID}, so Create failed with queue "" does not match the store's bound queue. That is the first write of every newly funded head, so no head was ever funded.
  • build/build.go created the PathBuild link as {PathID, Attempt, BuildID}, so the link write failed the same way. Without the link a path is never observed moving to building, so it stays pending and every later run re-dispatches it.

Both stores gained their queue-bound guard when their tables were re-keyed to lead with the queue (#543 for speculation_path_set, #502 for path_build); the producers were not updated to match.

Unit tests missed it because they use the gomock stores, which do not enforce the binding, and the storage contract suite missed it because it builds its own fixtures with the queue set. Only the e2e exercises a real producer against a real queue-bound store.

What?

Stamps the queue at both construction sites: the path set takes the run's queue, the link takes its batch's.

Turns both into regression tests rather than leaving them to the e2e: the build controller's link expectations are exact struct matches that now require the queue, and the speculate run test asserts the created set names it. Each fails against the unfixed code.

Test Plan

go test ./submitqueue/... ./service/...
✅ Verified each new assertion fails when its fix is reverted
make lint / make check-gazelle

## Summary

### Why?

Speculation never funded a path, so no build was ever dispatched and the submitqueue e2e hung until Bazel killed it at 300s, having spilled ~11MB of retry logs.

Two producers build an entity without naming its queue, and both of their stores are bound to one queue and reject a write that disagrees:

- `speculate/dispatch.go` created a head's first `SpeculationPathSet` as `{Head: batch.ID}`, so `Create` failed with `queue "" does not match the store's bound queue`. That is the first write of every newly funded head, so no head was ever funded.
- `build/build.go` created the `PathBuild` link as `{PathID, Attempt, BuildID}`, so the link write failed the same way. Without the link a path is never observed moving to `building`, so it stays `pending` and every later run re-dispatches it.

Both stores gained their queue-bound guard when their tables were re-keyed to lead with the queue (#543 for `speculation_path_set`, #502 for `path_build`); the producers were not updated to match.

Unit tests missed it because they use the gomock stores, which do not enforce the binding, and the storage contract suite missed it because it builds its own fixtures with the queue set. Only the e2e exercises a real producer against a real queue-bound store.

### What?

Stamps the queue at both construction sites: the path set takes the run's queue, the link takes its batch's.

Turns both into regression tests rather than leaving them to the e2e: the build controller's link expectations are exact struct matches that now require the queue, and the speculate run test asserts the created set names it. Each fails against the unfixed code.

## Test Plan

✅ `go test ./submitqueue/... ./service/...`
✅ Verified each new assertion fails when its fix is reverted
✅ `make lint` / `make check-gazelle`
@behinddwalls
behinddwalls merged commit 2d0f019 into main Aug 7, 2026
13 of 15 checks passed
@behinddwalls
behinddwalls deleted the fix/queue-stamp-writes branch August 7, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants