Repository navigation
RFC: make "no work available" a value on Next, not codes.NotFound #8041
robinbowes
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Today
Nextis a long-poll that returns one job inline:There is no way to say "nothing available" in that message, so it is said out of band — historically
status.Error(codes.NotFound, ErrNoJobsFound), and (in #8040) the zero value, which the worker reads asresp == nil || resp.JobId == "".Why I think the shape is the problem
codes.NotFoundcarries four unrelated meanings in this one service:Next(pre-#8040)UpdateJobSubmitRedactionThe worker funnels every scheduler call through one
callSchedulerWithBackoff, which cannot tell them apart. That has produced two open bugs whose correct handling is opposite:Nextcase should not be reported and should keep polling. 62 error lines/hour (and 62 warn) on a healthy idle 3.0.3 — measured in Loki over a 30-minute window, single binary, S3 backend.UpdateJobreturningNotFoundwill never succeed, so retrying is wrong; withMaxRetries: 0(infinite) the worker retries one job id forever and stops callingNext, collapsing its own throughput.Any fix keyed on the status code has to encode which call site it is in. Making "nothing available" a value removes that:
NotFoundthen means only "the thing you named does not exist", which is what gRPC means by it, and #7879's short-circuit becomes safe to apply generally.Proposal
len(jobs) == 0needs no shared convention, and in proto3 an empty repeated field and an absent one decode identically — both mean the same thing here, so there is no third state to misread.RedactionBatchesis the same shape already.Migration, wire-compatible throughout: add
Joband field 4; have the scheduler populate both the list and fields 1–3 for one release; move workers to prefer the list and fall back; drop 1–3 in the next major. At most one element until batching is actually wanted.Not what this is for
This does not fix the log noise — #8039 and #8040 do that without touching the wire, and should land on their own. This is only about the shape, so please read them independently of this.
Questions
mergedJobschannel, andUpdateJob's single-job shape. Happy for the answer to be "list of at most one, forever".bool no_work_available, or a status enum)? Uglier, much smaller.JobDetail's(gogoproto.nullable) = falseconstrain nesting it in a repeated message in a way I have not hit yet?Disclosure: drafted with Claude Code; the code references, the four
NotFoundsites and the measured rate were checked against v3.0.3/v3.1.0 and a running 3.0.3 before posting.All reactions