Repository navigation
On CI, restores refused by the download budget re-upload results the remote already has, and every re-upload is a billed 412 #950
Replies: 2 comments
|
Thanks for the detailed report and the request counts. The analysis was right. The download budget is the store's GC budget ( #951 changes two things. Prefetch now stops at half of the budget, so the build's own restores keep the rest. A restore the budget refuses now fails with a message that names the budget and It doesn't change what prefetch downloads, and it doesn't add a check before uploading blobs the remote already holds (ask 2). The billed 412s came from the recompiles, so with fewer refused restores they should mostly go away. Once the fix is released, could you post the PUT and 412 counts for a night? If 412s are still a large share, a HEAD check before the PUT would be the next step. AI-assisted — Tool: Claude Code; model: anthropic/claude-sonnet-5-5; version: unavailable. |
|
Windows numbers from the first nightly with 11-14 logged no AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5-5; version: 2.1.322. (Written by Claude Opus 5.5 with low) |
Uh oh!
There was an error while loading. Please reload this page.
What I was trying to do: About 40 GitHub-hosted jobs run nightly on one repository and share a Cloudflare R2 bucket as mbx's S3 remote. They use mbx 1.30.0 through mr-boxington-action with
MBX_REMOTE_MODE=read-write. R2 bills every PUT as a Class A request, including one refused with 412.What happened: On 2026-11-12 the bucket received 163,302 PUTs, and 151,043 of them failed with
412 Precondition Failed. R2's per-object analytics show that 99% of those 412s hit objects that already existed before that night. So this isn't concurrent jobs racing to create the same object.On these runners the action store budget resolves to 5 GiB on Linux and 10 GiB on Windows. 39 of 41 jobs reached it, mostly while prefetching:
After that, the build's own restores fail, and the unit is compiled again:
All 128 blobs that this job reported missing were in the bucket. Each one's only PUTs that night were 412s, so the recompiled results were re-sent in full, every object a conditional PUT that failed. The jobs on a platform share one prediction manifest, since it's keyed by
Cargo.lock. That seems to be why prefetch spends the budget on other jobs' units before the job's own restores.Workaround: we set
MBX_GC_MAX_SIZE=20GiBon hosted runners (moq-dev/moq#8631). We'll add before/after request counts here.What would help:
cached action is missing blob. That message reads as an expired object.AI-assisted — Tool: Claude Code; model: anthropic/claude-opus-5-5; version: 2.1.319.
(Written by Claude Opus 5.5 with high)
All reactions