Add asynchronous S3 chunk blob backfill

[bootjp]

Summary

• enqueue applied S3 chunk references without blocking the Raft apply goroutine
• restore missing local chunk blobs with a bounded worker pool, per-peer token buckets, SHA/size verification, and alternate-peer fallback
• recover snapshot-installed or dropped notifications through startup and periodic scans of every local shard store
• expose queue depth and outcome metrics and fail startup on invalid resource-bound configuration

Safety

• background writes use the same startup-rotation gate as peer push and proxy-on-miss repair
• shutdown stops backfill workers before closing peer connections
• local scans never use leader routing and queue pressure cannot stall Raft apply

Verification

• go test ./... -count=1
• go test -race ./adapter -run 'TestS3BlobBackfill|TestS3BlobTokenBucket' -count=3\n- golangci-lint run ./... --timeout=5m --allow-parallel-runners -j 2\n- focused backfill, shard-store, and metrics tests x10\n- git diff --check\n\nStacked on s3: add blob fetch rpc #1063.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ee939daa-1370-4298-a9e1-3811f7bc2bfa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[bootjp]

Signed head a1d6a3ac implements M2 follower apply/backfill. Full Go tests, focused race tests, full lint, repeated worker/scan/rate-limit tests, and diff checks pass.

@codex review

[gemini-code-assist]

Code Review

This pull request introduces an asynchronous S3 blob backfiller (S3BlobBackfiller) to handle follower catch-up work independently from the Raft apply loop, complete with background workers, periodic scanning, token-bucket rate limiting, and Prometheus metrics. The feedback highlights three key areas for improvement: ensuring that stopping the backfiller properly drains the queue and clears pending keys to prevent permanent blocks, optimizing the local blob existence check to avoid loading entire payloads into memory, and addressing a potential thundering herd issue in the custom token bucket implementation by adopting a stateful reservation mechanism.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a1d6a3aca6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Signed head 100acd3 addresses every current review finding.

Changes:

• drain queued and pending work on Stop and safely support restart
• use ExistsAt for local presence checks
• reserve token-bucket slots across concurrent workers and restore canceled reservations
• write peer-local repairs at the replicated chunkref commit timestamp without a follower TSO allocation
• atomically publish and clear the backfill server pointer
• route metadata, local blobs, and peer discovery by the chunkref placement key
• resolve lint findings by extracting helpers and parsing runtime configuration outside run

Caller audit:

• all storeFetchedS3ChunkBlob callers now pass the chunkref route key and replicated commit timestamp
• proxy-on-miss repair and asynchronous backfill are the only production callers
• OnApply, worker, scanner, and observer paths load the server through the atomic pointer

Verification:

• focused adapter, monitoring, and main tests repeated 20 times: pass
• focused adapter race tests repeated 5 times: pass
• full golangci-lint: 0 issues
• git diff --check: pass
• both commits verified with a valid bootjp contact@bootjp.me signature

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 100acd365f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Rebased the signed M2 commits onto #1063 head 0a7eed9. Current signed head is 07b963f; the M2 diff is unchanged and now includes the base multipart isolation fix.

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07b963fedc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

History rebuilt without content changes and restacked on the sanitized #1063 head.

• Previous head: 07b963fedcb35d121296e871931296c2e1630922
• Current head: b374525c5279d529356e2083dbcdb322e6e2b367
• Tree before/after: ad0dc6b36b88e7478cd05421c10981a11757984f
• git diff --exit-code <previous> <current>: clean
• Parent: 02e3a6d5b4fe559dbfc0144313995332736e1b30
• PR-visible commits: one
• Author and committer: bootjp <contact@bootjp.me>
• GitHub signature verification: valid

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b374525c52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Current signed head eaaa08bb45fd0fb73f46015024b2457cdc3065ef is restacked on corrected #1063 and addresses both current P2 findings.

Changes:

• self-heal corrupt same/newer local chunkblob versions with a peer-verified local repair timestamp while preserving the store-wide MVCC watermark
• validate local blob content during backfill scans instead of treating key existence as healthy
• start a listenless backfill lifecycle on peer-token nodes with no public S3 address

Caller audit:

• storeFetchedS3ChunkBlob has two production callers: GET proxy-on-miss and async backfill
• normal gRPC push and local replication retain the existing fail-closed storeChunkBlob semantics
• peer repair remains follower-safe and does not allocate through TSO

Verification:

• focused repair/backfill/main lifecycle tests
• focused tests repeated 20 times
• focused race tests
• all-package compile
• full golangci-lint: 0 issues
• proto breaking and regenerated diff checks
• git diff --check
• PR-visible author/committer and signature verification: valid

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: eaaa08bb45

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".