feat(media): add S3-truth per-community storage sweep#2044
Merged
Conversation
Adds the hourly S3 bucket sweep that computes per-community storage metrics directly from bucket listings (source of truth), with audit-log per-user attribution planned as a follow-on overlay. - buzz-media::bucket_index: pure key classifier (thumb/blob/sidecar/ auxiliary/unknown) + incremental aggregation fold, zero I/O, unit tested against synthetic listings covering multi-variant SHAs, mixed-case extensions, orphans, unmapped communities, and malformed keys. - buzz-relay::storage_sweep: single-flight spawned sweep task with a cached snapshot re-emitted every usage-metrics tick, so the DB- derived gauges keep their configured cadence regardless of sweep duration and a sweep from a demoted leader never publishes stale gauges. - Wires StorageSweepState into AppState and the leader-only branch of the usage-metrics tick; BUZZ_STORAGE_SWEEP_* env knobs plus a BUZZ_STORAGE_METRICS kill switch. - Documents the s3:ListBucket deployment prerequisite in the Helm chart's values.yaml. See PLANS/S3_STORAGE_METRICS_PLAN.md (Rev 3) for the full design. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…e tests Rev 3's Tasks section required a demoted-leader-never-emits test and a paused-time tick-cadence composite; neither existed in storage_sweep.rs despite an earlier self-report listing 'demoted-leader-never-emits' among the module's 12 tests. Adds two module-level tests that exercise the property this module owns without a DB harness: a completed-but-unharvested sweep attempt (the demoted-leader case, since harvest only happens inside maybe_spawn_sweep, which the relay only calls from the leader-only tick branch) emits zero storage gauges, and a stalled sweep across several simulated ticks emits health-only gauges on every tick, never double-spawns, then emits the real snapshot once on the first tick after completion. The DB-leader-transition half of the composite (no leader demotion across ticks) needs a live Postgres advisory lock via buzz_db::UsageMetricsLeader, which has no fixture-free construction — that half is documented as integration territory, matching the existing #[ignore = "requires Postgres"] convention used elsewhere in this crate for the same reason. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…uncated pages Failed sweep attempts discarded the SweepError without logging, making AccessDenied from a missing s3:ListBucket grant invisible except via a gauge. A truncated listing page with no continuation token silently returned a partial snapshot instead of failing.
…rics * origin/main: (96 commits) fix(desktop): prefer live agent mentions (#2149) fix(desktop): close focused threads on Escape (#2154) fix: skip membership lookup for open relays (#2107) fix(desktop): make invite QR downloadable (#2168) Archive managed agents when deleted (#2135) perf(relay): cache Git pack hydration (#2169) chore(deps): update rust crate rustls to v0.23.42 (#2151) chore(deps): update dependency @tanstack/react-virtual to v3.14.6 (#2153) Add Agent Config Core: harness capability model behind agent config surfaces (#2158) chore(deps): update all non-major dependencies (#2152) chore(deps): update rust crate getrandom to v0.4.3 (#2150) fix(relay): bound and observe Git read operations (#2167) fix(desktop): derive default clone URL for relay-hosted repos (#2166) fix(desktop): mask composer rounded corners (#2165) feat(desktop): add PR merge conflict recovery (#2164) fix(desktop): mask scrolled content behind channel and thread composers (#2163) feat(desktop): add inline PR diff comments (#2162) feat(desktop): add commit-scoped PR review decisions (#2161) fix(desktop): batch project work item queries (#2160) feat(desktop): make missing project checkouts actionable (#2159) ...
wpfleger96
marked this pull request as draft
July 20, 2026 16:59
wpfleger96
marked this pull request as ready for review
July 21, 2026 22:05
…gauge, add comments
F-EXT1: emit_storage_metrics now tracks per-community series emitted in the
previous tick via StorageEmittedKey{Bytes,Objects}(host). Any series whose
community disappears (unmapped, host rename, scope exclusion) is explicitly
zeroed on the next emission rather than lingering at its last nonzero value
until recorder idle-eviction. Mirrors the emit_in_memory_usage_metrics
pattern in main.rs. Adds regression test covering all three disappearance
scenarios in a single two-emission sequence.
F-EXT2: rename buzz_storage_sweep_failures_total → buzz_storage_sweep_failures
to avoid the _total counter-convention confusion (nothing has shipped, so
rename is safe per Will's forward-only ruling). Adds an inline comment
explaining the gauge/counter distinction. Updates values.yaml doc.
F-EXT3 (doc-only): adds a comment on should_spawn explaining that a failed
attempt retries at tick cadence (default 300 s), not sweep-interval cadence,
and why that is intentional. Adds a matching note in values.yaml.
F-EXT4: adds a comment on the SWEEP_CONFIG function-local OnceLock explaining
why it lives here rather than on AppState (6-call-site constructor churn
explicitly avoided; first-call init is effectively boot-time for this path).
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…ss 3/3 Fix 1 (storage_sweep.rs should_spawn): the prior note claimed a failing sweep costs 'a single cheap LIST call' per retry. This is only true for immediate permission failures; timeout, cap, and malformed-page failures exhaust the sweep's full paginated walk before failing. Reword to scope the cheap-call claim to the permission case and note that other failures are bounded by the sweep's own timeout and object caps. Fix 2 (main.rs SWEEP_CONFIG OnceLock): the prior comment cited six AppState constructor call sites (there are ten), argued that adding a StorageSweepConfig field would require touching all of them (Config has a single Self initializer, so it wouldn't), and claimed first-call init runs within seconds of startup (the first tick is jittered by [0, interval) and this path is leader-only). Replace with the accurate invariant: localized feature config, read once on the first leader tick, stable for the process lifetime. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
tlongwell-block
approved these changes
Jul 22, 2026
tlongwell-block
left a comment
tlongwell-block
left a comment
Collaborator
There was a problem hiding this comment.
Approving on the strength of two independent agent re-reviews at this head (802538226).
- Dawn: 9.5/10 — all four review items verified fixed; storage_sweep 15/15, buzz-media 97/97, full relay lib suite matches main (only the pre-existing mesh_demo flake); fmt clean; CI green at head.
- Max: 9.2/10 — stale-series blocker correctly closed (previously-emitted key tracking + set-difference zeroing, regression test covers disappearance/rename/scope-exclusion for bytes and objects); full check suite green.
Nice work — the fix mirrors the existing emit_in_memory_usage_metrics pattern and tests the actual failure modes.
loganj
pushed a commit
that referenced
this pull request
Jul 22, 2026
…ad-lifecycle * origin/main: fix(desktop): strip GIF metadata extensions before upload (#2425) feat(desktop): gate sign-out behind key backup + typed confirmation (#2424) ci(desktop): add signed macOS canary build (#2419) feat(desktop+acp): spawn a harness per (agent, community) pair at GUI startup — warm sockets, lazy LLM pool (#2122) Show team count separately in channel template rows (#2404) fix(dev): restore shared worktree identity from keyring (#2400) fix(onboarding): skip community profile setup for existing relay members (#2300) fix(mobile): preserve and display sidebar section icons (#2403) Unify sidebar chrome across themes (#2380) feat(media): add S3-truth per-community storage sweep (#2044)
brow
pushed a commit
that referenced
this pull request
Jul 22, 2026
…obile-releasing Co-authored-by: npub1sv749mw8zcmld4ygjx2mx2aqcn3zvtuj2nlmgatxgad3t9uweu9q5marze <833d52edc71637f6d4889195b32ba0c4e2262f9254ffb47566475b15978ecf0a@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub1sv749mw8zcmld4ygjx2mx2aqcn3zvtuj2nlmgatxgad3t9uweu9q5marze <833d52edc71637f6d4889195b32ba0c4e2262f9254ffb47566475b15978ecf0a@sprout-oss.stage.blox.sqprod.co> * origin/main: fix(desktop): strip GIF metadata extensions before upload (#2425) feat(desktop): gate sign-out behind key backup + typed confirmation (#2424) ci(desktop): add signed macOS canary build (#2419) feat(desktop+acp): spawn a harness per (agent, community) pair at GUI startup — warm sockets, lazy LLM pool (#2122) Show team count separately in channel template rows (#2404) fix(dev): restore shared worktree identity from keyring (#2400) fix(onboarding): skip community profile setup for existing relay members (#2300) fix(mobile): preserve and display sidebar section icons (#2403) Unify sidebar chrome across themes (#2380) feat(media): add S3-truth per-community storage sweep (#2044) feat(relay): log NIP-98 pubkey attribution on HTTP bridge requests (#2206) Co-authored-by: npub1sv749mw8zcmld4ygjx2mx2aqcn3zvtuj2nlmgatxgad3t9uweu9q5marze <833d52edc71637f6d4889195b32ba0c4e2262f9254ffb47566475b15978ecf0a@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub1sv749mw8zcmld4ygjx2mx2aqcn3zvtuj2nlmgatxgad3t9uweu9q5marze <833d52edc71637f6d4889195b32ba0c4e2262f9254ffb47566475b15978ecf0a@sprout-oss.stage.blox.sqprod.co> # Conflicts: # RELEASING.md
loganj
pushed a commit
that referenced
this pull request
Jul 22, 2026
…ad-lifecycle * origin/main: fix(desktop): strip GIF metadata extensions before upload (#2425) feat(desktop): gate sign-out behind key backup + typed confirmation (#2424) ci(desktop): add signed macOS canary build (#2419) feat(desktop+acp): spawn a harness per (agent, community) pair at GUI startup — warm sockets, lazy LLM pool (#2122) Show team count separately in channel template rows (#2404) fix(dev): restore shared worktree identity from keyring (#2400) fix(onboarding): skip community profile setup for existing relay members (#2300) fix(mobile): preserve and display sidebar section icons (#2403) Unify sidebar chrome across themes (#2380) feat(media): add S3-truth per-community storage sweep (#2044) Signed-off-by: npub1dpf98sl35hm9k65t8h6cvh5n6knn2msugh5k5nmysxwnw5wlh7uqn2wgp4 <685253c3f1a5f65b6a8b3df5865e93d5a7356e1c45e96a4f64819d3751dfbfb8@sprout-oss.stage.blox.sqprod.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the hourly S3 bucket sweep that computes per-community storage metrics directly from bucket listings (source of truth), with per-user attribution planned as a follow-on overlay via
audit_log.buzz-media::bucket_index— pure key classifier (thumb/blob/sidecar/auxiliary/unknown) plus an incremental aggregation fold over paginatedlist_pageresults, zero I/O, unit tested against synthetic listings covering multi-variant SHAs, mixed-case extensions, orphans, unmapped communities, and malformed keys.buzz-relay::storage_sweep— single-flight spawned sweep task with a cached snapshot re-emitted every usage-metrics tick, so the DB-derived gauges keep their configured cadence regardless of sweep duration, and a sweep from a demoted leader never publishes stale gauges.StorageSweepStateintoAppStateand the leader-only branch of the usage-metrics tick;BUZZ_STORAGE_SWEEP_*env knobs plus aBUZZ_STORAGE_METRICSkill switch.s3:ListBucketdeployment prerequisite and theAccessDenied/sweep_ok=0failure signature in the Helm chart'svalues.yaml.See
PLANS/S3_STORAGE_METRICS_PLAN.md(Rev 3) for the full design.