Skip to content
This repository was archived by the owner on May 13, 2026. It is now read-only.

fix(observability): cap chatty fold_db tracing channels to keep observability.jsonl tractable - #951

Merged
shiba4life merged 1 commit into
mainfrom
kanban/70e5a-observability-rotation
May 9, 2026
Merged

fix(observability): cap chatty fold_db tracing channels to keep observability.jsonl tractable#951
shiba4life merged 1 commit into
mainfrom
kanban/70e5a-observability-rotation

Conversation

@shiba4life

Copy link
Copy Markdown
Collaborator

Summary

  • $FOLDDB_HOME/observability.jsonl grows ~25 MB / 10 min (~150 MB/h) on a fresh node doing routine ingestion, with no rotation. The upstream observability crate writes a plain append-only file with no rotation knob, so until that's added upstream, we cap the noisiest channels at the node side.
  • Two fold_db tracing::* channels dominated ~72% of volume on a 2026-05-09 dogfood (132 Apple Notes + 5 photos + smart-folder of 3 files): fold_db::fold_db_core::mutation_manager (~60%) and fold_db::db_operations::atom_store (~12%). Both are noise unless you're explicitly debugging batch-mutation or atom storage.
  • This PR extends src/log_filter.rs (which already caps sled at INFO ahead of upstream init) with a NATIVE_DEFAULT_CAPS table covering those two targets, opt-in via FOLDDB_LOG_MUTATION_MANAGER=<level> / FOLDDB_LOG_ATOM_STORE=<level>. augment_rust_log is generalized to accept any target=level directive (was sled-only).
  • run.sh now honors an operator-supplied RUST_LOG (${RUST_LOG:-debug}) instead of unconditionally clobbering it. CLAUDE.md gets a short note under "Observability conventions" pointing at the new env vars.

Out of scope (per the task prompt): adding rotation to the upstream observability crate. If we want that, it should be a follow-up in ~/code/edgevector/fold_db adding tracing_appender::rolling to layers/fmt::build_fmt_writer, then a rev bump here.

Test plan

  • cargo fmt --all -- --check clean
  • cargo check --workspace clean (only the pre-existing macOS-only build_attendee_ingestion_records dead-code warning, gated #[cfg(target_os = "macos")] so CI on Linux doesn't see it)
  • cargo nextest run --workspace --lib — 1006 tests pass; 13 log_filter::tests pass including two new ones (appends_native_tracing_directive_with_module_path, default_directives_include_all_caps_at_default_levels)
  • cargo test --workspace --doc clean
  • scripts/lint-no-hardcoded-urls.sh clean
  • Manual smoke: run a fresh node for a few minutes; confirm mutation_manager / atom_store DEBUG lines no longer appear in observability.jsonl and that FOLDDB_LOG_MUTATION_MANAGER=debug restores them

🤖 Generated with Claude Code

…vability.jsonl tractable

The upstream observability crate writes a single append-only JSONL file
with no rotation knob. On a fresh node doing routine ingestion, two
fold_db channels dominated ~72% of line volume:

- fold_db::fold_db_core::mutation_manager (~60%)
- fold_db::db_operations::atom_store      (~12%)

Adds a small NATIVE_DEFAULT_CAPS table in src/log_filter.rs that the
existing apply_default_filters() walks before upstream init reads
RUST_LOG, capping each at INFO. Operators chasing a bug in either
channel restore verbosity with FOLDDB_LOG_MUTATION_MANAGER=debug or
FOLDDB_LOG_ATOM_STORE=debug — same opt-in pattern as the existing
FOLDDB_LOG_SLED knob. Generalizes augment_rust_log to accept a full
target=level directive (was sled-only).

Also tightens run.sh's hardcoded RUST_LOG=debug to honor an
operator-supplied value via ${RUST_LOG:-debug}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@shiba4life
shiba4life enabled auto-merge May 9, 2026 09:18
@shiba4life
shiba4life added this pull request to the merge queue May 9, 2026
Merged via the queue into main with commit 29a8803 May 9, 2026
12 checks passed
@shiba4life
shiba4life deleted the kanban/70e5a-observability-rotation branch May 9, 2026 09:29
@shiba4life
shiba4life restored the kanban/70e5a-observability-rotation branch May 9, 2026 11:11
@shiba4life
shiba4life deleted the kanban/70e5a-observability-rotation branch May 9, 2026 11:22
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant