This repository was archived by the owner on May 13, 2026. It is now read-only.
fix(observability): cap chatty fold_db tracing channels to keep observability.jsonl tractable - #951
Merged
Merged
Conversation
…vability.jsonl tractable
The upstream observability crate writes a single append-only JSONL file
with no rotation knob. On a fresh node doing routine ingestion, two
fold_db channels dominated ~72% of line volume:
- fold_db::fold_db_core::mutation_manager (~60%)
- fold_db::db_operations::atom_store (~12%)
Adds a small NATIVE_DEFAULT_CAPS table in src/log_filter.rs that the
existing apply_default_filters() walks before upstream init reads
RUST_LOG, capping each at INFO. Operators chasing a bug in either
channel restore verbosity with FOLDDB_LOG_MUTATION_MANAGER=debug or
FOLDDB_LOG_ATOM_STORE=debug — same opt-in pattern as the existing
FOLDDB_LOG_SLED knob. Generalizes augment_rust_log to accept a full
target=level directive (was sled-only).
Also tightens run.sh's hardcoded RUST_LOG=debug to honor an
operator-supplied value via ${RUST_LOG:-debug}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
$FOLDDB_HOME/observability.jsonlgrows ~25 MB / 10 min (~150 MB/h) on a fresh node doing routine ingestion, with no rotation. The upstreamobservabilitycrate writes a plain append-only file with no rotation knob, so until that's added upstream, we cap the noisiest channels at the node side.fold_dbtracing::*channels dominated ~72% of volume on a 2026-05-09 dogfood (132 Apple Notes + 5 photos + smart-folder of 3 files):fold_db::fold_db_core::mutation_manager(~60%) andfold_db::db_operations::atom_store(~12%). Both are noise unless you're explicitly debugging batch-mutation or atom storage.src/log_filter.rs(which already caps sled atINFOahead of upstream init) with aNATIVE_DEFAULT_CAPStable covering those two targets, opt-in viaFOLDDB_LOG_MUTATION_MANAGER=<level>/FOLDDB_LOG_ATOM_STORE=<level>.augment_rust_logis generalized to accept anytarget=leveldirective (was sled-only).run.shnow honors an operator-suppliedRUST_LOG(${RUST_LOG:-debug}) instead of unconditionally clobbering it. CLAUDE.md gets a short note under "Observability conventions" pointing at the new env vars.Out of scope (per the task prompt): adding rotation to the upstream
observabilitycrate. If we want that, it should be a follow-up in~/code/edgevector/fold_dbaddingtracing_appender::rollingtolayers/fmt::build_fmt_writer, then a rev bump here.Test plan
cargo fmt --all -- --checkcleancargo check --workspaceclean (only the pre-existing macOS-onlybuild_attendee_ingestion_recordsdead-code warning, gated#[cfg(target_os = "macos")]so CI on Linux doesn't see it)cargo nextest run --workspace --lib— 1006 tests pass; 13log_filter::testspass including two new ones (appends_native_tracing_directive_with_module_path,default_directives_include_all_caps_at_default_levels)cargo test --workspace --doccleanscripts/lint-no-hardcoded-urls.shcleanmutation_manager/atom_storeDEBUG lines no longer appear inobservability.jsonland thatFOLDDB_LOG_MUTATION_MANAGER=debugrestores them🤖 Generated with Claude Code