Drive indexing: stop the NAS first-scan stalling on snapshot dirs, and cut scan log volume ~90%

A reported NAS first-scan stuck near 50% was the recursive size scan grinding through `@Recently-Snapshot` — a Synology Btrfs snapshot tree holding full point-in-time copies of the whole share (44 TB of hardlinked data on a 10 TB volume). The scanner re-walked the entire filesystem once per snapshot over a single serialized SMB connection, so it could run for hours. The same walk firehosed the always-Debug log file (a fresh 50 MB rotation every few minutes, so a user's error report held only ~25 min).

Fix the scan:

- The fresh AND reconcile network scanners (`volume_scanner.rs`) no longer recurse into NAS snapshot/system pseudo-dirs (`@eaDir`, `@Recently-Snapshot`, `@Recycle`, `#recycle`, `#snapshot`, `.snapshot`, `$RECYCLE.BIN`, `System Volume Information`, …) via `system_dirs::is_recursion_excluded_dir` (case-insensitive). The dir's own row is still indexed, so it stays listed and navigable (you can walk into `@Recycle` to restore a file); only its subtree is left un-walked, so it shows an honest unknown size (`—`/`≥`) instead of a misleading roll-up. `@eaDir` (Synology thumbnail sidecar) is matched at ANY depth, not just the share root, since it lives inside every media folder.
- Summing snapshots was both ruinous (a full tree traversal per snapshot) AND wrong (the bytes are deduped, not real consumed space), so excluding them makes the number more correct, not just faster.
- Scope is the network scanner only (the home of these dirs); the local jwalk scanner has its own `should_exclude`.

Slim the logging (keeps DEBUG error-report bundles useful, just drops the plumbing flood):

- The reconciler's `parent path not in DB, skipping event` line (~13% of normal log volume, almost all harmless build-output churn) now routes through the existing skip-aggregator pattern at TRACE + a 5 s DEBUG summary. Generalized `reconciler.rs::unknown_path_skips` into a reusable `skip_aggregator` with two instances (`UNKNOWN_PATH_SKIPS`, `STALE_PARENT_SKIPS`).
- `SmbVolume::list_directory` per-listing lines (~9%) drop to TRACE.
- A throttled (~1/s) `volume_scanner: scanning|reconciling… N dirs, M entries, current: <path>` DEBUG heartbeat replaces the per-listing line as the scan signal, so an error report still shows the scan is alive, where it is, and how far along — without the per-directory flood.
- Fixed a latent UTF-8 panic in the skip-sample truncation: `&path[..80]` could slice mid-codepoint on accented NAS paths (e.g. "Külkeres síelés"); now truncates on a char boundary.

Tests: `skips_recursion_into_nas_system_dirs` (verified RED without the guard) plus `system_dirs` unit tests; full check suite green.

Not included: the `smb2` crate's per-frame DEBUG plumbing (signing/recv/execute, ~64% of the firehose) lives in a separate repo and needs its own release; demoting it to TRACE there is the right follow-up, much less urgent now that the snapshot walk no longer generates millions of SMB ops.

Author: vdavid

apps/desktop/src-tauri/src/file_system/volume/backends/smb.rs

@@ -11,7 +11,7 @@ use super::{
 };
 use crate::file_system::listing::FileEntry;
 use crate::file_system::listing::caching::try_get_watched_listing;
-use log::{debug, info, warn};
+use log::{debug, info, trace, warn};
 use smb2::client::tree::Tree;
 use smb2::{ClientConfig, SmbClient};
 use std::future::Future;
@@ -446,7 +446,11 @@ impl SmbVolume {
         let smb_path = self.to_smb_path(path);
         let display_path = self.to_display_path(&smb_path);
 
-        debug!(
+        // TRACE, not DEBUG: this fires per listing for both the live pane and the index
+        // scan, and was ~9% of normal file-log volume. The scan's own progress signal is
+        // the throttled `volume_scanner: scanning…` DEBUG heartbeat. Bump back with
+        // `RUST_LOG=cmdr_lib::file_system::volume::backends::smb=trace` when chasing a listing bug.
+        trace!(
             "SmbVolume::list_directory: share={}, input={:?}, smb_path={:?}",
             self.share_name, path, smb_path
         );
@@ -465,7 +469,7 @@ impl SmbVolume {
             .map(|e| directory_entry_to_file_entry(e, &display_path))
             .collect();
 
-        debug!(
+        trace!(
             "SmbVolume::list_directory: completed in {:?}, {} entries",
             start.elapsed(),
             entries.len()

apps/desktop/src-tauri/src/indexing/DETAILS.md

@@ -149,6 +149,10 @@ Three disciplines for network round trips (plan rabbit hole #3), all in `list_on
 
 A sub-directory that fails to list (permission, transient) is skipped and the walk continues (like jwalk skipping errored entries); failing to list the ROOT is fatal (nothing to index) so the caller discards.
 
+#### NAS snapshot/system dirs aren't recursed (`system_dirs.rs`)
+
+The BFS does NOT descend into NAS snapshot/system pseudo-directories (`@eaDir`, `@Recently-Snapshot`, `@Recycle`, `#recycle`, `#snapshot`, `.snapshot`, `$RECYCLE.BIN`, `System Volume Information`, …; matched case-insensitively by `system_dirs::is_recursion_excluded_dir`). Both the fresh scan and the reconcile walk apply it: the dir's own row is still indexed (so it stays listed and navigable — a user can walk into `@Recycle` to restore a file), but its subtree is never walked, so it rolls up as honestly-unknown (`—`/`≥`) rather than a misleading total. **Decision/Why:** these dirs are hardlinked, huge, and re-walking them costs a full filesystem traversal *per snapshot* over serialized SMB — a real first-scan stalled near 50% grinding `@Recently-Snapshot`, which alone reported 44 TB on a 10 TB volume. Summing them is both ruinous and wrong (the bytes are deduped, not real consumed space). The names are reserved vendor conventions (`@`/`#`/`$` prefixes) that don't collide with user folders, so a name match is safe. **Guardrail:** don't remove the exclusion to "fill in" the missing sizes — that re-triggers the stall. Scope is the network scanner only (the home of these dirs); the local jwalk scanner has its own `should_exclude`. `FileEntry` carries no DOS hidden/system attribute today; if one is plumbed through, "hidden + system" would generalize this without the hardcoded list.
+
 ### Direct-smb2 gating (`smb_index.rs`, plan rabbit hole #13)
 
 Indexing an SMB share requires Cmdr's own smb2 (`direct`) session, NOT the macOS `os_mount`: `CHANGE_NOTIFY` watching runs over smb2 anyway, and smb2 parallelizes listing far better than per-`readdir` round trips through the kernel mount. An `os_mount` share is registered as a `LocalPosixVolume` on an `smbfs` mount (its `smb_connection_state()` is `None`; the FE enriches it to `OsMount`); a direct one is an `SmbVolume` returning `Some(Direct)`. `ensure_direct_smb` therefore: `Direct` → index now; `Disconnected` → refuse (reconnect first); `os_mount` → trigger/await `upgrade_to_smb_volume_inner`, then re-check. Every refusal is a TYPED `SmbIndexGateReason` (`NotRegistered` / `NotAnSmbVolume` / `UpgradeFailed` / `CredentialsNeeded` / `Disconnected`) that crosses IPC as a snake_case tag — never a message substring (`.claude/rules/no-string-matching.md`). FDA-independent: SMB paths aren't TCC-protected, so `start_indexing_for_smb` never routes through `should_auto_start_indexing` (rabbit hole #12).
@@ -499,16 +503,25 @@ The clamps differ by error band (constants live in `eta.ts`): tier 1 caps at 0.9
 
 ## Log levels
 
-Two high-volume per-event lines are at TRACE (off by default; file chain captures
-Debug+ only): `Writer: UpsertEntryV2 inserted "X"` and
-`Reconciler: removal for unknown path, skipping: X`. Together they were ~90% of normal
-log volume; the writer line is fully redundant with the existing `Writer: +N msgs (...)`
-aggregate, and the reconciler skip is mostly harmless build-output churn.
-
-The reconciler emits a DEBUG aggregate every ~5s: `Reconciler: skipped N removals
-for unknown paths in Xs [total], sample: <path>`. Error reports retain the
-existence-of-drift signal. Live in `reconciler.rs::unknown_path_skips`. To get the
-per-event detail back: `RUST_LOG=cmdr_lib::indexing::reconciler=trace,cmdr_lib::indexing::writer=trace,debug`.
+High-volume per-event lines are at TRACE (off by default; file chain captures Debug+
+only) so the always-Debug file target doesn't drown in plumbing: `Writer: UpsertEntryV2
+inserted "X"`, the two reconciler skip lines (`removal for unknown path` and `parent path
+not in DB, skipping event`), and `SmbVolume::list_directory: …` (per listing, fires for
+both the live pane and the index scan). On a stuck NAS first-scan these dominated the log
+— a single rotated 50 MB file held only ~25 min, so a user's error report lost the
+relevant window. The writer line is redundant with the `Writer: +N msgs (...)` aggregate;
+the reconciler skips are mostly harmless build-output churn; `list_directory` is replaced
+as the scan signal by the heartbeat below.
+
+The reconciler emits a DEBUG aggregate every ~5s per skip class via
+`reconciler.rs::skip_aggregator` (two instances, `UNKNOWN_PATH_SKIPS` and
+`STALE_PARENT_SKIPS`): `Reconciler: skipped N <removals|events> <reason> in Xs [total],
+sample: <path>`. The `volume_scanner` emits a throttled (~1/s) DEBUG heartbeat
+`volume_scanner: scanning|reconciling… N dirs, M entries, current: <path>`, so an error
+report still shows the scan is alive, where it is, and how far along — the signal a
+triager needs without the per-directory `list_directory` flood. Error reports retain the
+existence-of-drift and scan-progress signals. To get the per-event detail back:
+`RUST_LOG=cmdr_lib::indexing::reconciler=trace,cmdr_lib::indexing::writer=trace,cmdr_lib::file_system::volume::backends::smb=trace,debug`.
 
 The writer's SQLite busy retry logger (`stall_probe::sqlite_busy` in `writer/mod.rs::spawn`) logs
 per-attempt at DEBUG: brief contention is routine (WAL checkpoints, long-lived readers). It

apps/desktop/src-tauri/src/indexing/mod.rs

@@ -39,6 +39,7 @@ pub(crate) mod scanner;
 #[cfg(any(target_os = "macos", target_os = "linux"))]
 mod smb_index;
 mod smb_watch;
+mod system_dirs;
 mod verifier;
 pub(crate) mod volume_scanner;
 pub(crate) mod watcher;

apps/desktop/src-tauri/src/indexing/reconciler.rs

@@ -43,15 +43,17 @@ use crate::pluralize::pluralize;
 /// events is always safe.
 const MAX_BUFFER_CAPACITY: usize = 500_000;
 
-/// Aggregator for "removal for unknown path, skipping" events. The per-event line is
-/// at TRACE (off by default; file chain captures Debug+ only); this module emits a
-/// single DEBUG summary every ~5 s, so error report bundles still carry the
-/// existence-of-drift signal without the per-event noise.
+/// Aggregator for high-volume reconciler skip events. Each per-event line is at TRACE
+/// (off by default; file chain captures Debug+ only); this aggregator emits a single
+/// DEBUG summary every ~5 s, so error report bundles still carry the existence-of-drift
+/// signal without the per-event noise. Two instances cover the two skip classes that
+/// dominate normal log volume: [`UNKNOWN_PATH_SKIPS`] (removal for a path not in the DB)
+/// and [`STALE_PARENT_SKIPS`] (create/modify whose parent dir isn't in the DB).
 ///
 /// Most skips are harmless build-output churn, but a sustained rate (or a sample path
 /// in an unexpected tree) can flag real reconciler/index drift. Triagers: if you need
 /// the per-event detail, run with `RUST_LOG=cmdr_lib::indexing::reconciler=trace,debug`.
-mod unknown_path_skips {
+mod skip_aggregator {
     use std::sync::Mutex;
     use std::time::Instant;
 
@@ -69,47 +71,74 @@ mod unknown_path_skips {
         sample: Option<String>,
     }
 
-    static STATE: Mutex<Option<State>> = Mutex::new(None);
+    /// One skip category: its own rolling state plus the words for the summary line
+    /// (`"skipped {unit}s {reason} in …"`).
+    pub(super) struct SkipAggregator {
+        state: Mutex<Option<State>>,
+        /// Counted noun, e.g. `"removal"` → "skipped 3 removals".
+        unit: &'static str,
+        /// Reason phrase, e.g. `"for unknown paths"`.
+        reason: &'static str,
+    }
 
-    /// Increment the counter and emit a summary if the flush interval has elapsed.
-    /// Called on every skipped removal (cheap: one mutex acquisition, one branch).
-    pub(super) fn record(path: &str) {
-        let mut guard = match STATE.lock() {
-            Ok(g) => g,
-            Err(p) => p.into_inner(),
-        };
-        let state = guard.get_or_insert_with(|| State {
-            count_since_last_flush: 0,
-            total: 0,
-            last_flush: Instant::now(),
-            sample: None,
-        });
-        state.count_since_last_flush += 1;
-        state.total += 1;
-        if state.sample.is_none() {
-            let s = if path.len() > SAMPLE_LEN {
-                format!("{}…", &path[..SAMPLE_LEN])
-            } else {
-                path.to_string()
-            };
-            state.sample = Some(s);
+    impl SkipAggregator {
+        const fn new(unit: &'static str, reason: &'static str) -> Self {
+            Self {
+                state: Mutex::new(None),
+                unit,
+                reason,
+            }
         }
-        if state.last_flush.elapsed().as_secs() >= FLUSH_INTERVAL_SECS {
-            let count = state.count_since_last_flush;
-            let total = state.total;
-            let sample = state.sample.clone().unwrap_or_default();
-            let secs = state.last_flush.elapsed().as_secs_f64();
-            state.count_since_last_flush = 0;
-            state.last_flush = Instant::now();
-            state.sample = None;
-            // Drop the lock before logging so the message format won't reenter under it.
-            drop(guard);
-            log::debug!(
-                "Reconciler: skipped {} for unknown paths in {secs:.1}s [{total} total], sample: {sample}",
-                pluralize(count, "removal")
-            );
+
+        /// Increment the counter and emit a summary if the flush interval has elapsed.
+        /// Called on every skip (cheap: one mutex acquisition, one branch).
+        pub(super) fn record(&self, path: &str) {
+            let mut guard = match self.state.lock() {
+                Ok(g) => g,
+                Err(p) => p.into_inner(),
+            };
+            let state = guard.get_or_insert_with(|| State {
+                count_since_last_flush: 0,
+                total: 0,
+                last_flush: Instant::now(),
+                sample: None,
+            });
+            state.count_since_last_flush += 1;
+            state.total += 1;
+            if state.sample.is_none() {
+                // Truncate on a CHAR boundary, not a byte index: NAS paths carry accented
+                // names (e.g. "Külkeres síelés"), and `&path[..SAMPLE_LEN]` would panic
+                // when byte SAMPLE_LEN lands mid-codepoint.
+                let s = if path.chars().count() > SAMPLE_LEN {
+                    format!("{}…", path.chars().take(SAMPLE_LEN).collect::<String>())
+                } else {
+                    path.to_string()
+                };
+                state.sample = Some(s);
+            }
+            if state.last_flush.elapsed().as_secs() >= FLUSH_INTERVAL_SECS {
+                let count = state.count_since_last_flush;
+                let total = state.total;
+                let sample = state.sample.clone().unwrap_or_default();
+                let secs = state.last_flush.elapsed().as_secs_f64();
+                state.count_since_last_flush = 0;
+                state.last_flush = Instant::now();
+                state.sample = None;
+                let (unit, reason) = (self.unit, self.reason);
+                // Drop the lock before logging so the message format won't reenter under it.
+                drop(guard);
+                log::debug!(
+                    "Reconciler: skipped {} {reason} in {secs:.1}s [{total} total], sample: {sample}",
+                    pluralize(count, unit)
+                );
+            }
         }
     }
+
+    /// Removals for a path that isn't in the DB (mostly harmless build-output churn).
+    pub(super) static UNKNOWN_PATH_SKIPS: SkipAggregator = SkipAggregator::new("removal", "for unknown paths");
+    /// Create/modify events whose parent dir isn't in the DB (stale intermediate dir).
+    pub(super) static STALE_PARENT_SKIPS: SkipAggregator = SkipAggregator::new("event", "for missing parents");
 }
 
 /// Buffers FSEvents during the initial scan and replays them after the scan completes.
@@ -921,7 +950,7 @@ fn handle_removal(
             // genuinely harmless. The aggregate at DEBUG (below) gives the existence-of-
             // drift signal without flooding the file.
             log::trace!("Reconciler: removal for unknown path, skipping: {normalized}");
-            unknown_path_skips::record(normalized);
+            skip_aggregator::UNKNOWN_PATH_SKIPS.record(normalized);
             return Some(affected);
         }
         Err(e) => {
@@ -985,8 +1014,12 @@ fn handle_creation_or_modification(
     let parent_id = match store::resolve_path(conn, parent_path) {
         Ok(Some(id)) => id,
         Ok(None) => {
-            // Parent not in DB -- stale event (intermediate directory missing), skip
-            log::debug!("Reconciler: parent path not in DB, skipping event for {normalized} (parent: {parent_path})");
+            // Parent not in DB -- stale event (intermediate directory missing), skip.
+            // Per-event at TRACE; a DEBUG aggregate every ~5 s keeps the drift signal in
+            // error reports without the per-event flood (this was ~13% of normal log
+            // volume, almost all harmless build-output churn).
+            log::trace!("Reconciler: parent path not in DB, skipping event for {normalized} (parent: {parent_path})");
+            skip_aggregator::STALE_PARENT_SKIPS.record(normalized);
             return Some(affected.clone());
         }
         Err(e) => {

apps/desktop/src-tauri/src/indexing/system_dirs.rs

@@ -0,0 +1,105 @@
+//! NAS system/snapshot directories the recursive size scan must not descend into.
+//!
+//! Synology, QNAP, NetApp, and Windows SMB shares expose reserved pseudo-directories
+//! that are catastrophic to recursively size:
+//!
+//! - **Snapshot trees** (`@Recently-Snapshot`, `#snapshot`, `.snapshot`) hold full
+//!   point-in-time copies of the whole share. Their bytes are hardlinked/deduped, so
+//!   summing them is both ruinously expensive (the scanner re-walks the entire
+//!   filesystem once per snapshot) AND wrong (the total isn't real consumed space).
+//!   One real report: a NAS first-scan stalled near 50% grinding through
+//!   `@Recently-Snapshot`, which alone reported 44 TB on a 10 TB volume.
+//! - **Thumbnail/metadata sidecars** (`@eaDir`, `.@__thumb`) live inside *every* media
+//!   folder, so a position-based ("only at share root") skip would miss them — they
+//!   have to be matched at any depth.
+//! - **Recycle bins** (`@Recycle`, `#recycle`, `$RECYCLE.BIN`) and other system dirs
+//!   are large and never what a size roll-up wants.
+//!
+//! These names are reserved vendor conventions (the `@` / `#` / `$` prefixes and
+//! `System Volume Information` don't collide with real user folders), so a name match
+//! is safe. We only SKIP RECURSION: the directory's own row is still indexed and stays
+//! listed and navigable (a user can walk into `@Recycle` to restore a file); we just
+//! don't auto-walk its subtree to compute a recursive size. Its size shows as unknown
+//! (`—`/`≥`), the honest state, rather than `0 B`.
+//!
+//! Scope: applied by the `Volume`-trait network scanner (`volume_scanner.rs`) only,
+//! which walks SMB/MTP shares — the home of these dirs. The local jwalk scanner has its
+//! own `should_exclude`. `FileEntry` carries no DOS hidden/system attribute, so matching
+//! the canonical names is the available signal; if attributes are plumbed through later,
+//! "hidden + system" would generalize this without a hardcoded list.
+
+/// Canonical names of NAS system/snapshot directories, matched case-insensitively
+/// (NAS shares are typically case-insensitive). Extend as new vendor conventions
+/// surface; keep it to reserved, non-user-collidable names.
+const EXCLUDED_DIR_NAMES: &[&str] = &[
+    // Synology
+    "@eaDir",
+    "@Recently-Snapshot",
+    "@Recycle",
+    "@sharesnap",
+    "@sharebin",
+    "@tmp",
+    // QNAP
+    "#recycle",
+    "#snapshot",
+    ".@__thumb",
+    // NetApp / generic
+    ".snapshot",
+    ".snapshots",
+    // Windows / SMB
+    "$RECYCLE.BIN",
+    "System Volume Information",
+];
+
+/// Whether the recursive size scan should NOT descend into a directory with this name.
+///
+/// `name` is a single path component (the directory's own name, not a path). Matched
+/// case-insensitively against [`EXCLUDED_DIR_NAMES`].
+pub(crate) fn is_recursion_excluded_dir(name: &str) -> bool {
+    EXCLUDED_DIR_NAMES
+        .iter()
+        .any(|excluded| name.eq_ignore_ascii_case(excluded))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn excludes_known_nas_system_dirs() {
+        for name in [
+            "@eaDir",
+            "@Recently-Snapshot",
+            "@Recycle",
+            "#recycle",
+            "#snapshot",
+            ".snapshot",
+            "$RECYCLE.BIN",
+            "System Volume Information",
+        ] {
+            assert!(is_recursion_excluded_dir(name), "{name} should be excluded");
+        }
+    }
+
+    #[test]
+    fn matches_case_insensitively() {
+        assert!(is_recursion_excluded_dir("@eadir"));
+        assert!(is_recursion_excluded_dir("@RECENTLY-SNAPSHOT"));
+        assert!(is_recursion_excluded_dir("system volume information"));
+    }
+
+    #[test]
+    fn keeps_ordinary_dirs() {
+        for name in [
+            "photos",
+            "Dori-Dropbox",
+            "videos",
+            "eaDir",
+            "recycle",
+            "snapshot",
+            "@myfiles",
+        ] {
+            assert!(!is_recursion_excluded_dir(name), "{name} should NOT be excluded");
+        }
+    }
+}