Skip to content

Commit f166b06

Browse files
committed
Indexing: Show replay progress overlay
Extract reusable `ProgressOverlay` from `ScanStatusOverlay`, then wire it to the existing `index-replay-progress` backend events that were being emitted but ignored by the frontend. - Extract `ProgressOverlay.svelte` into `$lib/ui/` with props: `visible`, `label`, `detail`, `progress`, `eta`. Two layout modes: compact (spinner + label) and full (progress bar + ETA) - Refactor `ScanStatusOverlay` to a thin wrapper feeding scan/aggregation state into `ProgressOverlay` - Add `ReplayStatusOverlay.svelte`: shows "Updating index..." after 4s of continuous replay, with progress bar and 50-50 blended ETA (total-based + sliding-window rate over ~5s) - Add replay state tracking in `index-state.svelte.ts`: `replaying`, `replayEventsProcessed`, `replayEstimatedTotal`, `replayStartedAt` with event listeners for `index-replay-progress` and `index-replay-complete` - Emit new `index-replay-complete` Tauri event from `event_loop.rs` after replay flush
1 parent c39b029 commit f166b06

14 files changed

Lines changed: 412 additions & 129 deletions

File tree

apps/desktop/coverage-allowlist.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,12 @@
207207
},
208208
"tauri-commands/search.ts": {
209209
"reason": "Tauri command wrappers, tested via integration"
210+
},
211+
"indexing/ReplayStatusOverlay.svelte": {
212+
"reason": "UI overlay component, depends on Tauri event listeners"
213+
},
214+
"ui/ProgressOverlay.svelte": {
215+
"reason": "Pure UI component, no logic to test beyond rendering"
210216
}
211217
}
212218
}

apps/desktop/src-tauri/src/indexing/CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Full design: `docs/specs/drive-indexing/plan.md`
1111
- **mod.rs** -- Public API (`init()`, `start_indexing()`, `stop_indexing()`, `clear_index()`), `IndexPhase` state machine, `IndexManager` (coordinates all subsystems), `DebugStats` (shared atomic counters for the debug window + phase timeline via `set_phase()`/`close_phase_with_stats()`). `start_scan()` takes a `scan_trigger: &str` parameter describing why the scan was initiated.
1212
- **enrichment.rs** -- `ReadPool` (lock-free thread-local read connections for enrichment and verification), `enrich_entries_with_index()` (called when entries are stored in the listing cache — streaming, watcher update, re-sort — NOT on `get_file_range`; index freshness is handled by `index-dir-updated``refreshIndexSizes``getDirStatsBatch`). Integer-keyed fast path: resolve parent dir once → batch-fetch child dir stats by ID → match by name. Falls back to individual path resolution for edge cases.
1313
- **event_loop.rs** -- `run_live_event_loop` (real-time FSEvents/inotify processing after scan completes), `run_replay_event_loop` (cold-start journal replay with two-phase approach), `run_background_verification` (post-replay bidirectional readdir diff), `merge_fs_events` (deduplication with flag priority), `process_live_batch`. All bounded-buffer constants live here.
14-
- **events.rs** -- Tauri event payload structs (`IndexScanStartedEvent`, `IndexScanProgressEvent`, `IndexScanCompleteEvent`, `IndexDirUpdatedEvent`, `IndexReplayProgressEvent`), `RescanReason` enum, `emit_rescan_notification()`, IPC response types (`IndexStatusResponse`, `IndexDebugStatusResponse`). Also: `ActivityPhase` enum (Replaying/Scanning/Aggregating/Reconciling/Live/Idle) and `PhaseRecord` for the phase timeline system tracked in `DebugStats`.
14+
- **events.rs** -- Tauri event payload structs (`IndexScanStartedEvent`, `IndexScanProgressEvent`, `IndexScanCompleteEvent`, `IndexDirUpdatedEvent`, `IndexReplayProgressEvent`, `IndexReplayCompleteEvent`), `RescanReason` enum, `emit_rescan_notification()`, IPC response types (`IndexStatusResponse`, `IndexDebugStatusResponse`). Also: `ActivityPhase` enum (Replaying/Scanning/Aggregating/Reconciling/Live/Idle) and `PhaseRecord` for the phase timeline system tracked in `DebugStats`.
1515
- **store.rs** -- SQLite schema v6 (integer-keyed entries with `name_folded` column on macOS, dir_stats by entry_id, meta), platform_case collation, read queries, DB open/migrate. `resolve_component` uses the composite index directly: on macOS queries by `(parent_id, name_folded)`, on Linux/Windows by `(parent_id, name)`. Schema version check: mismatch triggers drop+rebuild. Both path-keyed (backward compat) and integer-keyed APIs.
1616
- **memory_watchdog.rs** -- Background task monitoring resident memory via `mach_task_info` (macOS). Warns at 8 GB, stops indexing at 16 GB, emits `index-memory-warning` event to frontend. No-op stub on non-macOS. Started from `start_indexing()`.
1717
- **writer.rs** -- Single writer thread, owns the write connection, processes `WriteMessage` channel (bounded `sync_channel`, 20K capacity, backpressure via blocking). `WRITER_GENERATION: AtomicU64` (initialized to 1) bumped on every mutation (`InsertEntriesV2`, `UpsertEntryV2`, `DeleteEntryById`, `DeleteSubtreeById`, `TruncateData`) for search index staleness detection. Priority: `UpdateDirStats` before `InsertEntries`. `Flush` variant + async `flush()` method let callers wait for all prior writes to commit. Has both integer-keyed variants (`InsertEntriesV2`, `UpsertEntryV2`, `DeleteEntryById`, `DeleteSubtreeById`, `PropagateDeltaById`) and path-keyed backward-compat variants. The integer-keyed delete/subtree-delete handlers auto-propagate negative deltas via the `parent_id` chain (same pattern as the path-keyed variants). `propagate_delta_by_id` walks the parent chain using `get_parent_id` lookups. `UpsertEntryV2` initializes a zero-valued `dir_stats` row when inserting a NEW directory, so enrichment always has a row (subsequent `PropagateDeltaById` calls update it incrementally). Maintains `AccumulatorMaps` during `InsertEntriesV2` processing (two HashMaps: direct children stats and child dir relationships + an `entries_inserted` counter), cleared on `TruncateData`. On `ComputeAllAggregates`, passes accumulated maps to `aggregator::compute_all_aggregates_with_maps()` to skip expensive full-table-scan SQL queries. Accepts an optional `AppHandle` at spawn time to emit `index-aggregation-progress` events during aggregation (phase, current, total). Also emits `saving_entries` phase progress during `InsertEntriesV2` processing when the expected total is set via `set_expected_total_entries()` (an `Arc<AtomicU64>` shared between the writer thread and the `IndexWriter` handle). No index drop/recreate dance — the composite indexes (`idx_parent_name_folded` on macOS, `idx_parent_name` on Linux) use binary collation and stay present during scans.

apps/desktop/src-tauri/src/indexing/event_loop.rs

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ use tauri::{AppHandle, Emitter};
99

1010
use super::DEBUG_STATS;
1111
use super::enrichment::get_read_pool;
12-
use super::events::{IndexReplayProgressEvent, RescanReason, emit_rescan_notification};
12+
use super::events::{IndexReplayCompleteEvent, IndexReplayProgressEvent, RescanReason, emit_rescan_notification};
1313
use super::firmlinks;
1414
use super::reconciler::{self, EventReconciler};
1515
use super::scanner;
@@ -525,6 +525,15 @@ pub(super) async fn run_replay_event_loop(
525525
},
526526
);
527527

528+
// Emit replay complete
529+
let _ = app.emit(
530+
"index-replay-complete",
531+
IndexReplayCompleteEvent {
532+
volume_id: volume_id.clone(),
533+
duration_ms: replay_start.elapsed().as_millis() as u64,
534+
},
535+
);
536+
528537
// Emit a single batched index-dir-updated with all collected paths.
529538
// If affected_paths overflowed, emit a full refresh notification with
530539
// just "/" so the frontend refreshes everything.

apps/desktop/src-tauri/src/indexing/events.rs

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,13 @@ pub struct IndexReplayProgressEvent {
4444
pub estimated_total: Option<u64>,
4545
}
4646

47+
#[derive(Debug, Clone, Serialize, Deserialize)]
48+
#[serde(rename_all = "camelCase")]
49+
pub struct IndexReplayCompleteEvent {
50+
pub volume_id: String,
51+
pub duration_ms: u64,
52+
}
53+
4754
/// Why a full rescan was triggered instead of incremental replay.
4855
/// Sent to the frontend as `index-rescan-notification` so the UI can show
4956
/// a transparent, user-friendly toast.

apps/desktop/src-tauri/src/lib.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,9 +94,9 @@ mod linux_icons;
9494
mod macos_icons;
9595
mod mcp;
9696
mod menu;
97-
mod net;
9897
#[cfg(any(target_os = "macos", target_os = "linux"))]
9998
mod mtp;
99+
mod net;
100100
#[cfg(any(target_os = "macos", target_os = "linux"))]
101101
mod network;
102102
#[cfg(target_os = "macos")]

apps/desktop/src-tauri/src/mcp/executor.rs

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -832,9 +832,10 @@ fn run_search_and_postprocess(index: &search::SearchIndex, query: &SearchQuery)
832832

833833
// Fill directory sizes from the DB
834834
if result.entries.iter().any(|e| e.is_directory)
835-
&& let Some(pool) = crate::indexing::get_read_pool() {
836-
fill_directory_sizes(&mut result, &pool);
837-
}
835+
&& let Some(pool) = crate::indexing::get_read_pool()
836+
{
837+
fill_directory_sizes(&mut result, &pool);
838+
}
838839

839840
// Post-filter: remove directories that don't match size criteria
840841
let has_size_filter = query.min_size.is_some() || query.max_size.is_some();
@@ -1001,8 +1002,7 @@ async fn execute_ai_search(params: &Value) -> ToolResult {
10011002
}
10021003
}
10031004

1004-
let translate_result =
1005-
crate::commands::search::build_translate_result(ai_query).map_err(ToolError::internal)?;
1005+
let translate_result = crate::commands::search::build_translate_result(ai_query).map_err(ToolError::internal)?;
10061006

10071007
let query = SearchQuery {
10081008
name_pattern: translate_result.query.name_pattern,

apps/desktop/src/lib/indexing/CLAUDE.md

Lines changed: 34 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,13 @@ Rust counterpart: `apps/desktop/src-tauri/src/indexing/`
66

77
## Files
88

9-
| File | Purpose |
10-
| -------------------------- | ---------------------------------------------------------------- |
11-
| `index.ts` | Public API barrel export |
12-
| `index-state.svelte.ts` | Module-level `$state` for scan progress; listens for scan events |
13-
| `index-events.ts` | Listens for `index-dir-updated`, calls back with updated paths |
14-
| `ScanStatusOverlay.svelte` | Floating top-right spinner + live counters during scan |
9+
| File | Purpose |
10+
| ---------------------------- | ------------------------------------------------------------------ |
11+
| `index.ts` | Public API barrel export |
12+
| `index-state.svelte.ts` | Module-level `$state` for scan progress; listens for scan events |
13+
| `index-events.ts` | Listens for `index-dir-updated`, calls back with updated paths |
14+
| `ScanStatusOverlay.svelte` | Thin wrapper feeding scan/aggregation state into `ProgressOverlay` |
15+
| `ReplayStatusOverlay.svelte` | Thin wrapper feeding replay state into `ProgressOverlay` |
1516

1617
## Public API (`index.ts`)
1718

@@ -25,6 +26,10 @@ getAggregationPhase(): string // 'saving_entries' | 'loading' | 'sorting'
2526
getAggregationCurrent(): number
2627
getAggregationTotal(): number
2728
getAggregationStartedAt(): number // Date.now() timestamp
29+
isReplaying(): boolean
30+
getReplayEventsProcessed(): number
31+
getReplayEstimatedTotal(): number
32+
getReplayStartedAt(): number // Date.now() timestamp
2833
initIndexState(): Promise<void> // call once at app mount
2934
destroyIndexState(): void // call at app teardown
3035

@@ -35,14 +40,17 @@ initIndexEvents(onDirUpdated: (paths: string[]) => void): Promise<UnlistenFn>
3540
## Scan state (`index-state.svelte.ts`)
3641

3742
Module-level `$state` variables (`scanning`, `entriesScanned`, `dirsFound`, `aggregating`, `aggregationPhase`,
38-
`aggregationCurrent`, `aggregationTotal`, `aggregationStartedAt`) react to six Tauri events:
43+
`aggregationCurrent`, `aggregationTotal`, `aggregationStartedAt`, `replaying`, `replayEventsProcessed`,
44+
`replayEstimatedTotal`, `replayStartedAt`) react to eight Tauri events:
3945

4046
| Event | Payload | Effect |
4147
| ---------------------------- | --------------------------------------------------- | ------------------------------------------------------- |
4248
| `index-scan-started` | `{ volumeId }` | `scanning = true`, counters reset |
4349
| `index-scan-progress` | `{ volumeId, entriesScanned, dirsFound }` | Update counters |
4450
| `index-scan-complete` | `{ volumeId, totalEntries, totalDirs, durationMs }` | `scanning = false`, set final counts, reset aggregation |
4551
| `index-rescan-notification` | `{ volumeId, reason, details }` | Show info toast with reason-specific message |
52+
| `index-replay-progress` | `{ volumeId, eventsProcessed, estimatedTotal }` | `replaying = true` on first, update counters |
53+
| `index-replay-complete` | `{ volumeId, durationMs }` | Reset replay state |
4654
| `index-aggregation-progress` | `{ phase, current, total }` | `aggregating = true`, update phase/progress/ETA |
4755
| `index-aggregation-complete` | `()` | Reset aggregation state, dismiss overlay |
4856

@@ -63,19 +71,29 @@ comparison (relies on trailing-slash normalization).
6371

6472
## Scan status overlay (`ScanStatusOverlay.svelte`)
6573

66-
Rendered in the top-right corner of the main window while `isScanning()` or `isAggregating()` is true. Uses
67-
`pointer-events: none` so it never blocks clicks. Two modes:
74+
Thin wrapper that computes label, progress, and ETA from scan/aggregation state, then delegates rendering to
75+
`$lib/ui/ProgressOverlay.svelte`. Visible while `isScanning()` or `isAggregating()` is true.
6876

69-
- **Scan phase**: Spinner + live label like `Scanning... 42,000 entries, 1,200 dirs`.
70-
- **Aggregation phase**: Spinner + phase label (for example, "Computing directory sizes...") + progress bar with real %
71-
and ETA estimate. Phases: `saving_entries` (flushing writer backlog), `loading`, `sorting` (no progress bar),
72-
`computing`, `writing` (progress bar on `saving_entries`, `computing`, and `writing`). ETA is computed from elapsed
73-
time and current/total ratio, reset on phase transitions. Progress bar uses `--color-accent` fill with smooth CSS
74-
transition.
77+
- **Scan phase**: Label only (compact layout, no progress bar). Shows "Scanning... 42,000 entries, 1,200 dirs".
78+
- **Aggregation phase**: Column layout with phase label + optional progress bar + ETA. Phases: `saving_entries`
79+
(flushing writer backlog), `loading`, `sorting` (no progress bar), `computing`, `writing` (progress bar on
80+
`saving_entries`, `computing`, and `writing`). ETA is computed from elapsed time and current/total ratio, reset on
81+
phase transitions.
7582

7683
Uses `formatNumber` from selection-info-utils for number formatting (uses `'en-US'` locale, hardcoded via
7784
`toLocaleString('en-US')`).
7885

86+
## Replay status overlay (`ReplayStatusOverlay.svelte`)
87+
88+
Thin wrapper that computes label, progress, and ETA from replay state, then delegates rendering to
89+
`$lib/ui/ProgressOverlay.svelte`. Visible when `isReplaying()` is true AND more than 4 seconds have elapsed since replay
90+
started, AND not currently scanning or aggregating (to avoid stacking overlays).
91+
92+
- **Progress bar**: `eventsProcessed / estimatedTotal` ratio.
93+
- **ETA**: 50-50 blend of total-based ETA (elapsed extrapolation) and a sliding-window rate over the last ~5 seconds.
94+
Falls back to whichever is available if one can't be computed yet.
95+
- **Detail**: Shows "{N} events processed" with locale formatting.
96+
7997
## Key decisions
8098

8199
**Decision**: "Listen first, then query" initialization pattern in `initIndexState`. **Why**: The Rust indexer starts in
@@ -98,3 +116,4 @@ No unit or integration tests exist for this module yet. Manual testing via the R
98116
- `$lib/tauri-commands``listen`, `UnlistenFn`
99117
- `$lib/ui/toast``addToast` (rescan notification toasts)
100118
- `$lib/file-explorer/selection/selection-info-utils``formatNumber` (overlay only)
119+
- `$lib/ui/ProgressOverlay.svelte` — reusable progress overlay component (used by `ScanStatusOverlay`)
Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
<script lang="ts">
2+
import {
3+
isReplaying,
4+
getReplayEventsProcessed,
5+
getReplayEstimatedTotal,
6+
getReplayStartedAt,
7+
isScanning,
8+
isAggregating,
9+
} from './index-state.svelte'
10+
import { formatNumber } from '$lib/file-explorer/selection/selection-info-utils'
11+
import ProgressOverlay from '$lib/ui/ProgressOverlay.svelte'
12+
13+
const replaying = $derived(isReplaying())
14+
const eventsProcessed = $derived(getReplayEventsProcessed())
15+
const estimatedTotal = $derived(getReplayEstimatedTotal())
16+
const startedAt = $derived(getReplayStartedAt())
17+
const scanning = $derived(isScanning())
18+
const aggregating = $derived(isAggregating())
19+
20+
const showDelayMs = 4000
21+
22+
let delayElapsed = $state(false)
23+
let delayTimer: ReturnType<typeof setTimeout> | undefined
24+
25+
$effect(() => {
26+
if (replaying && startedAt > 0) {
27+
const remaining = showDelayMs - (Date.now() - startedAt)
28+
if (remaining <= 0) {
29+
delayElapsed = true
30+
} else {
31+
delayTimer = setTimeout(() => {
32+
delayElapsed = true
33+
}, remaining)
34+
}
35+
} else {
36+
delayElapsed = false
37+
if (delayTimer !== undefined) {
38+
clearTimeout(delayTimer)
39+
delayTimer = undefined
40+
}
41+
}
42+
43+
return () => {
44+
if (delayTimer !== undefined) {
45+
clearTimeout(delayTimer)
46+
delayTimer = undefined
47+
}
48+
}
49+
})
50+
51+
const visible = $derived(replaying && !scanning && !aggregating && startedAt > 0 && delayElapsed)
52+
53+
const progress = $derived(estimatedTotal > 0 ? Math.min(1, eventsProcessed / estimatedTotal) : 0)
54+
55+
const detail = $derived(`${formatNumber(eventsProcessed)} events processed`)
56+
57+
// Sliding window for rate estimation: snapshots of { timestamp, eventsProcessed }
58+
// over the last ~5 seconds, pruned on each update.
59+
const windowDurationMs = 5000
60+
61+
let windowSnapshots = $state<{ timestamp: number; eventsProcessed: number }[]>([])
62+
let lastSnapshotProcessed = -1
63+
64+
/** Update the sliding window when eventsProcessed changes; reset when replay stops. */
65+
$effect(() => {
66+
if (!replaying) {
67+
windowSnapshots = []
68+
lastSnapshotProcessed = -1
69+
return
70+
}
71+
const processed = eventsProcessed
72+
if (processed !== lastSnapshotProcessed) {
73+
const now = Date.now()
74+
windowSnapshots.push({ timestamp: now, eventsProcessed: processed })
75+
lastSnapshotProcessed = processed
76+
// Prune old snapshots
77+
const cutoff = now - windowDurationMs
78+
const firstValidIndex = windowSnapshots.findIndex((s) => s.timestamp >= cutoff)
79+
if (firstValidIndex > 0) {
80+
windowSnapshots = windowSnapshots.slice(firstValidIndex)
81+
}
82+
}
83+
})
84+
85+
/** Compute sliding-window rate ETA from recent snapshots. */
86+
function computeWindowEta(
87+
snapshots: { timestamp: number; eventsProcessed: number }[],
88+
remaining: number,
89+
): number | null {
90+
if (snapshots.length < 2) return null
91+
const oldest = snapshots[0]
92+
const newest = snapshots[snapshots.length - 1]
93+
const windowElapsed = (newest.timestamp - oldest.timestamp) / 1000
94+
if (windowElapsed <= 0) return null
95+
const windowRate = (newest.eventsProcessed - oldest.eventsProcessed) / windowElapsed
96+
return windowRate > 0 ? remaining / windowRate : null
97+
}
98+
99+
/** Blend two ETA estimates 50-50, falling back to whichever is available. */
100+
function blendEtas(a: number | null, b: number | null): number | null {
101+
if (a != null && b != null) return (a + b) / 2
102+
return a ?? b
103+
}
104+
105+
/** Format an ETA in seconds to a human-readable string. */
106+
function formatEta(seconds: number): string {
107+
if (seconds < 2) return 'Almost done'
108+
if (seconds < 60) return `${String(Math.round(seconds))}s left`
109+
return `${String(Math.round(seconds / 60))}m left`
110+
}
111+
112+
const eta = $derived.by(() => {
113+
if (!replaying || eventsProcessed === 0 || estimatedTotal === 0 || startedAt === 0) return null
114+
115+
const remaining = estimatedTotal - eventsProcessed
116+
if (remaining <= 0) return 'Almost done'
117+
118+
const elapsedSec = (Date.now() - startedAt) / 1000
119+
const totalBasedEta = elapsedSec > 0 ? elapsedSec * (remaining / eventsProcessed) : null
120+
const windowBasedEta = computeWindowEta(windowSnapshots, remaining)
121+
const blended = blendEtas(totalBasedEta, windowBasedEta)
122+
123+
return blended != null ? formatEta(blended) : null
124+
})
125+
</script>
126+
127+
<ProgressOverlay {visible} label="Updating index..." {detail} {progress} {eta} />

0 commit comments

Comments
 (0)