perf(picker): stream and cache file discovery

[fcoury-oai]

Find Files waits for the entire workspace scan before showing results, and filtering large file lists can block input. On a checkout with more than a million eligible files, this leaves the picker looking stuck on “Loading files…”.

This PR streams discovery results into the picker and moves matching and sorting off the UI thread. Typing, navigation, and cancellation remain responsive while files arrive. Existing ignore rules, filename/path ranking, previews, and file:line navigation are preserved without a file-count cutoff.

Completed indexes are reused within the editor. Ctrl+r refreshes the index; Ctrl+e switches hidden/ignored visibility. Refresh keeps the previous complete results searchable, including after cancellation. Automatic selection follows the best match until the user navigates, after which updates preserve the selected path. Enter cannot accept a result from an older query, and discovery errors end with an explicit incomplete status.

The cache is keyed by canonical root and visibility. It refreshes on reopening after 30 seconds or after known file-list changes; ordinary saves of indexed source files do not rescan. External changes require refresh or expiry. Unused indexes are evicted beyond four entries or 1 GiB of estimated row storage. Active indexes remain complete, so this is not a process-memory limit and refresh can temporarily retain two snapshots. The behavior and benchmark are documented in docs/performance.md.

Validation

Validated the pushed head 45a65a2c: 3,549 tests passed, 3 ignored, with formatting and Clippy clean across all targets and features. Seven TUI smoke scenarios passed, including cancelled refresh, visibility rules, rapid query changes, and line navigation.

On a local checkout with 1,252,204 eligible files, two final release benchmark runs produced the same complete path hash as the serial walker:

Measurement	Final runs
First results	135–138 ms
Complete index	5.008–5.020 s
Cached reopen	6–7 ms
Peak process RSS	1.09–1.12 GiB

The final TUI pass showed initial results in 122 ms; full discovery took 4.96 s initially and 13.24 s during refresh. These are local measurements, not CI thresholds: filesystem caches were not flushed and host load was not controlled. LSP and AI were disabled in the isolated smoke profiles.

How to Test

1. Build with cargo build --release --bin red, then run target/release/red --root /path/to/large-workspace /path/to/large-workspace/existing-file. Press Ctrl+p and type a known filename while scanning. Results should appear before scanning completes, input should remain responsive, and better matches should rise to the top until you navigate the list. After navigating, the selected path should remain stable as more files arrive.

2. After discovery completes, close and reopen the picker within 30 seconds. Results should appear from the cache. Search for a file with at least 20 lines using filename:20, wait for matching to finish, then press Enter; it should open at line 20.

3. Press Ctrl+r, close the picker while it reports refreshing, then reopen it. The previous complete results should remain searchable while the replacement scan runs. Add a file externally and refresh again; the new file should appear. In a fixture with hidden and ignored files, Ctrl+e should expose those files while still excluding .git and .bare metadata.

4. Run cargo test --release --lib ui::file_picker and cargo test --release --lib workspace_paths::tests. These cover stale-query acceptance, manual selection, refresh cancellation, cache invalidation, missing roots, and parallel/serial file-set parity. For a large-workspace measurement with full parity checking, run:

   RED_FILE_PICKER_BENCH_ROOT=/path/to/large-workspace RED_FILE_PICKER_VERIFY_PARITY=1 \
     cargo test --release --lib file_picker_streaming_large_workspace_performance \
     -- --ignored --nocapture --test-threads=1

   Expect the complete file count and hash to match the serial walker, with separate timings for first results, complete discovery, queries, cached reopening, and cancellation.