Skip to content

v0.19.0

Choose a tag to compare

@github-actions github-actions released this 25 Oct 08:25
· 33 commits to main since this release
Immutable release. Only release title and notes can be modified.
140425f

Searches keep running while the watcher applies changes, the index checks
itself and survives crashes and full disks, and fcs gains diff impact
(fcs impact, MCP diff_impact) and diagnostic captures. The web UI's
Ranking setting is gone: every search is ranked the same way.

Upgrading: no re-index and no config change is needed. The background
scrub ([diagnostics] scrub) is on by default and reads about 3,000 files
a minute; set scrub = false to turn it off. fcs --rank and rank in
the REST and gRPC APIs still work but no longer change the order with
default settings. Release binaries are about 3 MB larger because they keep
function names for readable stacks.

Changed

  • The web UI's Ranking setting (Auto, Fast, Full) is gone. With the
    default settings all three gave the same results; only a server with a
    file limit (search.fast_rank_files) told them apart. Every search is
    ranked the same way, and BEST MATCHES FIRST beside the result count
    now opens a short explanation of the order: how hits are scored, that a
    file's hits stay together, that nothing is hidden, and what happens on a
    very broad search. MCP's search_code no longer lists rank (an old
    client's value is accepted and ignored) and says how results are ranked
    in its description. fcs --rank is hidden but still works, as does
    rank in the REST and gRPC APIs. The book's Ranking page has the
    measurements behind the choice: stopping at the first page is up to 50
    times faster on tokio + Django, but for common words that page holds none
    of the 50 best hits.
  • lock_wait captures leave out the holding back. A search holds back
    while an install step waits (up to 2 s, as long as recent searches
    take), and that alone could pass the 2 s threshold. The capture now
    counts only the time a writer kept the search out, so it still means a
    step or a save held the lock longer than it should.
  • rdeps: and deps: filters no longer read every indexed path to find
    their file (when it is given as results print it, with or without :N)
    nor copy every unresolved import to count the broken ones. On a
    348,000-file index the filter's own cost fell from 464 ms to 0.1 ms
    median (examples/rdeps_timing.rs); a partial path or a folder still
    reads every path.
  • The server accepts request bodies up to 8 MiB (was 64 KiB), for diffs.

Fixed

  • Building the index got about 30% slower over October, mostly for
    nothing.
    A profile on tokio + Django put 42% of the build's CPU in
    Python import resolution and another fifth of Django's in filtering
    Python references, not in parsing. For every absolute import (most of
    them the standard library), resolution built and normalized candidate
    paths under every folder above the importing file; it now asks one
    folder lookup whether anything is indexed under root/<first name>
    before building any. The reference filter looked each reference up from
    the root of the syntax tree and climbed with a call that itself walks
    down from the root; it now walks each file's tree once. Results are
    unchanged (same 9,784 graph edges and 384,986 references; the new
    filter keeps exactly what the old one kept for all 492,103 identifiers
    in Django). Import resolution went from 2.4 s to 0.85 s, the whole
    build from 8.1 s to 6.7 s on the same machine (0.15.2: 6.1 s; the rest
    is the extra Python references and case folding). The corpus benchmark
    now reports the time of each build phase.
  • A search for a common word spent a third of its time sorting hits it
    never returned.
    Every hit (22,000 for return on tokio + Django) was
    sorted, then sorted again by file, to return one page; and each
    candidate file's src/test path signals were worked out again on every
    search. Now each file's best hit places the file, only the files the
    page reaches have their hits sorted, and the signals come from the
    file's precomputed metadata. Results are identical (53,810 results
    compared across 14 queries, both ranking modes and pages down to
    offset 20,000); return went from 51 to 39 ms, a filtered search from
    28 to 21 ms, a regex with no literal from 76 to 61 ms.
  • The count of changes still to apply counted events, not files. One
    save raises several file-system events, and each was counted, so the
    web page's "N file changes to apply" (and the same count in fcs, MCP
    and /api/status) overstated the backlog: 372,534 for 250,000 changed
    files in a 500,000-file scale run. Each file now counts once until its
    change is applied.
  • class ModelForm put class ModelFormOptions: first. Since plain
    searches match like rg -F (0.15.2), a query is also found inside longer
    names, and the shorter line won. A whole-word match now scores 1.5x, so
    the line that has what was typed comes first; the ranking benchmark is
    back to 20 of 20 at rank 1.
  • The benchmark history stopped on 13 Oct 2026. The ranking check in
    the Benchmarks workflow failed from that release on, and its failure
    skipped the steps that store the numbers, so the trend page stopped
    without a warning. The numbers are now stored first and the check fails
    the run after.
  • Loading an index with one changed file took about 70% longer.
    When any saved file had changed or gone, loading rebuilt every trigram
    posting list one id at a time through a hash lookup. Now that a file
    saved within seconds of an edit is re-read on load (below), the corpus
    benchmark hit this on every run, as does a restart soon after an
    edit. The lists are now rebuilt in one sorted pass from a lookup table,
    in parallel: on tokio + Django, load after 20 edits went from about
    560 ms to 365 ms (a clean load is about 310 ms). Saves after deletes
    use the same code.
  • Every save killed half way left a file as large as the index. A
    save writes a temp file next to the index and renames it into place; a
    server killed during the save (kill -9, a power cut, the OOM killer)
    left the temp file, and nothing removed it. The server now removes them
    when it starts.
  • A save that failed was forgotten. With save_after_updates, the
    count of unsaved updates was reset whether the save worked or not, so
    after a save onto a full disk the shutdown skipped its save and the
    saved index lacked those updates (the next start re-read the files, so
    no search was wrong, but it took longer). The count now stays until a
    save works.
  • A failed rename could delete the index. When moving the new index
    into place failed, the old index was deleted before trying again, so a
    second failure left no index at all. The old index is now moved aside
    and put back if the second try fails.
  • A panic while reading a file left it out of the index without a
    word.
    During the build, a file whose reading or parsing panicked was
    skipped and not reported. It is now tried again at the end of the
    build, and reported as unreadable if it keeps failing, like a file that
    cannot be read.
  • Every search waited 4 to 5 s soon after a restart. An index loaded
    from disk keeps its unresolved imports (import os, packages outside
    the indexed folders) but not each file's import list. At the end of the
    restart's check of the tree, each of those imports got one last try, all
    at once under the index lock: 368,000 of them on a tree of 200,000
    files. They are now tried in short steps like the rest of the import
    graph; the slowest search after a restart fell from 4.0 s to under 1 s.
  • The backlog diagnostic capture fired under steady edits. It fired
    when changes had been pending for 300 s without a break, which a steady
    stream of edits always causes. It now fires only when the watcher could
    not have applied what is pending in the last 10 s.
  • The watcher could fall far behind while people searched. While an
    install step waits for the index, new searches hold back so the running
    ones can finish and the step gets in. They held back a fixed 250 ms, so
    with searches longer than that a new one got in before the running ones
    ended and the step kept waiting: in a simulation with four people
    running half-second searches back to back, a change that takes 5 s to
    install took up to 62 s. A search now holds back for one and a half
    times the 90th percentile of recent searches' length (250 ms to 2 s);
    the same change takes about 10 s, and quick searches still wait only
    250 ms behind one slow one. The docs claimed "half speed or better"
    regardless; the book's "Incremental updates" page now has the table.
  • The watcher could index files the build leaves out, by ignore rules.
    When a file changed, the watcher decided whether it was ignored
    differently from the walk that builds the index (and from ripgrep): a
    rule deeper down (!a/b/ after /a) let files back in from a folder
    that is ignored; the nearest ignore file decided, where a parent's
    .ignore or .rgignore rule should beat a .gitignore next to the
    file; and rules in folders above a nested repository were not applied
    inside it. Such files became searchable after an edit and disappeared
    again at the next restart. The check now follows ripgrep's rules, and
    both are compared with rg --files on random trees of ignore files.
  • Folders ignored by a parent's .gitignore stayed out only until
    something happened in them.
    With sub/ in the root .gitignore, the
    watcher indexed everything in pkg/sub when the folder was created,
    renamed in, had an ignore file written in it, or was rescanned after
    lost events: a walk never checks whether the folder it starts from is
    ignored. It now does.
  • A restart re-indexed changed files that are now ignored. A file that
    changed while the server was down was re-read without checking the
    ignore rules (only the size cap and excludes), so a file a new
    .gitignore rule covered came back if it had also changed. Changed files
    now go through the same rules as discovery.
  • A file rewritten within the same second could stay stale. On file
    systems with coarse timestamps (FAT, HFS+, some network shares), a file
    rewritten with the same size in the same tick it was indexed kept its
    old content in the index across a restart or a rescan after lost events,
    because its size and time had not changed. A file read within 3 seconds
    of being modified is now read again by the next rescan or restart, as
    git does with "racily clean" files.
  • Short queries missed very small files. A file shorter than three
    bytes with a name shorter than three letters (x.py holding ab) has no
    trigram, and queries without a trigram of their own (two characters, a
    regex with no literal) scanned only files some posting list named, so rg
    found the line and fcs did not. The index now keeps the set of all its
    files, after a load and incremental changes too.
  • indexer.extra_binary_extensions works. The matching reference
    documented it, but the configuration rejected it as an unknown key. Every
    file walk and eligibility check now takes its settings from one place.
  • Rewriting a large file while it was searched could crash the server.
    Files over 1 MB were memory-mapped, and when another program truncated
    and rewrote one while a search read it, the server died with SIGBUS.
    Files are now read into memory when searched, so nothing can change
    under a search. The 3-second "vm.max_map_count is very low" pause at
    startup is gone with the mapping.
  • No more "server unavailable" at the end of a large build or while a
    saved index loads.
    Resolving the dependency graph at the end of a
    build held the write lock for seconds (about 6 s on 300,000 files), and
    loading a saved index held it for the whole load, so searches were
    turned away. Both now work outside the lock and hold it only for short
    steps or the final swap; a 300,000-file cold build and restart answered
    every search.
  • Ignore files behave like ripgrep's. .rgignore is honoured, and
    when a .gitignore, .ignore or .rgignore changes (a branch switch,
    say) the watcher re-checks the files under it, so newly ignored files
    leave the index and newly allowed ones join it. Before, each file kept
    its old state until it changed itself.
  • A file that can't be read is retried and reported, not silently
    dropped.
    A read that failed (a file locked by an editor or virus
    scanner, permissions changing mid-save) removed the file from the index
    until it next changed, and the initial build left such a file out
    without a word. The watcher now keeps the old entry and retries for
    about half a minute; then the file is left out and every search says
    how many files could not be read. A panic while applying watcher
    changes used to skip the whole batch; the batch is now applied again,
    and a change that fails twice is reported the same way.
  • A large change no longer leaves the index "updating" for minutes
    while people search.
    Each watcher step got the lock only after the
    running searches finished and then installed just 512 files, so with
    searches running the watcher applied a few hundred files a second. On a
    200,000-file tree, 100,000 rewritten files were still 17,000 behind four
    minutes later. A step now stays in about as long as it waited, up to
    one second; the same change is applied 10 s after it stops, and on a
    million-file tree 200,000 changed files catch up in about 3.5 minutes,
    with no search turned away. examples/watch_load.rs now checks that
    the index matches the disk again after the load.
  • Searches no longer stop while the file watcher applies many changes.
    One slow search plus any watcher step used to stop every other search
    until the slow one finished, and those still waiting after five seconds
    got "Index is currently being updated". The watcher (and the startup
    build) now take the write lock only in a moment when no search holds it,
    so a search waits for a short install step at most, never for other
    searches. On a million-file tree with 50,000 files rewritten at once,
    searches turned away fell from 3% to none and the slowest search from
    5.1 s to 1.5 s, while the watcher applied 5.6 times as many changes in the
    same time. Measurements are in the book (How it works > Incremental
    indexing).

Added

  • More ways the tests look for wrong answers. A model-based test
    (tests/lifecycle_model.rs) drives the index through random sequences of
    edits, renames, folder moves, ignore-file changes, lost events, saves,
    restarts and crashes, and after every step compares it with a naive
    model, page by page; it found the first two fixes above. Property tests
    (tests/search_properties.rs) compare literal and regex searches with
    ripgrep on random text full of case-folding oddities (ß, ſ, the
    Kelvin sign, the sigmas, dotted and dotless i). A generator of seeded
    Python codebases with an answer key (tests/python_codebase.rs) checks
    the import graph, symbol and reference searches and strict
    -deps/-rdeps scopes, after a build and after the watcher rewrites the
    tree; grimp and Python check the key itself. Book page: How it works >
    How fcs is tested.
  • Every surface walked to the last hit, and the first page pinned.
    tests/contract_walk.rs runs searches with thousands of hits through
    REST, gRPC, MCP (HTTP and fcs mcp), fcs and the web page in headless
    Chromium, pressing LOAD MORE and "Show N more hits" until they are gone,
    and every surface must show the same hits in the same order with honest
    counts. tests/rank_snapshot.rs keeps the first page of about a hundred
    searches in a file, so a change that reorders results shows in its pull
    request. The ripgrep backtest also runs on an index restarted from disk
    and on one the watcher filled (--history reload|watch), with regexes
    made up from the corpus (--generate-regex), and
    tests/discovery_oracle.rs compares the indexed files and the FILES box
    with rg.
  • The import graph is checked on monorepo layouts. Generated
    repositories shaped like real ones (scripts/graph-oracle/layouts.py,
    ts_layouts.mjs): Python projects side by side each with its own
    src/, a root folder that is a package, a namespace package split over
    two folders, and npm workspaces with tsconfig paths and extends,
    barrels and .mts/.cts. grimp and TypeScript give the answer. Every
    edge fcs misses or draws wrongly, there and on Django and Vite, is now
    a known gap listed with its reason and exact count in
    benches/graph/known_gaps.toml, instead of a count to stay under. The
    Python oracle (fcs check included) now builds one graph across
    ROOT and ROOT/src, so imports from one into the other are compared
    too.
  • Kill and restart. examples/chaos.rs kills the server binary with
    SIGKILL at random moments (during a build, a load from disk, a stream of
    edits, a periodic save, a burst of changes, and the save at shutdown),
    starts it again and checks that it loads the saved index, leaves no temp
    file, catches up with every change and passes /api/check. The "Scale
    scoreboard" workflow runs it nightly and on pull requests that touch
    saving or watching (job "Kill and restart").
  • More fault points (--features failpoints, src/faults.rs): out of
    file handles, a file gone between listing and reading, a slow disk, a
    panic while reading or parsing a file, events lost by the OS, a save
    onto a full disk and a failed rename, each with a test in
    tests/fault_injection.rs.
  • A scale scoreboard. examples/scoreboard.rs runs the server binary
    over HTTP in seven scenarios (cold build, warm restart, branch switch, a
    ten-minute stream of edits, overloaded searching, saves under load, and
    an event-queue overflow) and reports the same numbers for each: searches
    turned away, latency, time until the index matches the disk, memory,
    CPU, diagnostic captures, crashes and what /api/check finds. Targets
    are in benches/scoreboard/targets.toml; the "Scale scoreboard"
    workflow runs it at 200,000 files nightly and on pull requests that
    touch locking or watching. Book page: Benchmarks > Scale scoreboard.
  • The lock protocol on a simulated clock. fast_code_search::sim runs
    threads in turns on simulated time, and the engine lock and the
    watcher's install steps read and sleep through it, so
    tests/lock_sim.rs runs the real code under minutes of made-up search
    load in about a second, the same on every run and OS. It found the
    watcher fix above, and its tuning table is the evidence for the
    hold-back and MAX_HOLD values.
  • The index checks itself. fcs check --live asks the running server
    whether its index agrees with itself, whether indexed files still match
    the disk, and whether it holds the files a fresh walk finds. It prints
    counts only, so the output is safe to share (--show-paths for file
    names); GET /api/check returns the same as JSON. A background scrub
    ([diagnostics] scrub, on by default) reads 3,000 files a minute, about
    0.1% of a core. It logs and re-indexes any file the index has out of
    step, which is a lost change, and checks the index's own consistency
    hourly; the counts are on the diagnostics page. The watch, fault and
    backtest harnesses now run the same checks after every scenario. Book
    page: Guides > Checking a live index.
  • Fault-injection tests. A failpoints cargo feature compiles in
    named fault points (src/faults.rs) that tests arm to make a read fail
    or a step panic; tests/fault_injection.rs uses them to check that the
    build and the watcher retry and report failures. CI runs them on every
    platform.
  • Diagnostic captures. With [diagnostics] capture = true the server
    notices when it is slow or stuck and writes what it was doing to a JSON
    file: a search slower than 5 s or kept waiting over 2 s for the index
    lock, the watcher or index build making no progress for 120 s, a search
    still running after 120 s, the index lock never free, or a watcher
    backlog lasting 300 s. A capture holds the trigger, the search's phase
    timings, the searches running, the index and watcher state, recent
    watcher bursts, memory and threads, the last log lines and, for a hang,
    every thread's state and stack (gdb or eu-stack when installed).
    Query text, paths and file names are left out unless
    include_queries = true. Captures are rate-limited and capped (50 files,
    100 MB). fcs diagnose [DIR] summarizes them, and /api/diagnostics
    says where they go. Off by default; thresholds are measured
    (watch_load --capture-dir). See the book's "Diagnostic captures" page.
    Release binaries now keep function names (about 3 MB larger) so thread
    stacks and panic backtraces are readable.
  • Diff impact: fcs impact, POST /api/impact, MCP diff_impact.
    Give it a diff (by default your uncommitted changes) and it lists, for
    each function, class, method, type or constant the diff changes, the files
    that use it, best first, with the use lines: finer than the file-level
    change_impact. Functions and classes are traced through importers that
    name them (renames followed), methods and fields by name among every
    importer with files naming the class first. Each use is a recorded
    reference, a code mention or a text mention (strings and comments).
    On Django's 248 most recent commits that change Python, checked against Jedi, it lists 100%
    of the files that use a changed function or class (79.5% of those it
    lists are confirmed by Jedi), in 49 ms median per commit. No re-index is
    needed. See the book's Diff impact
    page for what it misses.
  • Search responses report file changes the watcher has seen and not
    applied yet (index.pending_changes). The web UI shows "Index updating ·
    N file changes to apply"; fcs and the MCP tools print it as a note.

Install

Download the archive for your platform below, unpack it, and put fast_code_search_server and fcs on your PATH.
Each archive also contains config.toml.example, the startup unit files (deploy/) and the CLI and startup guides.

Platform Archive
Linux x86_64 (glibc) fast_code_search-v0.19.0-x86_64-unknown-linux-gnu.tar.gz
Linux x86_64 (static, musl) fast_code_search-v0.19.0-x86_64-unknown-linux-musl.tar.gz
macOS Apple silicon fast_code_search-v0.19.0-aarch64-apple-darwin.tar.gz
macOS Intel fast_code_search-v0.19.0-x86_64-apple-darwin.tar.gz
Windows x86_64 fast_code_search-v0.19.0-x86_64-pc-windows-msvc.zip

Verify a download with the .sha256 file next to it: sha256sum -c fast_code_search-v0.19.0-<target>.tar.gz.sha256.
Each archive is also signed through Sigstore; with cosign installed:

cosign verify-blob --bundle fast_code_search-v0.19.0-<target>.tar.gz.sigstore.json \
  --certificate-identity-regexp '^https://github.com/jburrow/fast_code_search/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  fast_code_search-v0.19.0-<target>.tar.gz

Other ways to install: cargo binstall fast_code_search (downloads these archives), cargo install fast_code_search (builds from crates.io), or the container image ghcr.io/jburrow/fast_code_search:0.19.0.

First run: fast_code_search_server --init config.toml, add your paths, start the server, then fcs 'fn main'.
Full changelog: https://github.com/jburrow/fast_code_search/blob/main/CHANGELOG.md