Skip to content

Releases: Collegium-Siderum/bookrack

v0.13.0

Choose a tag to compare

@github-actions github-actions released this 24 Oct 15:10

Added

  • An error the control plane could not classify is logged on the
    daemon side.
    A handler failure matching no known error type, and
    an ops error wrapping an untyped report, were answered as -32603
    and recorded nowhere but in the client's reply. The daemon now emits
    one warning for each — unclassified handler error with the method
    name and the flattened chain, unclassified ops error with the chain
    — so it reaches the rolling log, bookrack logs, the log event
    channel and the diagnose bundle. The reply is unchanged, and a
    failure the classifier judged a fault is not logged.
  • A failed queue job records why, in parts. The queue worker runs
    the failure through the classifier the control plane uses for a write
    RPC, and keeps the result on the row as problem: summary, detail,
    hint, and retryable, the same four parts the error envelope carries.
    queue.list, the last_finished of queue.tick, and the recent
    rows of MCP session.queue_status carry it; error stays as it was.
    The queue document moves to schema 7; a document from an earlier build
    loads unchanged, and an earlier build refuses one this build has
    written, naming both versions. On the command line, ingest --wait,
    papers ingest --wait, intake ocr --wait and queue wait name a
    failed job by the recorded summary instead of the whole chain and
    print each distinct hint under the summary line; queue list --state failed prints the hints under each cause header. Under --json, a
    record whose failure was classified carries problem.
  • MCP session.queue_status says why a job failed, and can list one
    state.
    A recent row carries finished_at once the job has ended
    and error when it failed, with a dependency source location
    crate-relative as on the error envelope. The new optional state
    argument lists only jobs in that state, under the same cap of ten,
    newest enqueued first; the counts still cover the whole queue, so
    {"state": "failed"} reaches a failure that has scrolled out of the
    unfiltered window. A state no row can be in is refused as invalid
    params, with the accepted names in the hint.
  • bookrack queue wait re-attaches to queued work. It waits on
    every job that is pending or running when the queue is read — jobs
    submitted later are not waited on — and ends with the summary and
    exit code ingest --wait uses: 5 when any job failed or was
    cancelled. --job <id> names jobs instead, by any part of their id,
    and may repeat; a named job that has already finished is reported as
    it ended, so queue retry followed by queue wait --job reports a
    failure even when the retried job finished first. A paused queue
    with selected pending jobs, or a daemon without a queue worker, is
    refused up front with exit 2 instead of waiting on work nothing
    will run. Both are checked once, when the wait attaches;
    docs/operating.md lists what a wait does not see after that.
  • Terminal queue jobs can be retried. The new control-plane method
    queue.retry flips failed, needs_ocr, or cancelled rows back
    to pending in place — one row by its id or any part of it, or
    every row in one state — clearing the previous run's error and timestamps, so the
    worker re-runs them as fresh submissions. Nothing else in the queue
    document moves, and a paused queue holds the rows until resume.
    bookrack queue retry is the verb: with no flag it retries every
    failed row, --state needs-ocr or --state cancelled picks another
    state, --job <id> picks one row, and the human output lists the
    ids it flipped.
  • queue list can show one state, capped, with failures grouped by
    cause.
    --state <s> keeps the rows in that state and --limit N
    shows at most N of them, oldest first; the per-state footer still
    counts the whole queue, and a capped listing says showing N of M.
    --state failed groups its rows under one header per cause with the
    group's size, so a long failed set reads as a handful of reasons
    instead of a wall of rows. Under --json the jobs array carries the
    selected rows and the rest of the reply is the daemon's.
  • distill build shows which stage of which book is running. The
    build ran each book's pipeline without a word until the book's stage
    table printed. It now draws 2/5 <slug> [3/7] <stage> on stderr,
    naming a stage as it starts; with stderr not a terminal it appends
    one such line per stage, and under --json or --quiet it draws
    nothing. distill lint and distill verify are unchanged.
  • A book's embedding pass reports how far it has got. While the
    queue worker embeds a book, worker.progress carries stage: embed
    with stage_progress set to the fraction of the book's chunks
    written to the vector store — always for the first and the last
    batch, throttled in between — so ingest and intake ocr show a
    percentage during the longest stage instead of a bare spinner. Paper
    jobs report processing only.
  • A write's reply names the library it acted on. Every control-plane
    method that takes the write mutex — the vectors, stamps,
    corpus, metadata, remove, and dryrun writes on both sides,
    and library.fork / mount / unmount — answers with library:
    the registry name the call resolved to, which for an unnamed call is
    the default pointer at that moment. The reply kept no trace of it, so
    an operator had nothing but memory to tell which library a write had
    landed on once library.set_default had moved the pointer. A reply
    that already carried fields keeps all of them; library.fork
    reports the source library. papers metadata writes end their
    sentence with Library: <name>.
  • ingest.cancel returns the full id of the job it cancelled. The
    method takes a prefix and answered { ok: true }, so the caller
    could not tell which job the prefix had matched; the reply now
    carries job_id.

Fixed

  • Book-side reads on a library that has not ingested yet answer
    instead of failing.
    The book catalog is created by init or the
    first ingest, so a root that libraries add registered and nothing
    wrote to has none; the book reads opened it read-only and surfaced
    the SQLite open failure as an internal error. That took down
    bookrack list, find, stats, categories, ocr-pending, the
    per-book show family, the metadata listings and the audit trails,
    and the matching control-plane methods and MCP tools, while
    search and doctor on the same root already read it as empty.
    Those reads now probe for the catalog first, the way the paper-side
    reads have since 0.12: listings return an empty page, stats and
    categories count zero, and a per-book read reports the intake as
    unknown. Nothing is created on disk by looking.

  • A registered library whose root carries no identity manifest is
    refused, not served empty.
    With a library's root on a volume that
    was not connected, the mount point was an empty directory that
    passed the existence check, and the daemon came up serving it as an
    empty library: searches answered nothing, status showed no counts,
    and an ingest wrote a second catalog, corpus, and vector store into
    the mount point, where the volume covered them the moment it came
    back. A registry entry that caches a uuid is now served only from
    a root whose bookrack-library.toml records that uuid. A root with
    no manifest is refused at bring-up and by libraries mount, with
    the hint to reconnect the volume first or re-register the library
    where it now is; a manifest recording another uuid is refused naming
    both. Nothing is locked or written to a refused root. An entry with
    no cached uuid is served as before, and doctor's registry row for
    a cached uuid without a manifest now says to reconnect the volume
    before re-registering, since re-registering against an empty mount
    point would mint a new identity for a directory that is not the
    library.

  • An unknown library name is refused with the names that would
    resolve as the hint.
    -32010 invalid library carried the
    registry's names inside the message, no library named "x" (available: a, b), and nothing in error.data. The message now
    states only the failure, and error.data.hint reads Use one of: a, b.; against an empty registry it names bookrack libraries register. The command line prints the hint under the summary line,
    so the names are still shown. The MCP tools' refusal of an unknown
    library argument carries the same parts.

  • A dry run no longer announces a library change. dryrun,
    papers.dryrun, and the dry_run legs of remove, papers.remove,
    corpus.rebuild, papers.corpus_rebuild, vectors.reembed and
    papers.vectors_reembed ended with a library.changed event, so a
    subscriber refreshed a library that had not changed. They still take
    the write mutex for the length of the preview — daemon.state reads
    writing, mcp.availability pauses and resumes, and a concurrent
    write sees -32001 busy — but the event is published only by the
    writes themselves.

  • A session or data-root lock that the filesystem refuses is no
    longer reported as held by another process.
    bookrack run and the
    daemon's data-root lock read every failure of the non-blocking lock
    call as a conflict: on a filesystem whose flock fails for some other
    reason (a network mount, some FUSE drivers) the error said bookrack session already running or bookrack data root already in use,
    quoted whatever pid the lock file last recorded, and the callers that
    branch on that verdict — the second-instance handoff, the daemon's
    "serve the root unlocked" fallback, libraries remove --purge — took
    the conflict branch. Only a lock held elsewhere is a conflict now; any
    other failure surfaces as an I/O error naming the lock file and its
    cause. The conflict me...

Read more

v0.12.0

Choose a tag to compare

@github-actions github-actions released this 16 Oct 18:40

Added

  • A daemon brought up on an unregistered root with an identity
    manifest registers it and serves the registry.
    bookrack run --data-dir <root> (or the data-root variable, or the portable
    layout) on a root the registry did not know served that root alone,
    anonymously, however complete its manifest. Bring-up now records the
    root under its manifest name, leaves the default pointer as it
    was, and comes up serving every registered library, as it would had
    libraries add run first. A root with no manifest, an unreadable
    manifest, or a name another root already holds is still served
    alone, with the reason in the daemon log. daemon.status reports
    the names a session registered as auto_registered, and
    bookrack status shows them on a registered_at_startup row; both
    are empty on the next start, which finds the entry in place.

  • -32018 STATE_UNUSABLE: a store this build cannot serve has its
    own error code.
    A catalog or corpus written by a newer schema or
    demanding a newer reader, an index without build stamps or with
    stamps that disagree with this build, and a vector sidecar naming an
    unknown ANN kind or a mismatched dimension all answered -32603,
    which tells a client to report a bug, and the CLI exited 1 on them —
    while every one of them has a next step that is the operator's to
    take. Those nine conditions now answer -32018, bare or through any
    pipeline wrapper, with the step in error.data.hint: run a newer
    build, bookrack corpus rebuild, bookrack vectors reset,
    bookrack vectors rebuild, or bookrack stamps reconcile. The CLI
    exits 2 on the code, as it does for caller input, through a variant
    of its own that keeps the hint. docs/control-plane.md adds the
    code to the list, the write-class mapping, and the exit-code table.

  • bookrack diagnose bundles now cover the paper stores, the
    reference store, and the queue document.
    A bundle carried the book
    pipeline's three stores and nothing else, so a fault in the paper
    catalog, a reference.db from another schema, or a queue document
    the daemon refuses to start on left no trace in it. The paper
    catalog, corpus, and vector sidecar now land under papers/ in the
    same shapes as their book-side sections; refs/summary.json records
    each distilled book's build provenance — schema, parser, build time,
    intake, entry and warning counts — and the entry and overlay totals,
    without any bibliographic column; queue/queue.json is the daemon's
    queue document re-serialised through the scrubber, so job paths are
    redacted like every other path. A store that is missing or cannot be
    opened records that in open-error.json, as the book sections
    already did; a queue document from a newer binary is reported as
    refused, with both versions named, rather than as malformed.

  • library.categories: the library-wide category distribution. A
    new control-plane method and MCP tool return every category tag with
    the number of books carrying it, most-used first, plus the books no
    tag reaches and the registry total. It is the browse entry the
    categories filter of library.find_books lacked: that filter asked
    the caller to name a tag without any way to see which tags the
    library carries, and a library nobody has tagged answered every such
    call with an empty page that read like "no such books". The
    distribution shows the vocabulary before the filter is used, and an
    untagged library reports itself as one — every book under
    uncategorised. Reachable from the CLI as bookrack rpc call library.categories; find_books's tool description now points at
    it.

Changed

  • An index-profile file from an earlier schema still loads. The
    loader refused any schema_version other than its own, so the first
    bump of that constant would have rejected every user-written profile
    in both directions, with no migration to run. It now reads its own
    schema and every earlier one, the way the library manifest does, and
    refuses only a file from a later one, naming both versions. A bump
    that changes the shape ships a reader for the previous schema in the
    same change; docs/UPGRADE.md carries the row.

  • bookrack --help opens with the first run. The trailer named
    run, rpc list, and doctor but never bookrack init, the step
    everything else assumes. It now says what init does and follows it
    with the same three-command sequence the wizard's closing screen
    prints, rendered from the same list, so the two surfaces cannot
    drift apart in wording.

  • doctor and library.info state what happened to a registry
    default a path-class selection passed over.
    The data-root row
    called the default "shadowed by" the flag or variable, which on a
    daemon serving the whole registry is not so: an unnamed call still
    reaches that default. Both surfaces now say which source selected
    the root and where the default points; the row keeps its remedy.

  • bookrack runs show reads its profiles: section from the run's
    rollup instead of re-counting the audit tables.
    The rollup row a
    run materialises at close (n_books, n_papers, the histograms)
    never moved, but the profile buckets beside it were counted live on
    every show. Both audit tables hold one projection row per item,
    rewritten by whichever run judges the item last, so after a
    glean --force or a per-item re-audit the buckets of the earlier run
    lost rows while its n_papers did not, and the two disagreed on the
    same screen. compute_run_summary now stores the buckets as a JSON
    column on pipeline_run_summary, and runs show renders that
    snapshot. Rollups materialised before this build carry no snapshot;
    for those the command still counts live and labels the section
    profiles (counted now, not at run close) with a note that the
    numbers need not add up. pipeline_run_id on an audit row is
    documented as the pointer to the pass that judged the item last, not
    as run membership. Catalog schema advances to user_version 17.

  • bookrack distill build --dry-run registers its run as
    distill_dryrun.
    A preview used to open a pipeline_runs row under
    distill_build, so bookrack runs list showed a rehearsal and a
    build that wrote reference.db as the same command, and a
    --command distill_build filter counted both. The preview now
    carries its own name, matching dryrun and papers_dryrun; it still
    writes its book_distill_audit rows, and runs show still renders
    them. Existing rows keep the name they were written under.

  • bookrack verify now judges its report, and prints it for a
    reader.
    Since 0.4.0 the command fetched the report over the control
    plane, printed it as JSON, and exited 0 whatever it said — a catalog
    whose schema failed verification and a healthy one returned the same
    code, so the exit-code table gave scripts nothing to branch on, and
    the renderer that used to draw the report for a person was never
    called. The command now renders the report as text (the raw result
    stays available under --json) and exits 1 when a finding says the
    library is damaged: a store that does not verify or cannot be read
    (catalog_schema_error, corpus_schema_error, intake_scan_error),
    an unreadable vector sidecar (vectors_meta_error), one store missing
    while the other is present, or an intake row whose file is gone
    (missing_intake_files). An uninitialised root, a library that never
    built its vector index, and any amount of churn exit 0: not built
    is not broken. The report is drawn before the exit code is set, so
    the exit code adds no line of its own. Scripts that parsed the
    default output as JSON pass --json. The exit-code table in
    docs/control-plane.md gains the source.

Fixed

  • The MCP library.info card describes the library it was asked
    about.
    With several libraries served, a call carrying library
    took its counts from the selected library but its identity
    (data_dir, library_name, resolution_source,
    embed_model_configured and the registry-identification fields)
    from a snapshot of the library the daemon came up on, so one card
    paired one library's counts with another's root. The MCP tool now
    builds the card's identity from the selected library, as the
    control-plane method already did; only the MCP listener address, a
    property of the process, still comes from the snapshot.

  • A running daemon stops returning the rows of a removed book.
    The vector store handle the daemon keeps warm for search was opened
    on a LanceDB connection that never checked for versions committed by
    other connections, so remove, vectors rebuild, and every other
    command that writes the store through its own handle left the search
    path reading the table as it stood at startup. Status reads, which
    open a fresh handle per call, reported the removal while search kept
    serving the removed book's chunks with an empty breadcrumb, and only
    an unmount and mount of the library brought the two into agreement.
    The store now opens its connection with a zero read-consistency
    interval, so every read resolves the latest on-disk version first.

  • Paper-side reads on a library that has not gleaned yet answer
    instead of failing.
    The paper catalog is created by the first
    glean, so a library that has only ingested books has none; the paper
    reads opened it read-only and surfaced the SQLite open failure as an
    internal error. That took down bookrack list in its default scope,
    bookrack stats, every papers read, and the MCP library.stats /
    library.list_papers family. Those reads now probe for the catalog
    first: listings return an empty page, stats count zero papers in
    every status, and a per-paper read reports the intake as unknown.
    Nothing is created on disk by looking.

  • **bookrack init reports a refusal as one line with a hint, a...

Read more

v0.11.0

Choose a tag to compare

@github-actions github-actions released this 08 Aug 21:06

Added

  • A running daemon mounts and unmounts libraries. The set a daemon
    serves used to be decided once, at bring-up: a library registered
    afterwards was invisible to it until a restart, and a root it held
    could only be released by stopping it. bookrack libraries mount <name> opens a registered library and starts serving it; bookrack libraries unmount <name> stops serving one and gives its data root
    back. Neither needs a restart.

    Mounting runs the same checks bring-up runs — a root another mounted
    library already claims, an index profile that disagrees with the
    served set on the reranker stage, an embed backend that cannot serve
    — so a library that starts cleanly also mounts cleanly. It takes a
    registry name, never a path: registering a root stays bookrack libraries add.

    Unmounting refuses three libraries, each because the daemon would
    otherwise go on describing something it no longer serves: the
    registry default, which is where every unqualified command goes; the
    library the daemon came up under, which is what bookrack status
    reports it as; and a library with queue jobs still to run, which
    would fail on their next pull. The root is released once any call
    still using that library finishes, so it may come free a moment
    after the command returns.

    This is what bookrack libraries remove --purge needed: purging a
    root a daemon serves no longer means stopping the daemon, only
    unmounting that one library.

  • A forked library is served without a restart. bookrack libraries fork mounts the clone it just built, so the new library
    answers on the same command. A clone the daemon cannot serve is
    still built and registered — undoing it would delete data to report
    a serving problem — and the command says so, naming the reason and
    the libraries mount that finishes the job. The next steps it
    prints are the operator's now; they used to be written to the
    daemon's own terminal, where the person who typed the command never
    saw them.

  • bookrack libraries default reaches the running daemon. The
    pointer has two homes — the registry file and the cache a daemon
    seeds from it at start — and the command wrote only the file, so the
    listing reported the new default while every unqualified command
    kept reaching the old library. It now hands the change to the daemon
    when one is listening; the daemon writes the same file and follows
    the pointer immediately. With nothing listening it writes the
    registry directly, as before. A library the daemon is not serving is
    mounted rather than refused, so the pointer is no longer confined to
    whatever happened to be mounted.

  • Three verbs read across both pipelines. bookrack list,
    bookrack find, and bookrack search read the book catalog and the
    paper catalog together, so a page covers the library rather than one
    pipeline. --scope narrows any of them to book or paper, and
    defaults to both; the per-pipeline namespaces stay and mean the same
    thing.

    list and find page each side separately — --limit and
    --offset apply per side, and each side reports its own total.
    find takes the filters both catalogs share, plus the ones only one
    of them has: --format on the book side, --year / --venue /
    --doi on the paper side, each requiring a --scope that names its
    own side. A scope reaching the other side is refused before anything
    is sent, naming the catalog that carries the column. search ranks
    both corpora against one query and has no paging; a merged search is
    not recorded in the retrieval sidecar, since its results span two
    corpora that no single fingerprint describes.

    Under the rows is the library they came from. --json is assembled
    by the command rather than forwarded: items carries the rows of
    every side that was read, each naming its own kind and the id that
    addresses it. See Browsing and searching both
    catalogs
    .

    bookrack papers find gains the three shared filters it was missing
    (--contributor-role, --language, --status), so it and --scope paper are the same query.

  • bookrack show <kind>:<id> reads one item, whichever pipeline it
    belongs to.
    Books and papers are two catalogs numbering their
    intakes independently, so 101 names one of each and a bare number
    cannot address either without a namespace to fix it. An item id now
    carries its own kind — book:12, paper:101,
    reference:name_alpha/smith — and the top-level show reads
    whichever the id names. It is the first read verb on the book side of
    the external command line; bookrack papers show stays, and means
    the same thing.

    A card names the library the record came from, so an item read on a
    multi-library daemon is not left ambiguous. An id whose kind has no
    read path yet says so and points at the surface that does carry it,
    rather than at a command that would fail.

    The same <kind>:<id> form is accepted everywhere a bookrack papers command takes an intake id — paper:101 as well as 101 —
    and a book:12 there names the other catalog and is refused while
    arguments are read. It is also what listings print, so an id copied
    out of one command can be pasted into the next. See Naming an
    item
    .

  • --library now travels with the call. The flag named a library
    and then stopped at the CLI: every control-plane method that takes a
    library parameter was reached without one, so the daemon resolved
    the registry default and the flag survived only as an assertion that
    the running daemon served the same library. It is now put on the
    wire — under library for every routed method, and under name for
    library.info, whose own parameter predates that spelling.

    A method that reports on the daemon rather than on a library —
    status, doctor.gather, libraries list — cannot honour a
    selection, and naming one on such a command is now refused with
    the reason and what to run instead, rather than silently answering a
    question about a different library. Methods that describe the
    process (diagnose, the queue verbs, logs) are unaffected: a
    selection there is meaningless, not wrong.

    A library the caller wrote out by hand always wins, so bookrack rpc call stays an escape hatch.

  • bookrack libraries list marks which libraries the daemon
    serves.
    The listing reads the registry file, so it shows every
    registered library whether or not a daemon is running — and gave no
    way to tell those two sets apart. A served column now marks the
    entries a running daemon holds; the sets differ whenever the daemon
    came up on a root the registry does not name, because such a daemon
    serves that root alone. With no daemon answering, the column is
    absent rather than a column of noes: nobody was asked, which is a
    different answer from nobody serving them.

  • daemon.status reports every library the daemon serves. A new
    served array carries one row per mounted library — name,
    data_dir, default, primary — so a client can see the whole
    served set, which library an unnamed call resolves to, and which one
    the daemon came up under. Those last two are separate bits: a daemon
    started under a library that is not the registry's default carries
    them on different rows. The existing library and data_dir fields
    keep their meaning, the primary's identity.

  • daemon.methods reports how each method takes a library. Every
    row gains selection (routed / process / unrouted) and
    library_key, so bookrack rpc list — and any other client — reads
    which methods act on one library, and under which parameter, off the
    daemon instead of keeping a list of its own.

  • Every paper curation edit now leaves an audit row. papers metadata set / clear / void / ack / approve / reject /
    reopen / contributor-add / contributor-remove each append one
    metadata_audit row to the paper catalog, carrying who made the
    edit, the field, the value it replaced, and the reason when one was
    given. Until now the paper side wrote the change and nothing else,
    so a corrected title left no record that anyone had corrected it,
    and the two questions the book side has always answered — who
    edited this, and what did it say before — had no answer on a paper.

    The edit is attributed to the surface that made it, so an agent
    editing over MCP and a curator editing at the CLI are told apart in
    the trail rather than both recorded as human. Each call is also
    logged in mcp_tool_calls under its method name, the way paper
    reads already were.

  • The paper metadata audit is readable per field.
    library.show_paper_metadata_report recomputes the plausibility
    audit from a paper's cached extraction against its current effective
    metadata and returns the whole judgement: per-field origin, grade,
    flags and hint, the cross-field flags, and the CSL type the
    required-field matrix was chosen by. The book side has had this read
    since the audit existed; on the paper side the only way to see why a
    record was graded needs_work was to run a re-audit and watch the
    two-scalar rollup move.

    Each response also carries the judgement stored on the paper's audit
    row. The two disagreeing is the answer to a question nothing could
    answer before — whether the paper has been edited since it was last
    judged — and papers metadata reaudit is what closes the gap.

  • The paper review queue is browsable.
    library.list_paper_metadata lists papers with their confidence and
    review status, and library.list_paper_pending_reviews is the preset
    for the ones still needing a decision. The paper side had the four
    review verbs but no way to ask which papers they applied to, so a
    record entering the queue at ingest had no route out that...

Read more

v0.10.0

Choose a tag to compare

@github-actions github-actions released this 26 Jul 21:11

Added

  • retrieval: show flags a recall set that is entirely weak.
    bookrack retrieval show <call-id> now closes its hit table with one
    advisory line when every recorded hit of the call sits at or above the
    weak-match distance threshold, naming the threshold that made the
    verdict so the line can be checked against the library's
    configuration. The verdict is over the whole set — a single strong hit
    vetoes it, and a call with no hits draws no line, because an absence is
    not a weak result. This is the first consumer of
    search.weak_threshold / BOOKRACK_SEARCH_WEAK_THRESHOLD: the knob
    resolved through the full precedence chain but nothing read the
    resolved value. The threshold comes from the library retrieval show
    was pointed at, under the usual environment override. Nothing grades
    an individual hit, and the search responses are unchanged.

  • search: recall can exclude named books and papers. library.search
    accepts exclude_book_intake_ids and exclude_paper_intake_ids on
    both the MCP and control-plane surfaces, dropping the named items'
    passages server-side instead of leaving the caller to filter a result
    set it has already paid for. The exclusion is a predicate on the ANN
    query, so a caller that has read a dominant book can push it out of
    the way and see what the rest of the library says. The two id name
    spaces are unrelated, so each list applies only to its own side and
    naming the side the call does not search is rejected rather than
    ignored; an omitted or empty list filters nothing. kind="all"
    carries both lists at once and applies each to its own store.
    library.search_in_book and library.search_in_paper take no
    exclusions — recall there is already confined to one item. The
    recorder logs a non-empty exclusion set alongside the ANN knobs.

  • reference: wildcard lookup can exclude books. reference.lookup
    accepts exclude_books, a list of book slugs dropped from the
    result when book="*" spans every registered reference book. The
    exclusion holds across the whole resolution chain, including the
    latin-key fallback retry and the redirect hop, so an excluded book
    cannot re-enter as a redirect target. Passing the list with a
    single-book scope is an error rather than a silent no-op.

  • runtime: the daemon mounts every registered library. With the
    primary root selected through the registry (--library or the
    registry default), bring-up now warms a handle for each registry
    entry: every library answers reads, queue jobs route to their
    target library by name, and each served root's lock is held for the
    daemon's lifetime (so libraries remove --purge on any registered
    root needs bookrack quit first). Two registry entries pointing at
    the same root refuse startup naming both. A path-class root the
    registry does not know keeps the previous single-library form.
    Runtime mount/unmount and per-library status surfaces are later
    milestones.

  • cli: bookrack status, a one-screen status card. One
    no-argument command answers "is a daemon running, which library does
    it serve, is it busy": with a live daemon it renders a single card
    with daemon.* / library.* / queue.* sections; with no daemon it
    prints a short card and exits 0 ("not running" is an answer, not an
    error); a held session lock whose control plane stays silent for 2s
    exits 3 with the same stale-lock contract as bookrack run. --json
    emits the card as one JSON object in every state; --quiet answers
    through the exit code alone. The identity rows come from the daemon
    over RPC, not from the lock file. Embedder and reranker health stay
    with bookrack doctor; the card links to it instead of paying the
    network round-trips.

  • runtime: the status RPC reports the served library. The
    response gains library (registry name, null for a data root
    selected directly by path) and data_dir (the served root), so any
    control-plane client can name what the daemon serves without reading
    the session lock. Both are single-library snapshot fields; a
    multi-library daemon will report them in plural form.

  • config: registry writes are serialized by a sibling lock file.
    Every registry write verb — set-default, upsert, repoint, remove —
    now takes a short-held exclusive lock on <registry>.lock around its
    read-modify-write window before replacing the file atomically. The
    daemon became a registry writer when library.fork and now
    library.set_default persist to it, so the single-writer assumption
    behind bare atomic replacement no longer holds; the lock closes the
    window where two writers' read-modify-write cycles would silently
    swallow each other. Readers take no lock — the atomic rename still
    hands them the old file or the new one. The lock file carries no
    content and can be deleted at any time.

Changed

  • config: mcp_addr and log_directive are retired config.toml
    keys.
    Neither ever resolved. Both are process-level knobs, read
    before any data root is known — the log subscriber is installed and
    the MCP listen address is chosen by the binary that then goes looking
    for a library — so a value written into a library's config.toml was
    accepted, stored, and never consulted: the write succeeded, a daemon
    restart changed nothing, and no surface said so. The two keys now join
    embed_model and index_profile as retired: a file carrying either is
    refused by name on every command that resolves a data root, and the
    error gives the knob's real home (BOOKRACK_MCP_ADDR or bookrack run --mcp-addr; BOOKRACK_LOG and BOOKRACK_LOG_CONSOLE) alongside the
    libraries config <name> --unset <key> that deletes the line. Both
    environment variables are unchanged. libraries config <name> also
    annotates a retired line when it prints a file verbatim — the one read
    surface such a line used to pass through unremarked. Upgrading a
    library that still carries either line requires the unset; see
    docs/UPGRADE.md.

  • cli: a caller without a terminal is asked to confirm, not
    refused.
    vectors drop, vectors reset, their two papers
    peers, index-profile apply, and the libraries add uuid-clash
    resolution refused outright when stdin was not a TTY. That predicate
    answers "is a human at a terminal", not "can an answer arrive", and
    the two differ often enough to matter: this repository's own
    end-to-end tests drive the strongest gate in the product — retyping
    a library name before libraries remove --purge — over a pipe, and
    ssh host … without -t, docker exec without -t, and Git Bash
    on Windows all put a human behind one. Refusing them pushed
    operators to --yes, which skips the retype entirely. All three now
    prompt like every other destructive verb, under the answer window
    BOOKRACK_CONFIRM_TIMEOUT_SECS bounds (see Fixed). A caller that
    supplies no answer still fails with the same
    directed message and the same exit 2; what changed is that a
    caller that does supply one is now heard. DestructiveGate is gone:
    its stdin axis moved to render::confirm's answer_window, and
    keeping it would have been a second copy of one rule.

  • extract: a fullwidth References heading terminates the paper
    metadata-scan window.
    extract_paper_metadata_text documented its
    References-heading match as running against the fullwidth-folded
    page text, but the fold was applied only after the window was cut,
    so a heading printed in fullwidth forms (U+FF01..U+FF5E) never
    closed the window and bibliography text leaked into the DOI / venue
    scan. Each page is now folded before the heading match, making the
    documented order the real one.

  • glean, catalog: three paper-audit grade columns no longer stuck
    at missing.
    The audit's per-field reports and the
    node_paper_audit projection disagreed on field-key names:
    GRADE_COLUMNS looked up arxiv and container where the graders
    record arxiv_id and container_title, so grade_arxiv and
    grade_container wrote missing regardless of the paper; and the
    abstract grader read the nonexistent effective key abstract where
    the effective view names it abstract_text, so grade_abstract
    always graded missing and the abstract_too_short signal could
    never fire. The keys are now unified on the effective-attrs names
    end to end (graders, roll-up candidates, CSL requirement matrix,
    projection). Audit rows written before the fix carry the stuck
    missing grades until their papers are re-audited
    (papers.metadata.reaudit).

  • glean: a thesis with its institution can audit clean. The CSL
    requirement matrix marks publisher (the institution, in CSL
    terms) as required for a thesis, but no grader produced that field,
    so the roll-up saw it as missing on every thesis and floored the
    verdict to needs_work with low confidence regardless of the
    actual metadata. A presence-only publisher grader now feeds the
    roll-up; the grade travels in the report JSON and has no dedicated
    grade_* column on node_paper_audit. Thesis rows audited before
    the fix stay needs_work until re-audited
    (papers.metadata.reaudit).

  • distill: the splits coverage counter is actually written. The
    Coverage doc promised its counters were written by the matching
    stages, but neither splitter stage touched splits, so every
    book_distill_audit row stored splits = 0 regardless of the run.
    Both splitters now record their output count; audit rows written
    before the fix keep the zero until their book is re-distilled.

  • glean: venue_aliases is actually consulted. The paper-audit
    data set documented venue_aliases as a map of informal-to-canonical
    venue spellings, but no grader read it: grade_container checked
    the whitelist against the raw value only, so an aliased spelling of
    a whitelisted venue was still fla...

Read more

v0.9.0

Choose a tag to compare

@github-actions github-actions released this 18 Jul 09:57

Added

  • config, cli: the library manifest carries the index-profile
    reference; the registry entry becomes a regenerable cache of it.

    bookrack-library.toml gains an optional index_profile key, and
    declaring a profile — through index-profile apply or libraries config <name> index_profile=<p> — writes it there in one atomic
    write, then refreshes the registry cache and sweeps a superseded
    config.toml declaration. The reference now travels with the data:
    copy a root to another machine, libraries register it, and its
    profile comes along; libraries scan refills the cache for every root
    it finds. Resolution reads manifest, then config.toml, then the
    registry entry, so references declared the old way keep working
    untouched — see docs/UPGRADE.md for the one-command
    migration. format_version stays at 1: the key is optional and
    additive, and unknown keys were always tolerated.

  • session, runtime: data roots are guarded by an exclusive
    .bookrack.lock.
    The daemon takes the new root lock during
    bring-up — after resolving its configuration, before opening anything
    under the root — and holds it until shutdown, so a second daemon
    pointed at the same data root fails to start instead of racing the
    first one through SQLite, LanceDB, and the queue snapshot. This
    closes the gap that two different BOOKRACK_RUNTIME_DIR values left
    open: the session lock only ever bound one daemon per runtime
    directory, never one writer per data root. Offline read-only commands
    (distill, runs, retrieval, doctor diagnostics) do not take the
    lock and keep working alongside a serving daemon; a read-only data
    root, which cannot host a lock file at all, is served unlocked with a
    warning.

  • runtime: working daemon state, and the lifecycle state is now
    derived instead of assigned.
    daemon.state / status.state gain a
    fifth value, working: the queue worker is executing a job, without
    the write-mutex and MCP-pause semantics that writing implies.
    Internally the single state flag is folded from independent activity
    sources (RPC write session, executing queue jobs, degraded
    conditions, shutdown) by precedence stopping > writing > working > degraded > idle, so overlapping activities can no longer clobber
    each other's transitions — an RPC write finishing while a queue job
    still runs falls back to working, not idle.

  • mcp, runtime: library.show_toc / library.show_paper_toc are
    paginated.
    Both TOC reads accept optional offset / limit
    parameters (the limit defaults to and is clamped by the 2000-entry
    cap) and return total and a next_offset cursor alongside the
    existing truncated flag. A TOC larger than one page is now
    reachable page by page — previously everything past the cap was
    silently unreachable. Requests without the new parameters return the
    same entries as before.

  • mcp, runtime: TOC reads gain titles_only and max_depth
    projections.
    With titles_only, each TOC entry shrinks to
    node_id / title / depth — enough to scan a book's structure
    and address a follow-up span read at a fraction of the response
    size. max_depth keeps only the shallow levels (the book root is
    depth 0, so max_depth: 1 is the root plus its top-level
    divisions). Both compose with pagination, and total counts the
    filtered set.

  • mcp, runtime: library.show_book / library.show_paper carry
    toc_stats.
    The detail responses gain an aggregate of the
    ingested TOC — entry_count (the book root included) and
    max_depth — or null when the item has no corpus nodes, so a
    caller can pick a show_toc pagination or projection strategy
    before pulling any entries. The show_metadata_audit response
    embeds the same detail shape and carries the field too.

  • mcp, runtime: TOC reads gain a title_substring filter.
    library.show_toc / library.show_paper_toc can keep only entries
    whose title contains a substring (ASCII case is ignored; LIKE
    metacharacters match literally), so locating one chapter by name is
    a single filtered call instead of a full TOC pull. Composes with
    pagination and the projections; total counts the filtered set.

  • config, cli: config.toml gains index_profile and a [search]
    table.
    A library's per-root config can now name the index profile it
    runs under and pin the two retrieval knobs (search.top_k,
    search.weak_threshold) that were previously env-only. bookrack libraries config accepts the three new keys — dotted keys edit the
    nested [search] table, numeric values are written as TOML numbers,
    and an index_profile reference is refused at write time when it does
    not resolve or fails static validation. The resolution chains are
    pinned: BOOKRACK_EMBED_MODEL > explicit embed_model > the
    profile's model > default, and BOOKRACK_SEARCH_* > [search] >
    default.

  • index-profile, runtime: index-profile current and index-profile diff. current [--library <name>] reports, offline and read-only,
    the profile a library effectively runs under — the reference's origin
    (manifest, config_toml, or registry), any lower-priority source
    still naming a different profile (drift), the resolved
    embed/ann/reranker combination, the effective embed model, and a
    field-by-field comparison against the built index stamps. diff <a> <b> compares two profiles field by field. Both take --json.

  • index-profile, runtime, cli: index-profile apply reconciles a
    library with a profile.
    The high-level path for switching embedding
    models or ANN parameters: statically validate the target profile,
    refuse when BOOKRACK_EMBED_MODEL or an explicit embed_model field
    would mask it, compare the recorded stamps and the persisted ANN
    configuration per pipeline (books and papers derive independently;
    an unused pipeline is skipped), and derive the action list — a
    destructive reset (followed by the rebuild that realizes the
    declared index) when the embed model or dimension changed, a
    reembed for a chunking/normalization version bump, a rebuild or
    drop that realizes the declared ANN state, a stamps reconcile
    for missing stamp records. Applying to an already-consistent library
    still writes the profile reference.
    --dry-run prints the plan offline; executing routes every action
    through the daemon queue in order, one result line per step. A plan
    containing a reset demands the library name retyped (or --yes), and
    the recommended rehearsal is libraries fork → apply on the copy →
    verify → apply in place. The target declaration is written to the
    library manifest after confirmation and before the first action, so an
    interrupted run reports "declaration ahead of stamps" in current and
    a re-run re-derives only the remaining actions.

  • config, runtime, cli: bookrack doctor --install-reranker installs
    the pinned reranker artifacts.
    The reranker backend's two native
    artifacts — the llama-server binary (llama.cpp release b9934,
    unpacked whole so the executable keeps its dynamic libraries beside
    it) and the Qwen3-Reranker-0.6B GGUF model — are now pinned by
    SHA-256 in the config crate (LLAMA_SERVER_VERSION.md documents the
    pair) and installed into per-user managed directories by the new
    doctor verb, which downloads whichever is missing and is a no-op
    when both are in place. Two new escape hatches,
    BOOKRACK_LLAMA_SERVER_BIN and BOOKRACK_RERANKER_MODEL, name the
    artifacts directly and are authoritative when set.

  • rerank, runtime, config: the reranker backend — a supervised
    llama-server subprocess with an operator-URL escape hatch.
    A new
    rerank crate wraps llama-server's /v1/rerank endpoint behind the
    same typed error contract as the embed client. When a library's
    effective profile enables a reranker stage, the daemon now brings up
    a verified backend before serving: by default it spawns the pinned
    llama-server on a loopback port, holds startup until /health
    reports the model loaded, restarts it with backoff if it crashes,
    stops it TERM-then-KILL at shutdown, and kills a recorded orphan
    left by a killed daemon. Setting reranker.url in config.toml (or
    BOOKRACK_RERANKER_URL) switches to an operator-run server, probed
    once at startup instead of spawned. The [reranker] table also
    carries ctx and threads for the supervised server, all three
    keys writable through bookrack libraries config. bookrack doctor
    grows a reranker section (binary, model, backend status) that
    appears only when the effective profile enables the stage.

  • index-profile, ops, runtime: the reranker stage is live —
    qwen3-4b-quality becomes a runnable profile.
    A profile whose
    [reranker] section names a cross-encoder no longer refuses
    startup: the daemon brings up the verified backend, and every search
    op then recalls the profile's top_k_in candidates, scores them
    through the reranker, and returns at most top_k_out passages
    ordered by relevance, each citation carrying a rerank_score
    alongside its ANN distance (the field is absent for reranker-less
    libraries, so their wire shape is unchanged). The unified
    book+paper search reranks the merged candidate window, so the stage
    scores one profile-sized set regardless of how candidates split
    across stores. A stage failure fails the search — the profile
    promises an atomic retrieval combination, and under the supervised
    backend an outage is a self-healing restart window — rather than
    silently returning the unreranked order. The built-in
    qwen3-4b-quality profile is finalized to the implemented backend
    (llama-server, candidate window 50 -> 10), the validator now
    rejects any other reranker backend, and index-profile current/apply drop their not-implemented notes; apply hints at
    the daemon restart that...

Read more

v0.8.0

Choose a tag to compare

@github-actions github-actions released this 08 Jul 13:01

Added

  • index-profile, runtime, cli: bookrack index-profile names and
    statically checks a retrieval configuration.
    A new index-profile
    crate couples the three retrieval knobs — the embedding model, the ANN
    index shape, and the reranker stage — into one named, validatable atom,
    with two built-in presets compiled into the binary
    (qwen3-0.6b-default, a product-quantized IVF index with no reranker;
    qwen3-4b-quality, an HNSW index with a cross-encoder reranker stage).
    Three read-only verbs resolve locally with no daemon: index-profile list merges the built-ins with any user profiles under the per-user
    index-profiles/ directory (marking each [builtin]/[user] and
    noting shadows), index-profile show <name> prints a profile's source
    and its validation result, and index-profile validate <name> runs the
    static checks and exits non-zero on any error. Validation enforces the
    product-quantization constraints (num_sub_vectors required, dividing
    the dimension, with dim / num_sub_vectors <= 8), warns on the
    HNSW recall regression, checks the cross-encoder reranker contract, and
    consults an offline model registry that --allow-unknown-model
    bypasses. A user profile shadows a built-in of the same name.

  • runtime: bookrack doctor checks registry–manifest consistency and
    index-profile coherence.
    Two new row groups. The first walks every
    registry entry and compares it against its on-disk identity manifest,
    warning when the cached uuid is stale, when the data root is missing (a
    move or an unmounted volume), or when the manifest is unreadable — a
    name alias and a kind override stay legal and are not flagged. The
    second, for every entry that records an index_profile, resolves and
    statically validates the profile and compares its embed model and
    dimension against the library's built index stamps, warning when a
    referenced profile is undefined, invalid, or disagrees with the built
    index (which would make the daemon refuse to start). Both run on the
    daemon-side and daemon-free doctor paths alike; the coherence check
    opens the corpus read-only and skips silently when it cannot.

  • config, cli: offline libraries config reads and edits a library's
    config.toml.
    libraries config <name> resolves the library's data
    root from the registry with no daemon and prints its config.toml
    (the parsed knobs under --json, the raw file otherwise);
    libraries config <name> KEY=VALUE ... and --unset KEY edit the
    file in place, preserving hand-written comments and layout. Accepted
    keys are the four RootConfig knobs — ollama_url, embed_model,
    mcp_addr, log_directive — and an unknown key or a malformed
    ollama_url / mcp_addr value is refused (exit 2) before anything is
    written. A missing config.toml is created on first write. Editing
    embed_model warns that re-ingestion is required; a set value already
    shadowed by an environment variable, and the need to restart a running
    daemon, are noted. config exposes set_root_config_values,
    read_root_config_text, ROOT_CONFIG_KEYS, and
    root_config_env_override.

  • config, ops, runtime: a path-resolved root is identified against the
    registry.
    When the data root is won by a path — the --data-dir
    flag, BOOKRACK_DATA_DIR, or the portable layout — config now
    matches it back against the registry, by the root's manifest uuid
    first and then by path, and claims the registered library name instead
    of leaving the root anonymous. bookrack info reports how the name was
    identified (manifest uuid or path), and bookrack doctor appends
    the same fact to its data root row; a registry selection reports
    nothing extra, since the source already conveys it. The LibraryInfo
    DTO gains an optional library_identification field, and config
    exposes the LibraryIdentification enum and a find_library_by_uuid
    helper. The precedence itself is unchanged; a path-class daemon that
    previously reported the placeholder name default now reports the
    registered name.

  • config, cli, runtime: a shadowed registry default is now visible.
    When a path-class source wins the data root — the --data-dir flag,
    BOOKRACK_DATA_DIR, or the portable exe-neighbour layout — a registry
    default the operator set still applies on disk but has no effect on
    which root is served. config now computes that eclipse once and
    surfaces it three ways: bookrack info adds a shadowed_default line
    (registry default '<name>' is shadowed by <source>), bookrack doctor turns the data root row from OK to WARN with the same fact
    and the remedy (unset the variable or pass --library <name>), and
    libraries default <name> warns on stderr when BOOKRACK_DATA_DIR is
    set to another root and will shadow the default just written. The
    precedence itself is unchanged; only the diagnostics are new. The
    LibraryInfo DTO gains an optional shadowed_default field.

  • config, cli: offline libraries add, register, and remove, plus
    scan --register.
    Three write verbs bring an existing data root into
    the registry with no daemon. libraries add <name> <path> registers
    under an explicit name; libraries register <path> derives the name
    from the root's manifest (or its directory), taking --name <alias>
    when the derived name is taken. Both write a fresh identity manifest to
    a root that has none — previewed and confirmed first, unless --yes —
    and degrade to a uuid-less entry rather than failing on a read-only
    volume. A registration whose identity uuid already belongs to another
    entry stops for a decision: interactively, move the existing entry to
    the new path; non-interactively, exit with the exact move/--new-uuid
    commands. libraries remove <name> forgets an entry and keeps the
    data; --purge additionally deletes the data root behind a detect gate
    and a typed confirmation. libraries scan --register registers every
    confirmed root it finds — turning scan --volumes --register into a
    one-command registry rebuild after a reinstall — and skips probable
    roots with a note to add them by hand.

  • config, cli: offline libraries detect and libraries scan. Two
    read-only, daemon-free subcommands answer "is this path a bookrack
    data root?". libraries detect <path> returns a three-way verdict —
    confirmed (a readable identity manifest), probable (no manifest
    but the catalog.db + corpus.db pair is present), or not a
    library
    — reporting an unreadable manifest (foreign magic or a future
    schema version) as its own distinct outcome; it exits 0 for
    confirmed/probable, 1 otherwise, and 2 for a missing or non-directory
    path. libraries scan <parent> walks a parent directory's immediate
    children (or, with --volumes, mounted volumes two levels deep),
    lists the data roots found, prunes at each hit, and always reports how
    many entries it skipped. Both honour --json.

  • config, runtime: data roots carry a self-describing library
    manifest.
    Every data root now holds a bookrack-library.toml
    naming the library and its stable UUIDv7 identity (format,
    format_version, uuid, name, kind, description,
    created_at), so a library describes itself and the registry becomes
    a regenerable cache over the manifests rather than the sole record of
    identity. bookrack init writes one for the new root and caches the
    uuid into the registry entry; libraries fork writes the fork its own
    manifest with a freshly generated uuid. A root without a manifest
    stays valid, and a v1 reader tolerates unknown keys within v1.x.

  • config, cli, runtime: the library registry carries per-library
    metadata and libraries default persists.
    Registry entries may
    now be metadata-bearing tables ([libraries.<name>] with data_dir,
    kind, description, index_profile, created_at, uuid)
    alongside the legacy bare-path form, which stays permanently
    readable and is treated as { data_dir = "<path>", kind = "prod" }.
    Any write rewrites the file into the table form atomically
    (same-directory temp file plus rename) and prints
    info: registry upgraded to entry-table format on stderr the first
    time it upgrades a legacy file. bookrack libraries default <name>
    now resolves locally and writes the on-disk default pointer
    directly, so it needs no running daemon and the choice survives a
    restart; naming a library the registry does not define exits 2. The
    library.set_default RPC remains but is documented as affecting only
    the running daemon session. libraries list gains a kind column
    (and kind field under --json).

  • corpus, catalog, ops, cli: every single-store search invocation is
    recorded against the corpus state that served it.
    A new
    Corpus::compose_corpus_fingerprint hashes the five corpus stamps
    (embed model, vector dimension, chunk version, normalize version,
    and the ANN kind from vectors_meta.json, brute-force when no
    meta file exists) into a 16-hex prefix. M[15] adds the sidecar pair:
    retrieval_calls, one row per library.search /
    library.search_in_book / library.search_paper /
    library.search_in_paper call carrying that fingerprint, the
    requested depth, and the query, and retrieval_call_hits, one row
    per returned passage keyed (call_id, ord) with its
    norm_chunk_sha256 and distance. The recorder writes the log row
    and the sidecar in one transaction
    (Catalog::record_tool_call_with_retrieval) and stamps the
    fingerprint into mcp_tool_calls.extras; the unified two-store
    search records no sidecar because one row cannot carry two
    fingerprints. Catalog::prune_retrieval_calls_with_stale_fingerprint
    / prune_retrieval_calls_older_than trim the tables (no CLI entry
    yet); bookrack retrieval list [--last N] [--corpus-fingerprint HEX]
    and bookrack retrieval show <call-id> r...

Read more

v0.7.0

Choose a tag to compare

@github-actions github-actions released this 26 Jun 11:34

Added

  • Reference-book pipeline (preview). A new bookrack-distill
    stage runner and bookrack-refs read store land the third
    bookrack item kind alongside books and papers: a Reference
    variant on core::ItemKind, a reference.db per library with the
    baseline schema (reference_books, reference_entries,
    reference_entry_overlays, a reference_entries_resolved view
    that merges payloads through json_patch, and an FTS5 trigram
    sidecar), the staged transform Source → Pages → Blocks → Raws → Splits → Drafts built from segment / splitter / extractor /
    finalize stages, and the three controlled vocabularies (property,
    quality-flag, stage catalogs) with a cross-catalog validator. The
    surface lands as three bookrack distill {build,verify,list} CLI
    subcommands and two MCP tools reference.lookup and
    reference.overlay_set.

    This is a preview. The feature is not finished:

    • OCR Markdown is the only supported input. The distill
      pipeline reads the polyocr Markdown product of one reference
      book; born-digital sources are not yet wired in.
    • Operators must author a book.toml per book. The pipeline
      is config-driven: anchor rules, stage chain, property writes,
      and lookup indexes are all declared in a per-book TOML file.
      There is no auto-detection. The expected schema is documented
      by the controlled vocabularies shipped under
      crates/distill/data/; mis-shaped configs surface as
      ParseError::CatalogViolation at load time.
    • No queue integration. A Reference job submitted to the
      queue worker today panics the spawn-blocking task with
      unreachable!; the CLI runs distill locally in-process
      instead.

    Treat the surface as unstable until a follow-up release lifts the
    preview marker.

  • CLI shared render layer behind three new top-level flags:
    --json switches every reading subcommand from the default
    human renderer (tables, key-value cards, one-line summaries) to
    the daemon's structured payload; --quiet suppresses progress
    spinners and confirmation chrome for non-interactive use; and
    --no-color disables ANSI styling even when stdout is a tty. The
    flags resolve once into a process-wide RenderCtx so individual
    subcommands stay free of mode plumbing. Reading subcommands that
    previously dumped JSON only now branch on the context; writing
    subcommands respect --quiet on their progress and confirmation
    surfaces.

  • The session lock records the data-root path the daemon is
    serving as a new library_root=<abs path> line. The line is
    append-only (Phase 1's "unknown keys ignored" rule still
    applies), so an older client reads the file unchanged. The CLI
    warns on stderr when the shell's intended library selection
    (--data-dir / --library / BOOKRACK_DATA_DIR) disagrees with
    the running daemon's recorded root, so an operator does not
    silently send writes against the wrong library.

  • CLI subcommand help is filled out across the surface: every
    bookrack <subcommand> carries a one-line summary and a long
    help body, and the global flags (--data-dir, --library,
    --audit-profile) move under a shared Common Options group
    so they no longer scroll past the subcommand's own arguments.
    Argument-level help is filled in on ingest, remove,
    vectors, and the papers action arms.

  • intake.ocr control-plane method and the matching bookrack intake ocr <ocr_md> --from-pdf <pdf> top-level CLI subcommand. The handler
    enqueues a single OCR-intake job onto the persistent ingest queue;
    the worker dispatches it as a book ingest whose source is the OCR
    markdown product paired with the scan PDF anchor, by way of a new
    LibraryHandle::ingest_ocr runner.

  • bookrack queue top-level CLI subcommand covering the existing
    control-plane methods that previously had no one-shot entry point:
    queue list (→ queue.list), queue pause / queue resume /
    queue clear (→ the matching queue.* RPC), and queue cancel <job-id-prefix> (→ ingest.cancel). The shared QueueAction
    grammar gains List and Cancel variants.

  • bookrack libraries info [--name <NAME>] and bookrack libraries default <NAME> round out the libraries subcommand: info prints
    the per-library status card the daemon serves via library.info,
    default moves the registry's default-library pointer through
    library.set_default. Both RPCs were already on the control plane;
    this lifts them out of the bookrack exec library.* catch-all into
    first-class top-level commands.

  • bookrack logs is now a top-level subcommand for the daemon's
    log stream. --tail N snapshots the most recent N events via
    the logs.tail RPC and exits; --follow (the default with no
    other flag) subscribes to the broadcast and streams new events
    as they arrive; both together emit the snapshot first and then
    switch to the live stream. --level TRACE|DEBUG|INFO|WARN|ERROR
    (case-insensitive) drops every event below the given severity.
    Human mode renders each event as HH:MM:SS LEVEL target | message;
    the top-level --json flag emits newline-delimited LogEvent
    records so jq -c pipelines stay clean. The existing
    bookrack exec logs follow|tail [<n>] keeps working as an
    unsurfaced compatibility alias.

Removed

  • bookrack repl standalone control-socket REPL client and the
    in-process reedline REPL hosted by bookrack run (including its
    --legacy-repl transition flag). Operators reach the daemon
    through one-shot subcommands, bookrack exec <method> '<json>'
    for ad-hoc control-plane RPCs, the desktop tray, or MCP. The
    cmd::repl_client module, the tests/repl_e2e.rs integration
    test, the reedline and shlex workspace dependencies, and the
    ReplCli / ReplCommand wrapper types from the former
    bookrack-repl-grammar are deleted; the surviving crate is
    renamed to bookrack-cli-grammar and keeps its leaf
    clap::Subcommand types (IngestArgs, WriteMetadataAction,
    WriteVectorsAction, CorpusAction, StampsAction,
    QueueAction, RemoveArgs, DryrunArgs, IntakeAction,
    Papers*) for the top-level CLI to consume.

Changed (breaking)

  • corpus.rebuild, vectors.reembed, remove, and their
    papers.* peers now require plan_id on the execute leg. The
    transitional unpinned fallback that ran the legacy path with a
    deprecation warning is removed; a missing plan_id returns
    -32602 INVALID_PARAMS pointing at the dry-run leg. All
    in-tree clients (one-shot CLI subcommands, the tray) already
    drive the two-step pinned protocol.

Changed

  • Every predictable CLI failure now flows through a typed
    bookrack_cli::error::BookrackCliError enum and the same
    color-eyre reporter, ending the old split between
    eprintln!+std::process::exit(2) for daemon-not-running, an
    ad-hoc lock-conflict path in bookrack run, and doctor's
    hand-rolled std::process::exit(1). Variants
    (DaemonNotRunning, DaemonUnreachable, StaleSessionLock,
    SessionLockUnreadable, DoctorUnhealthy) each carry their
    own exit code and one-line message; unexpected errors still
    fall through to the full color-eyre cause chain so internal
    bugs stay debuggable. helpers::connect_or_exit is renamed to
    helpers::connect and returns Result<Arc<ControlClient>>;
    every call site uses ? for propagation. The
    oneshot_subcommands_consistent_no_daemon integration test
    continues to expect exit 2 and a bookrack daemon not running
    hint on stderr.
  • Destructive-action confirmation runs through a single
    bookrack_cli::render::confirm::confirm_destructive helper with
    two strengths: Soft accepts a case-insensitive yes / y
    (used by corpus rebuild, remove, vectors reembed, the
    papers.* peers, and libraries fork); Hard { token }
    requires the operator to retype the literal sentinel (used by
    vectors reset for the RESET token). Prompts move from stdout
    to stderr so a piped stdout stays clean, and libraries fork
    now prints Forked library to '<name>' at <path> (<mode>). on
    success instead of dumping the daemon's JSON response. The
    legacy crate::util::confirm / confirm_token /
    confirm_vectors_reset are removed.
  • bookrack queue list, bookrack libraries list, bookrack libraries info, bookrack papers list, bookrack papers find, and
    bookrack papers show render a human table or key-value card by
    default instead of the daemon's raw JSON. The top-level --json
    flag restores the legacy pretty-printed payload for scripts.
    queue list shows an 8-character UUID prefix in the id column;
    queue list --long prints the full UUIDv7 for copy-paste.
    papers list / papers find collapse to columns id, title,
    author, year, container with … truncation, and append a
    (showing N of TOTAL; ...) hint when the result set is truncated.
  • bookrack ingest <path>, bookrack papers ingest <path>, and
    bookrack intake ocr <ocr_md> --from-pdf <pdf> wait for the
    enqueued job(s) to reach a terminal state by default and print a
    one-line human summary on stdout (Ingested <basename> as <id8> in 12.4s (done), OCR-ingested <basename> ...) instead of
    exiting at queue-ack with a JSON job-id list. The historical
    immediate-return behaviour is available under --no-wait, and the
    top-level --json flag switches the wait-summary to a
    JobOutcomeReport payload for scripts. The CLI subscribes to the
    event stream before submitting the RPC so a fast worker cannot
    race past the wait loop's first recv.
  • The persistent queue document schema bumps to v4: QueueJob grows
    an intake_ocr sidecar that carries the from_pdf, expected_pages,
    and allow_partial fields the OCR ingest path needs. The field is
    optional and defaults to absent, so a v3 document loads unchanged
    and reads as a plain book ingest. The upgrade is one-way — ...
Read more

v0.6.1

Choose a tag to compare

@github-actions github-actions released this 16 Jun 14:27

Fixed

  • bookrack-runtime raises its crate-level recursion_limit to
    256 so the trait solver can prove Send on the
    methods!-generated dispatch future under tokio::spawn. The
    macro collapses 30+ control-plane handlers into one async match;
    combined with the layered portable_atomic shim pulled in
    transitively through lance on targets without native 128-bit
    atomics (Linux x86_64 without cmpxchg16b), the default limit of
    128 was just past the depth required to walk every handler's
    future and every wrapper type. Targets with native 128-bit atomics
    (macOS aarch64, macOS x86_64 with the feature) stayed under the
    default limit, which is why local builds passed while CI's Linux
    job failed with E0275. Zero runtime cost; the knob only widens
    the compile-time search budget.

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 14 Jun 03:11

Added

  • Two-store library: a papers cluster (papers_catalog.db,
    papers_corpus.db, lancedb_papers, papers_dir/) sits beside the
    existing book cluster under the same data root, with parallel
    intake / corpus / vector / metadata schemas. The config crate
    exposes paper-side getters; the in-process LibraryRegistry owns
    both stores and routes work by item kind, so books and papers share
    one scheduler, one MCP surface, and one search call.

  • bookrack-glean: paper-side pipeline crate covering
    REGISTER → IDENTIFY → EXTRACT → STRUCTURE → CHUNK → EMBED. Driven
    through the control plane by glean.submit, which the queue worker
    consumes as paper-kinded jobs. REGISTER archives the source PDF
    bytes next to the envelope and stamps intake.source_pdf_path;
    IDENTIFY runs an offline pass that lifts DOI, arXiv id, venue, and
    abstract from the cached extraction; EXTRACT reuses the shared PDF
    adapter; STRUCTURE runs the paper-only coloring + Section tree
    pass; CHUNK and EMBED reuse the book-side primitives against the
    papers vector store.

  • bookrack papers subcommand tree (one-shot CLI and REPL): papers ingest [--recursive] walks .pdf directories and forwards each
    path to glean.submit; papers source streams the archived source
    PDF back from papers_dir/; papers remove cascades a paper out
    of the papers catalog, corpus, vector store, and papers_dir/;
    the papers <read> commands mirror the book-side read family
    against the paper backend.

  • MCP surface for papers: library.search accepts a kind switch
    (book / paper / unset for both); library.read_context /
    library.read_span and the other library read tools accept a
    kind field on their arguments; the papers.* read family,
    papers.fetch_source, and glean.submit expose papers to agent
    clients.

  • CSL-JSON item model with two-way conversion in bookrack-catalog.
    Biblio and Contributor gain paper-shaped fields (DOI,
    container-title, volume / issue / pages, publication date parts,
    …) and round-trip through the M8 paper columns without lossy
    detours.

  • KindedNodeId: every cross-store addressing surface
    (read_context, read_span, search citations, queue jobs) now
    carries the item kind alongside the intake id, so a hit, a context
    read, or a queued job is unambiguous between the two stores.
    library.search citations report the store they came from.

  • Extraction envelopes are written with an explicit kind prefix in
    their filename; bookrack-extract tolerates the un-prefixed legacy
    layout on read. bookrack doctor --rename-envelopes [--dry-run] is
    the one-shot migrator that rewrites existing envelopes into the new
    naming.

  • glean_paper STRUCTURE now assembles a Section tree from the
    heading-colored block stream: a BlockKind::Heading{1} opens a
    Section organizer under the Work root, a Heading{2} opens a
    Subsection (auto-opening a Section first when none is outstanding),
    depth-3+ Heading blocks stay as Heading leaves under the deepest
    open organizer, Body blocks attach as Paragraph leaves, and
    BlockKind::Caption blocks land as FigureCaption structural
    leaves. The abstract leaf is bit-for-bit unchanged (Tier 1 vector
    anchor): same NodeId allocation order, same intake:{id}:abstract
    stable anchor, same text / norm hashes, and the page bounds stay
    NULL. Body-leaf stable anchors continue to count Body blocks only,
    so a re-glean of a Phase-1 envelope still produces the same body
    hashes. When the heading pass identifies no candidates the tree
    falls back to the flat Phase-1 shape. The STRUCTURE audit row's
    metric_summary JSON gains sections, subsections, and
    headings counters alongside the existing body_leaves.

  • bookrack_extract::pdf_paper::extract_paper_structured: a paper-only
    coloring pass that promotes [BlockKind::Body] blocks to
    BlockKind::Heading / BlockKind::Caption using the PDF outline that
    the adapter already attached to the extraction's toc first, then a
    port of the pdffigures2 SectionTitleExtractor rule set over
    BlockStyle (font-size median, bold-majority, above-gap ratio,
    numbered prefixes including the CJK \u{7b2c}…\u{7ae0} /
    \u{7b2c}…\u{8282} families). Cross-page running headers whose
    case-folded text recurs on three or more pages are dropped from the
    candidate pool before promotion. The selected signal is recorded on
    Provenance::source_of_structure as outline / heuristic /
    mixed / none. glean_paper runs the pass against PDF
    extractions inside its EXTRACT stage; book-side adapters are not
    affected. A new BlockKind::Abstract variant is reserved for the
    paper structuring pass — book-side adapters never emit it.

  • bookrack doctor --install-pdfium: downloads the pinned PDFium
    build, verifies its SHA-256 against the pin, and unpacks the
    library into a per-user managed directory that the loader searches
    automatically. The first-run wizard offers the same download when
    the library is missing.

  • metadata.set / metadata.clear accept a reason that lands on the
    audit row, closing the gap where field edits — unlike approve / ack /
    reject — could never record their justification. The CLI and REPL
    expose it as an optional --reason flag; the MCP write tools require
    it, so an LLM edit always carries its rationale in the audit trail.

  • show_book (and the metadata.show report that embeds it) lists the
    book's active overrides — which fields are curated rather than
    extracted, by whom and when, with notes — alongside the merged
    effective_biblio, so curation state is visible without replaying
    the audit trail.

  • metadata.set accepts a confirmed mark (CLI/REPL --confirmed,
    MCP optional confirmed: true) recording that the curator checked
    the value against the source itself, not an external catalog. The
    audit grades a confirmed override strong unless a validation check
    (ISBN checksum, BCP-47 syntax, year range, language/body mismatch)
    fails. Rewriting the field without the mark drops it.

  • metadata.void (CLI/REPL bookrack metadata void, MCP
    library.metadata.void): suppress a field's extracted value with a
    NULL override when the value is known wrong and no correct value is
    at hand — the field reads as absent until one is set. metadata.clear
    removes the suppression. The MCP tool requires a reason.

  • metadata.reaudit (CLI/REPL bookrack metadata reaudit, MCP
    library.metadata.reaudit): re-run the metadata plausibility audit
    from the book's cached extraction envelope — no re-extraction — so
    the stored verdict / confidence reflect the current effective
    metadata after field corrections instead of reporting the
    ingest-time outcome forever. Only the rollup pair and one
    pipeline-audit row are written; the review status is untouched.
    Returns the previous and new verdict / confidence.

  • library.show_metadata_report (MCP): recompute the metadata
    plausibility audit from the book's cached extraction and return the
    full per-field report — origin (extracted / override /
    override_confirmed / voided), grade, flags, and hint per field,
    plus TOC shape flags and copyright-page block candidates — next to
    the stored rollup and review status, so a curator can see why a
    record reads needs_work / low without re-deriving the signals.
    Pure read, runs the default audit profile; metadata.reaudit
    remains the write path that refreshes the stored rollup.

  • Contributor curation: metadata.contributor_add writes an
    origin = "user" attribution (closed role set: author /
    translator / editor / other) that survives re-ingest and is
    immediately matched by find_books contributor_name;
    metadata.contributor_remove deletes one row by the
    contributor_id now listed in show_book, whatever its origin —
    the path for stripping a wrong extracted attribution. CLI/REPL:
    bookrack metadata contributor-add / contributor-remove; MCP:
    library.metadata.contributor_add / _remove with a required
    reason.

Changed

  • M8 catalog migration: state tables shed their book_ prefix in
    favour of item-shaped names (item_review, item_pipeline_audit,
    …) and gain a kind column where the table is shared between
    books and papers; scope strings in the catalog API are replaced by
    core::ItemKind. New paper-shaped columns land on biblio,
    contributor, and intake (source_pdf_path). The migration
    runs automatically at daemon preflight against an older library;
    the per-database backup it writes is clustered by database stem so
    the books-side and papers-side migrations cannot overwrite each
    other.

  • All MCP library read tools now reach the catalog and corpus through
    the control plane, closing the last direct-handle path. The
    standalone bookrack-mcp process and the in-tree daemon share one
    in-process scheduler for both reads and writes — completing the
    control-plane split started in 0.4.0.

  • The extraction envelope type is hoisted out of bookrack-ingest
    into bookrack-extract, so the book-side ingest pipeline and the
    paper-side glean pipeline consume one shared definition.

  • extract_paper_structured is now precision-first: it returns an
    empty TOC rather than guess. The pass runs in two cooperating
    stages — outline-guided heading promotion and a strict numbered
    heuristic with sequence validation — both gated so a noise line
    cannot survive: a candidate must carry a numbered prefix
    (N, N.M, N.M.P, Roman, or Appendix), be single-line, fit
    under 80 characters, have no math / arrow / geometric-shape
    Unicode, no @, no boilerplate prefix (we, the, vol.,
    https://, …). The sequence check accepts only candidates that
    advance the ascending series at their level, and the outline pass
    matches anchored entries against block text by label prefix (with
    numeric / Roman str...

Read more

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 10 Jun 19:31

Added

  • JSON-RPC control plane: the daemon serves a control socket, and
    every other process is a client of it. The method surface covers
    the write commands with progress events, queue lifecycle
    (queue.pause / queue.resume / queue.clear), verify.run,
    library.fork, diagnose.run, introspection (daemon.methods,
    daemon.mcp_tools), and a subscribable log/event channel.
  • bookrack repl: standalone control-socket client process. Reedline
    runs in the client; every command is dispatched as a JSON-RPC call
    over the daemon's control plane. The prompt shows a live
    [<state-indicator>bookrack:<library>/queue:<n>] > status line
    driven by events.subscribe. When stdin is not a TTY the client
    runs a batch loop: each line is parsed via the shared
    bookrack-repl-grammar and dispatched in sequence, and the
    process exits non-zero on the first RPC failure. When the daemon
    is not running the client exits with code 2 and the message
    bookrack daemon not running; start it with: bookrack run.
  • library.read_context and library.read_span MCP tools:
    structural passage reads around a hit, backed by a document-order
    leaves_in_doc_span corpus query, so an agent can widen a citation
    to its surroundings without a second search.
  • Search citations carry the intake id and structural anchors of the
    cited span.
  • crates/app: Tauri shell hosting the daemon in-process with a
    minimal Svelte 5 panel — groundwork for the tray GUI, not yet part
    of the release artifacts. Control-plane DTOs and events derive
    ts-rs schemas, generated as .ts files for the webview at test
    time.
  • WizardDriver trait behind the first-run wizard, with the CLI
    driver as the first implementation and room for a GUI driver.
  • The daemon raises its soft RLIMIT_NOFILE to the hard limit at
    startup, and bookrack doctor reports the effective limit. A
    GUI-launched daemon previously inherited a 256-fd default and
    starved mid-batch on LanceDB fragment files.

Changed

  • bookrack run defaults to the silent daemon: it no longer reads
    stdin, no longer spawns reedline, and emits no banner. Open an
    interactive REPL with bookrack repl in another process.
  • One-shot subcommands are rewritten as control-plane clients and
    dispatch their work to the running daemon over the control socket.
  • A second bookrack run against a live session brings the existing
    daemon to the front and exits cleanly; bookrack-mcp gains the
    same re-entry behaviour.
  • The ingest queue pauses on process-level failures instead of
    rapidly failing every remaining job; the error is the process's,
    not the books'.
  • Ingest error strings keep their flattened source chains, so root
    causes such as Too many open files reach the pipeline audit
    instead of an opaque vector store error.

Deprecated

  • bookrack run --legacy-repl: one-release transition flag (hidden
    from --help) that re-enables the in-process reedline REPL for CI
    scripts that fed it via stdin. The flag will be removed in the
    next release after this one; migrate to bookrack repl (which
    accepts stdin in batch mode for the same scripted use case).

Fixed

  • The RESET confirmation on destructive vector-store commands
    survives the control-plane client split.
  • The daemon process exits after a control-plane shutdown request
    instead of lingering, and the --legacy-repl path prints its
    session banner again.
  • The hint printed after library.fork names the correct
    model-switch command.