v0.4.2

Indexing a crawl full of large PDFs no longer runs a server out of memory.

Changes

• Large pages are read for text one at a time. Finding the text in a PDF can take far more memory than the file: in one crawl, a 3.3 MB PDF took 750 MB and a 12 MB one 1 GB, while a 33 MB one took 107 MB. With four workers each able to be in one at once, importing the CDC collection from Browsertrix used all of a 4 GB server's memory. A record over 2 MiB is now fetched and read alone. (#205)
• New index.max_extract_mb setting (default 16). No more than this much of any page is read for its text. HTML and rendered text past it are not searchable. A PDF larger than it is indexed by its file name, with a warning, and still replays. indice config shows it. (#205)
• The Docker image returns freed memory to the system. On Linux, glibc's allocator kept the memory each large PDF freed, so an import ratcheted up until a container memory limit killed it. The image now sets GLIBC_TUNABLES to hand that memory back. With both changes, the CDC crawl that was killed at a 2.5 GB container limit indexes in 1.3 to 2.1 GB. If you run the binary directly on Linux with little memory to spare, the configuration docs give the setting to use. (#206)

Upgrading

No reindex needed, and no configuration changes are required.

A reindex applies the new limit to existing crawls: a PDF over 16 MiB that was searchable by its contents is then found by its file name only. Set index.max_extract_mb higher first if you want to keep those, on a machine with the memory for it.

The container image is ghcr.io/edsu/indice:v0.4.2 (also :latest), for linux/amd64 and linux/arm64.

Full changelog: v0.4.1...v0.4.2

v0.4.1

Faster replay of whole collections, and ReplayWeb.page 2.5.4.

Changes

• Replaying a collection opens only the crawls a page needs. Replaying a page from a whole collection used to open and index every crawl in it before showing anything. For crawls streamed from Browsertrix that meant hundreds of remote reads, and seconds of blank page after the loading spinner went away. ReplayWeb.page now asks indice which crawl holds the page and opens that one. In testing, one page of a 9-crawl collection went from opening all 9 WACZs in 152 reads to 3 WACZs in 88. The other crawls opened hold that page's own stylesheets and scripts. (#203)
• A real capture comes first. When several crawls in a collection captured the same URL, replay now lands on a successful (2xx) capture ahead of an archived error page, however many captures there are. (#203)
• ReplayWeb.page 2.5.4, with wabac.js 2.27.4, which fixes how archived JavaScript is rewritten. (#202)
• Imports say what they skip. A re-run of indice import browsertrix prints a line for each WACZ it skips because it's already imported, so a resumed import no longer looks like it's starting over. (#201)
• rustls 0.23.43. (#196)

Upgrading

Nothing to do: no reindex, and no configuration changes.

The container image is ghcr.io/edsu/indice:v0.4.1 (also :latest), for linux/amd64 and linux/arm64.

Full changelog: v0.4.0...v0.4.1

v0.4.0

Draft a collection's finding aid with a language model, for a curator to review and edit, and give every replayed page an address of its own.

New

• Drafting descriptions. A curator can ask a language model for a first draft of a collection's description, narrative, subjects and dates. The draft is built from what indice already knows: the finding aid's existing fields, each crawl's provenance and seed pages, facet counts, and a sample of pages from the search index. The sample starts with each crawl's front page and then spreads across the collection's sites, skipping error pages. A draft never touches the creator, curator or rights fields. (#197)

  • Any OpenAI-compatible model, hosted or local (Ollama, llama.cpp, vLLM). It is off unless an operator adds an llm: section to config.yaml; the API key, if one is needed, goes in INDICE_LLM_API_KEY.
  • In the editor, a Draft a description button fills empty fields and offers the draft beside fields you've already written. Before anything is sent, the editor shows which model and service will receive it, the operator's note about that service (llm.notice), and the exact pages and prompt. Nothing is saved until you press Save.
  • From the command line, indice collection draft <name> prints a draft as YAML. --apply fills empty fields, --replace FIELD overwrites named ones, and --show-prompt prints what would be sent without calling a model.
  • Provenance. The finding aid records which fields hold a model's drafted text, with the model, the service and when. The collection page says so in one line. A field you rewrite leaves the record. Each draft request is in the audit trail.
  • Configurable. describe.max_input_chars sets how much text is sent (24,000 characters by default), and describe.prompt_file replaces the built-in prompt with your own.

  See Drafting descriptions.

• Every replayed page has an address of its own. Replay links are now short and readable, in the manner of Wayback-style archives: /crawl/{id}/{timestamp}/{archived url}, or /collection/{id}/{timestamp}/{archived url} for a whole collection. The address bar follows the reader from page to page, so copying it always gives a link to the page in view. The request log records only which crawl or collection was replayed, never the page. (#200)

Changes

• A finding aid's narrative uses the full width of its column on the collection page. (#197)
• compose.yaml passes INDICE_LLM_API_KEY to indice, so a Compose deployment can use a hosted model. (#197)

Upgrading

Nothing to do: no reindex, and drafting stays off until you configure a model. Old /replay/viewer?… links no longer work; links from indice's own pages all use the new addresses. If you deploy with Docker Compose, fetch the compose.yaml from this release to pass the API key through.

The container image is ghcr.io/edsu/indice:v0.4.0 (also :latest), for linux/amd64 and linux/arm64.

Full changelog: v0.3.1...v0.4.0

v0.3.1

A fix release for a regression in v0.3.0.

Fixes

• Streaming a remote WACZ no longer reads the whole file, several times over. v0.3.0 upgraded the zip library to 9.0, whose archive reader scans the bodies of zip entries written with data descriptors, which is how Browsertrix writes WACZs, just to open the file. On a local disk that goes unnoticed. Over HTTP every byte is a range request, so streaming a 1.51 GB public Browsertrix WACZ read 10.4 GB in 35,723 requests. v0.3.1 uses zip 9.2, which reads 1.36 GB in 1,139 requests, as v0.2.0 did. Most of that is genuine: the collection was mostly PDFs, which indice reads whole to index their text. A new test opens a WACZ through a byte-counting reader and fails if opening reads more than its directory, so a future zip upgrade can't bring this back unnoticed. (#193)
• A tab icon. indice now names its icon, so browsers no longer request /favicon.ico, which an auth proxy answers with 403 on every page view. (#192)

Upgrading

Nothing to do beyond upgrading: no reindex, no configuration changes. Crawls added in streaming mode with v0.3.0 were indexed correctly, only slowly.

The container image is ghcr.io/edsu/indice:v0.3.1 (also :latest), for linux/amd64 and linux/arm64.

Full changelog: v0.3.0...v0.3.1

v0.3.0

Highlights

Ready for the open web

v0.3.0 is the release that makes it reasonable to run indice as a public server that strangers can read and approved people can change.

• Replay runs on an origin of its own. Archived pages run their own JavaScript. indice now serves replay from a second listener and hostname, so a captured page's script can't act as the person reading it or reach indice's write API. Each origin sends its own Content-Security-Policy.
• Three roles, from a file you can commit. users.yaml lists who is a curator (adds and describes) and who is an admin (removes, moderates, and imports through the instance's Browsertrix and Archive-It credentials). Anyone else is a reader, signed in or not. indice re-reads the file when it changes, so approving someone needs no restart. A server refuses to start without one.
• Custody and an audit trail. Each crawl records who accessioned it, and a curator can undo their own additions. An append-only log under events/ records who changed what.
• Limits on what anyone can ask of a server. Request limits per client and overall, bounded uploads (uploads.max_mb, uploads.max_total_gb), and crawls added by URL or upload only, never by a path on the server's disk.
• Privacy. Readers' searches and addresses stay out of the logs, a public failure answers with a reference rather than the error, and replay and search results are kept out of search engines.

Read Securing a server and the deploy guide before putting one on the open web.

A new look

The reading room and the workroom now look different on purpose: a manila header and accents when you can change the archive, the archive's own words in a serif, time drawn as a real axis on search and on collections, and readable names for crawls. Colours, faces and spacing are design tokens by role (tokens.css), in a layer meant for theming.

Name your archive

config.yaml gains a site: section: the archive's name (header, homepage, page titles, replay viewer), a tagline, a Markdown banner for every page, and request_access, shown to someone signed in but not on the roster. See Site name, tagline and banner.

Notes, found again

A note's passage is highlighted when you open its page, clicking a note scrolls to it, and links from the notes list and from search results open the page at the note.

Indexing large WACZs

A WACZ is indexed a page at a time, so memory no longer grows with the crawl, and no file inside a WACZ is inflated past index.max_member_mb (default 1 GB), so a small WACZ can't unpack into gigabytes. indice reconcile checks that the index and the manifest agree, and says how to repair them.

Learn Rust through indice

The Rust primer is rewritten as a series on the docs site, teaching Rust through indice's own code.

Upgrading

• No reindex is needed. Existing search indexes open as they are.
• Downgrading needs one. The search engine moved to tantivy 0.27, which records a few fields' settings in a new form the first time it writes to an index. v0.2.0 then refuses that index as having "an older schema"; indice reindex rebuilds it.
• On a workstation, --manage is gone. Bound to loopback, the operator always has the workroom, so drop the flag: indice serve --manage now fails with an unknown argument. Replay listens on the next port up (8081 beside 8080, or --replay-bind), which must be free.
• A server needs more than it did:
  • --site-url, the address readers use. indice no longer infers it from request headers.
  • A second hostname for replay, given as --replay-url (and --replay-bind for its listener), different from the site's host.
  • A users.yaml. An empty users: [] is a read-only archive.
  • The shipped stack has changed shape. compose.yaml adds oauth2-proxy, for sign-in through any OIDC provider, and Caddy serves a second site for replay. If you run the stack, take compose.yaml and Caddyfile from this release and follow the deploy guide.
• The HTTP API no longer accepts a server-side path for a new crawl: add one by http(s) URL or upload. indice index on the command line still takes local files.

The container image is ghcr.io/edsu/indice:v0.3.0 (also :latest), for linux/amd64 and linux/arm64.

Full changelog: v0.2.0...v0.3.0

v0.2.0

Highlights

Page annotations

Attach notes to a specific capture — a whole page, or a passage you select in the replayed page. Notes are freeform Markdown, public to read, and only signed-in users can write them (edit/delete your own only).

• A Notes panel beside the replay; passage notes highlight in place in the archived page, without modifying it — the anchor is a W3C Web Annotation TextQuoteSelector, re-located on each visit.
• A per-collection annotations index at /collection/{id}/annotations, linked from the collection page.
• Notes are full-text searchable, appearing as their own "Note by …" results.
• Stored as one JSON-LD line per note in collections/<slug>/annotations.jsonl — diff-friendly and committable alongside the finding aid.

Documented at indice.page/docs/guides/annotations.

A website and a manual

indice.page now has a landing page and a full documentation manual — install, quickstart, guides for searching, importing from Browsertrix and Archive-It, managing, deploying and scaling, plus CLI/API/configuration reference.

Replay engine updated

Vendored ReplayWeb.page 2.4.6 → 2.5.3 (wabac.js 2.26.4 → 2.27.2).

Fixes

• reindex no longer drops Browsertrix crawls. A wrong source check meant every indice reindex silently skipped Browsertrix-imported crawls, leaving them out of the rebuilt index. Re-run indice reindex to restore them.
• Deleting a collection now removes its annotations from the search index instead of leaving them searchable.
• Creating an annotation validates the collection exists (404) and requires a URL and timestamp, rather than creating an orphan.
• The replay Notes panel no longer leaves an empty column when there's no collection context, and a stale highlight no longer persists across page navigation.

Internals

Large files were split into focused modules for legibility — index.rs (4,028 lines), server.rs (3,967), search.rs (2,947) and views.rs (1,593) are now directory modules, and no file carries more than ~650 lines of non-test code. The ingest pipeline gained a WaczAccess handle that cut index_one from ~300 lines to ~50 and removed the remote-vs-local duplication. All of it verified as behaviour-preserving.

Security hardening: collection ids are now a validated CollectionId newtype, so building a filesystem path from an unvalidated id is a compile error, and the property is exhaustively tested. The annotation sanitizer is pinned against 15 XSS bypass techniques plus an end-to-end test. CI runs with least-privilege token permissions, and CodeQL moved to advanced setup so vendored bundles aren't scanned as if they were ours.

Upgrading

The search schema gained a field, so run indice reindex once after upgrading — serve will refuse to start until you do, and tell you so.

Full changelog: v0.1.0...v0.2.0

What's Changed

• Homebrew tap tooling + RELEASING guide (deploy .8) by @edsu in #106
• Landing site (Astro → GitHub Pages) — user-manual 8qtx phase 1 by @edsu in #107
• Docs manual (Starlight) + slim README by @edsu in #108
• site: serve at indice.inkdroid.org and add Plausible analytics by @edsu in #112
• site: serve at indice.page by @edsu in #113
• Page annotations: storage + endpoints + UI (gnqf.2 + gnqf.3) by @edsu in #111
• site/docs: document annotations; surface imports + annotations on the landing page by @edsu in #114
• Fix: reindex no longer skips Browsertrix sources (nk69) by @edsu in #115
• Bump the cargo-minor group with 4 updates by @dependabot[bot] in #110
• Bump the actions group with 6 updates by @dependabot[bot] in #109
• Annotations: warn at authoring that notes are public (yrxe) by @edsu in #116
• Split index.rs into a legible index/ module (qrd1) by @edsu in #117
• Ingest: WaczAccess handle + a short index_one pipeline (wti6) by @edsu in #118
• Split server/views/search into modules; free the accession-desk JS; validated CollectionId by @edsu in #119
• ci: least-privilege GITHUB_TOKEN permissions for the CI workflow by @edsu in #120
• codeql: switch to advanced setup to exclude vendored ReplayWeb.page bundles by @edsu in #122
• Upgrade vendored ReplayWeb.page 2.4.6 → 2.5.3 by @edsu in #121
• test: pin the annotation XSS invariant end-to-end by @edsu in #123

Full Changelog: v0.1.0...v0.2.0

v0.1.0

An initial release 🎂 🥳 🎈

Ok, this is mostly to bootstrap a homebrew recipe ... but it should be pretty functional. A manual separate from the README.md is in the works.