v0.4.2

Indexing a crawl full of large PDFs no longer runs a server out of memory.

Changes

• Large pages are read for text one at a time. Finding the text in a PDF can take far more memory than the file: in one crawl, a 3.3 MB PDF took 750 MB and a 12 MB one 1 GB, while a 33 MB one took 107 MB. With four workers each able to be in one at once, importing the CDC collection from Browsertrix used all of a 4 GB server's memory. A record over 2 MiB is now fetched and read alone. (#205)
• New index.max_extract_mb setting (default 16). No more than this much of any page is read for its text. HTML and rendered text past it are not searchable. A PDF larger than it is indexed by its file name, with a warning, and still replays. indice config shows it. (#205)
• The Docker image returns freed memory to the system. On Linux, glibc's allocator kept the memory each large PDF freed, so an import ratcheted up until a container memory limit killed it. The image now sets GLIBC_TUNABLES to hand that memory back. With both changes, the CDC crawl that was killed at a 2.5 GB container limit indexes in 1.3 to 2.1 GB. If you run the binary directly on Linux with little memory to spare, the configuration docs give the setting to use. (#206)

Upgrading

No reindex needed, and no configuration changes are required.

A reindex applies the new limit to existing crawls: a PDF over 16 MiB that was searchable by its contents is then found by its file name only. Set index.max_extract_mb higher first if you want to keep those, on a machine with the memory for it.

The container image is ghcr.io/edsu/indice:v0.4.2 (also :latest), for linux/amd64 and linux/arm64.

Full changelog: v0.4.1...v0.4.2