v4.100.242

Documents are read in a process of their own

Reading a book used to happen inside the session's own process, which in VS
Code is the extension host that every installed extension shares. A long
scanned PDF is hundreds of megabytes of decoding and text recognition, and when
that process ran out of memory the whole host went down: the conversation
stopped, and every other extension restarted with it.

Each document is now read in a separate process, started for that one file and
gone when it has answered. This covers library_check, library_add and
extract_document, for every format, in the extension and the CLI.

• A reader that runs out of memory, or stops for any other reason, is one file
  that was not read. The call's report names the file, the page it stopped on
  and why, and goes on with the rest. The session is not affected.
• Progress, cancelling, and the vision model reading scanned pages all work as
  before.
• At most two documents are read at once, however many a single turn asks for.
• The log says, for each file, how long the read took and how much memory it
  needed at its peak.
• A book's pictures stay on disk from the moment they are read until they are
  filed, and are filed one at a time. Before, every picture of every book in a
  library_add call was held in the session's memory until the call ended:
  396 MB for one 769-page book. Filing that book now peaks at 311 MB in all.
• Pictures left behind by a session that ended mid-import are cleared on a
  later import.

Where a reader process cannot be started, the document is read in the session's
own process as before, and the log says so.

EPUBs that would not open

About one EPUB in twenty-five failed to open with "Cannot read properties of
undefined (reading 'navMap')" or "(reading 'metadata')": in one import, 21 of 35
books. The files were fine. Their package files write each element with a
namespace prefix (<opf:package>, <ncx:navMap>, <ns0:package>,
<odfc:container>), which is valid and which the EPUB parser did not
understand. They now open, as do books with an empty table of contents or an
empty guide, a chapter listed in the reading order but missing from the book,
or a link to a stylesheet the book does not carry.

Measured on a shelf of 7,157 EPUBs, each opened and every chapter loaded
with the old parser and the new one: 512 books that failed now open, and no
book that opened before fails now. The 37 that still fail are damaged (cut
short, not a ZIP archive, broken XML) or encrypted, and each says so.

The import report stays

When library_check, library_add or another call that shows live progress
ends, its row now keeps what it came to: a Report line under the call,
collapsed to its first line, that opens to the whole report and has a copy
button. Before, the progress lines went with the call and the row was left
with nothing but the file names, so a run with many failures could not be
reviewed.

Export what you tick, and delete with a Yes

In Settings > Library, the Export button on every section, shelf and
book is gone. Each has a box to tick instead, and the boxes mix: some books
from one shelf, a whole shelf from another section and a whole section can go
into one file. A section or shelf partly ticked shows a partly filled box.
Export all… is now Export…: with nothing ticked it writes the whole
Library, with something ticked only that, and the line under it says which.

Delete is a trash-bin button, which leaves the row more room for the
title, and it asks a question with Yes and No before anything moves:
"Move "Godot Tilemaps" to the trash? It is kept there 30 days." Removing a
shelf or section says how many books go to the trash with it.

The checklist reads lists without boxes

A model that kept its checklist as a plain list, ["Inventory the books", "Import Data & AI", "Report"], had it read as nothing: no checklist at the
top of the task, and the task_progress tool answering "No checklist sent"
to every list it was given. Each entry is now an item, open unless it starts
with [x], and the same goes for the list sent as the text of an array, or
as plain lines with no boxes. Nothing is ticked from the wording: "Data & AI:
done" stays open until the model ticks it.

The reminder of the checklist, written to the model every few calls, no
longer shows at the end of a tool's Report. It used to end every third
import report with "Task progress (0/0 done)".

Files up to 1 GB are read, and the limit is a setting

The Document Reader refused any file over 200 MB, a limit from when books
were read inside the extension itself. Each file is read in a process of its
own now, so the limit is 1 GB, and Largest file to read sets it: in the
Document Reader's options and in Settings > Library, one setting shown in
both. A file over it is refused by name, and the message says where to change
it.

Measured on the largest books of a real shelf: a 228 MB EPUB read in 3 s with
812 MB of memory at its peak, a 411 MB scanned PDF in 80 s with 2.6 GB. Two
files are read at once, so lower the limit on a machine with little memory.

Related documents are no longer taken for versions

The Library's duplicate check took documents that share much of their text
for versions of one another. A folder of AMD's processor manuals showed why
that is wrong: the references for two different processors share 77% of
their text, because whole register chapters are reused, more than two
revisions of one revision guide do (68%). Every pair was reported as
"probably another version", and the model stopped to ask.

A document with pages is now also known by its running heads, the line
printed on most of its pages ("56683 Rev 1.04 - Nov 2021 Revision Guide for
AMD Family 19h Models 00h-0Fh"). When their identifying part differs, the
documents are different, however much text they share, and nothing is
reported. Between revisions only the revision and the date change, so those
are still found. A shared title with almost no text in common no longer
counts either: every volume of the AMD64 programmer's manual carries the
title of the whole set.

On that folder of 61 manuals, the pairs reported went from every processor
reference against every other to the ones that matter: the same file twice,
revisions of one document, and parts of one volume.