Extraction guide: concurrency, the claim, and the page-hang bound
Ed asked what happens with two or three analysts logged in at once.
Nothing was ever at risk of corruption - every write is an idempotent
upsert - but two workers would select the same rows and send the same
files to the extractor, which for OCR is minutes of CPU paid for twice.
§9.4 documents the claim: a conditional atomic UPDATE to a transient
`extracting` status, why rowCount() is not enough to know which rows
were won, and the stale-claim reaper. Includes the trap that `extracting`
must ALSO be in the indexer's reconsider list, because the claiming
worker is the one that then indexes - the same shape as the `pending`
bug in §9.3, and worth stating as a rule: a transient status nothing
acts on is just a different kind of leak.
§9.5 on the opportunistic drain running inside somebody's request, and
why it needed all THREE bounds - one item, a wall-clock deadline, and a
shorter extractor timeout, because a single file cannot be interrupted
once curl is waiting on it so the deadline alone does not help. Plus why
that timeout is restored in a finally.
§9.6 on the new "attachments that are not searchable" list: counts
answer "is something wrong", only a filename answers "why can't I find
that invoice".
Status table gains `extracting` and marks `unsupported` as not final.
Search: attachment extraction tier 2 (#1076-#1077). Discussion #53 closed.
The extraction guide's §9 said "what tier 2 will need". It now describes
something that exists.
Documents the thing that shapes the whole tier: CONFIGURED and AVAILABLE
are different states. tikaExtract() returns three outcomes, and a
service being DOWN must write `pending`, never `failed` - a five-minute
outage must not blacklist every PDF that arrived during it. Also why
there is deliberately no health check before each file: the extraction
attempt IS the check, and a ping per attachment would be an extra round
trip each time.
Why both drains exist: a cron-only design does nothing at all on an
install that has not set one up, which includes every evaluation.
And §9.3 on the two bugs, because they share a shape worth recognising -
a queue that looks busy and clears nothing, which is much harder to
notice than a crash. One: the indexer would not reconsider `pending`, so
the drain could never move the depth. Two: the requeue was
indiscriminate and put files Tika is never asked about into a queue that
could not clear them. The general rule is written down: a status nothing
will ever act on is a leak, so make the worker able to terminate those.
Operational notes that will otherwise cost somebody an afternoon: use
the -full image or there is no OCR, and Tika has no authentication at
all so it must never be published.
User page: PDFs and scanned documents move from "not read" to "read once
you add a document reader", with where to set it up and where the
background-work switches live.
Full-Text-Search: all three of the discussion's original questions now
answered yes. §8.2's three tiers are two-and-a-half - the external
SEARCH ENGINE remains deliberately unbuilt and §9.5's argument still
stands.
All 13 files named verified to exist; all links resolve.
Add "Attachment text extraction — Developer Guide" (#1075)
A page of its own rather than another section on the corpus guide: the
subject carries more security reasoning than anything else in search,
and it needs room.
Covers the two tiers and, importantly, that the split is drawn on a
SECURITY line rather than a convenience one - tier 2 formats need a real
document parser, and the input arrives from anyone who can email the
service desk through an endpoint with no authentication at all.
Includes an answer to the question that will keep coming up: "a PDF with
a text layer is just compressed text, can't PHP do that?" Partly, and it
is the most reasonable-sounding wrong turn available. Font encodings,
compressed cross-reference tables, empty-password encryption and
positioned-not-flowed text each break it, and the objection is the
FAILURE MODE rather than the effort: unsupported is honest, silent
mojibake looks indexed and quietly matches nothing. Records what a
pure-PHP tier would have to promise (fail closed, never emit doubtful
text) if it is ever wanted.
Documents why attachment_text is the durable record and search_documents
the derived copy, with the two things it buys - a rebuild reads no
files, and whole-ticket reindexing re-opens nothing.
Every guard with the reason it exists, including that zip bombs are
refused on DECLARED sizes before an entry is read, and that writing the
test for that guard exhausted PHP's memory twice, which is the
demonstration.
The OOXML welding trap in full, since it is the one that silently ruins
search. Plus the known gaps stated honestly: docx headers/footers/
footnotes, xlsx inline numbers, pptx speaker notes.
User page: attachments are now a table of what IS and is not read,
leading on the fact that FreeITSM says which files it could not read
rather than staying quiet.
All links resolve; all eight files named verified to exist.