Searching inside tickets: the actual Tika docker command, the -full/OCR trap, localhost binding, and the full format list
Scheduled tasks: add the attachment extraction worker, and catch up
Checking rather than assuming turned up a real gap: the cron page listed
FOUR jobs and the attachment worker shipped as a fifth. Somebody setting
up a new install from that page would have configured everything except
the one added this week.
Added to the table and to both the Windows and Linux command blocks,
with a note that it is the odd one out: its absence is NOT silent
failure, because documents are still read opportunistically while people
use FreeITSM. The job keeps up with a busy desk rather than being the
only thing that works. That distinction matters on a page whose whole
premise is "these fail silently".
Full-Text-Search: #1078 and #1079 added to the shipped list.
Searching-Inside-Tickets: "if a document is not turning up" now says
System > Search names the file and the reason, rather than only implying
totals.
Search: attachment extraction tier 2 (#1076-#1077). Discussion #53 closed.
The extraction guide's §9 said "what tier 2 will need". It now describes
something that exists.
Documents the thing that shapes the whole tier: CONFIGURED and AVAILABLE
are different states. tikaExtract() returns three outcomes, and a
service being DOWN must write `pending`, never `failed` - a five-minute
outage must not blacklist every PDF that arrived during it. Also why
there is deliberately no health check before each file: the extraction
attempt IS the check, and a ping per attachment would be an extra round
trip each time.
Why both drains exist: a cron-only design does nothing at all on an
install that has not set one up, which includes every evaluation.
And §9.3 on the two bugs, because they share a shape worth recognising -
a queue that looks busy and clears nothing, which is much harder to
notice than a crash. One: the indexer would not reconsider `pending`, so
the drain could never move the depth. Two: the requeue was
indiscriminate and put files Tika is never asked about into a queue that
could not clear them. The general rule is written down: a status nothing
will ever act on is a leak, so make the worker able to terminate those.
Operational notes that will otherwise cost somebody an afternoon: use
the -full image or there is no OCR, and Tika has no authentication at
all so it must never be published.
User page: PDFs and scanned documents move from "not read" to "read once
you add a document reader", with where to set it up and where the
background-work switches live.
Full-Text-Search: all three of the discussion's original questions now
answered yes. §8.2's three tiers are two-and-a-half - the external
SEARCH ENGINE remains deliberately unbuilt and §9.5's argument still
stands.
All 13 files named verified to exist; all links resolve.
Add "Attachment text extraction — Developer Guide" (#1075)
A page of its own rather than another section on the corpus guide: the
subject carries more security reasoning than anything else in search,
and it needs room.
Covers the two tiers and, importantly, that the split is drawn on a
SECURITY line rather than a convenience one - tier 2 formats need a real
document parser, and the input arrives from anyone who can email the
service desk through an endpoint with no authentication at all.
Includes an answer to the question that will keep coming up: "a PDF with
a text layer is just compressed text, can't PHP do that?" Partly, and it
is the most reasonable-sounding wrong turn available. Font encodings,
compressed cross-reference tables, empty-password encryption and
positioned-not-flowed text each break it, and the objection is the
FAILURE MODE rather than the effort: unsupported is honest, silent
mojibake looks indexed and quietly matches nothing. Records what a
pure-PHP tier would have to promise (fail closed, never emit doubtful
text) if it is ever wanted.
Documents why attachment_text is the durable record and search_documents
the derived copy, with the two things it buys - a rebuild reads no
files, and whole-ticket reindexing re-opens nothing.
Every guard with the reason it exists, including that zip bombs are
refused on DECLARED sizes before an entry is read, and that writing the
test for that guard exhausted PHP's memory twice, which is the
demonstration.
The OOXML welding trap in full, since it is the one that silently ruins
search. Plus the known gaps stated honestly: docx headers/footers/
footnotes, xlsx inline numbers, pptx speaker notes.
User page: attachments are now a table of what IS and is not read,
leading on the fact that FreeITSM says which files it could not read
rather than staying quiet.
All links resolve; all eight files named verified to exist.
Search: System → Search screen (#1074), and tidy the status tables
User page: the backfill section is replaced by "Checking on it later:
System → Search". Rebuilding no longer needs a command line, which is
the part that mattered on hosting that does not give you one. Keeps the
three cases where you WOULD rebuild, and points at the minimum word
length as the thing that explains most "search finds nothing" reports.
Full-Text-Search: #1074 moved into the SHIPPED table where it belongs -
it had landed under "not built" - and the two §8.4 selectors listed as
deliberately deferred in its place, with the reason.
Dev guide gains §10 on the screen and, more usefully, on why the rebuild
is CHUNKED: it is the one operation that scales with the install, and a
single request over years of tickets dies past max_execution_time
leaving a half-built index and no way to tell how far it got. Records
the three details that make resuming work - last_ticket_id,
tickets_remaining, and articles riding the final slice only because they
are not keyed by ticket id.
Sections renumbered (old §10 is now §11) and its stale "⌘K content
search" entry replaced, since that shipped in #1072.
All 25 files named in the file table verified to exist.
Search: the index keeps itself current (#1070-#1071)
All three search pages said indexing was manual. It is not any more.
Searching-Inside-Tickets: the backfill is now genuinely one-off, and the
warning that new tickets stay invisible until you re-run it is replaced
by what actually happens. Adds the three cases where you WOULD run it
again - after correcting the full-text settings, on suspected drift, and
after restoring an old backup - because "never" is not true either.
"Attachment text" replaces it as the honest what-it-does-not-find entry.
Full-Text-Search: status block updated, and §3.5's open question is
marked answered. It asked whether to index history or only from
switch-on; the answer is both, and the two share one document builder.
Command-palette content search moves onto the not-built list, since the
discussion asked for it and it is genuinely still missing.
Search-Corpus-Developer-Guide gains §9 on the subscriber - why it
reindexes the WHOLE ticket rather than the changed row (ordering-immune,
self-healing, cheap), why the dispatch must come AFTER the commit, and
why the document construction moved out of the backfill: if the two
drifted, a result would depend on how a ticket happened to get indexed.
And §9a on the trap that will catch the next subscriber too:
ticket.created fired from ONE place, the analyst path, so it missed the
two channels most tickets arrive through. The mailbox file documents the
mistaken belief in a comment.
Search: the user page and developer guide for what actually shipped
The design page was carrying the whole feature on its own, which conflates two
different readers: someone who wants to search their tickets, and someone who
has to maintain the thing.
Searching-Inside-Tickets — how to use it. The new field, what a result means,
the three query forms, and honestly what it does NOT do: attachments are not
searched, and nothing indexes new tickets until the backfill is run again. Also
the two one-off setup steps, and why D007 is worth running ONCE at setup rather
than only when something looks broken -- because database search fails quietly,
and a "longest word" limit makes words like authentication unfindable while
shorter ones work perfectly.
Search-Corpus-Developer-Guide — the traps, in the order they will bite:
- one corpus rather than an index per table, because relevance scores from
different full-text indexes are not comparable
- tenant_scope, because NULL means OPPOSITE things in the sources
- the predicate goes INTO the query, and why post-filtering starves results
worst for the least privileged user
- query translation is not pass-through: requiring a term below the index
minimum makes the WHOLE query match nothing
- InnoDB not exposing uncommitted rows to MATCH, and the three ways that
surfaces
- the three server settings that break search silently, and that WAMP ships
one of them wrong in [wampmysqld64], not [mysqld]
The design page now points at both and says plainly that it is the reasoning
rather than the manual -- its lasting value is the ruled-out section, which is
the part nobody can reconstruct from the code.
Every file named in the guide verified to exist; every wiki link verified to
resolve. Both pages added to the sidebar under the command palette.