Help Assistant Internals 3 Answers And Misses

✍️ Help assistant internals, part 3: writing the answer, and learning from misses

Part 3 of 3. Part 1: from help pages to an index · Part 2: the hybrid search · Every design question, answered

Part 2 ended with five ranked sections and a judgement of how sure the search is. This page covers what happens next: the ladder of what the assistant can do with whatever is set up, the optional chat AI that writes an answer, permissions, the list of questions the help could not answer, and the traps found on the way.

---

1. The ladder: it works at four levels

Nothing about the assistant needs AI. Each thing an admin adds raises it a level, and each level falls back to the one below when something is missing or fails:

Level	Needs	What the asker gets	helpAssistantLevel()
Search help	nothing	canned line + help cards, ranked by words (BM25)	search
+ matching by meaning	an embedding model (key)	the same, ranked by words and meaning merged	search_meaning
+ written answers	a chat AI (key)	a written answer in their language, citing the help	ask
both	both	the AI is given the merged results - the full hybrid	ask

flowchart TD
    Q[Question] --> W[Word search - always]
    W --> V{Embedding model<br>ready?}
    V -- yes --> M[Merge words + meaning]
    V -- no / fails --> R[Words alone]
    M --> A{Written answers<br>on?}
    R --> A
    A -- no --> C[Canned line + cards]
    A -- yes --> AI[Chat AI reads top 4 sections]
    AI -- answer --> ANS[Answer + 'From the help' cards]
    AI -- NOT_COVERED --> NC[Honest line + closest cards<br>+ unanswered list]
    AI -- error / no credit --> C

Loading

The two AI jobs are separate settings with separate keys (namespaces help_embed and help_ai) - so they can be different providers. The embedding model only chooses sections; the chat AI only reads text. That is why Claude Haiku writing answers works perfectly well with OpenAI's vectors choosing what it reads: Haiku never sees a vector.

The System page's first panel says this in plain words before any key field - "The help assistant works without AI" - with a live "Right now: Search help" line showing the current level.

---

2. Written answers (the Ask layer)

includes/help_ask.php. helpAssistantAnswer() runs the search exactly as in part 2, then, if helpAskReady() (switch help_ask_enabled plus a configured help_ai provider), calls helpAskAnswer().

What goes to the AI

• The best four sections, numbered [1]-[4] (HELP_ASK_SECTIONS).
• Each up to 3,000 characters (HELP_ASK_SECTION_CHARS), taken from the passage the search matched when the section is long - helpRank() returns its best passage for this. Tickets → Settings is 10,000 characters; the AI gets the Departments part.
• Sections in the asker's language where they exist (part 1's per-section fallback), so menu names come out as the asker sees them.
• A label [needs permission the user does not have] on sections the asker cannot act on (section 4).
• The question - which never includes ticket data, names or anything else from the database. The AI sees help text and the question, nothing more.

The rules

helpAskSystemPrompt(), in short:

1. Only the numbered sections (and any fetched with search_help) - never other products, never general IT knowledge.
2. Never invent a menu, button, field or page; write names exactly as the sections do - they are the words on the screen.
3. Answer in the user's language, by name ("Hungarian"), or the question's language if clearly different - keeping screen names as written.
4. Brief; start with the answer. Never "Based on the help sections…".
5. Check the sections really are about the question (analysts are not requesters). If not, or the question is not in English, call search_help in English - up to twice. Then NOT_COVERED, never a guess.
6. Without the permission: say who can do it, not how.
7. End with SOURCES: 1, 3.
8. Signpost: the way there as a bold path starting from the module - Tickets → Settings → Departments, System → Database verification - each step in the sections' own words and language (German: Tickets → Einstellungen → Abteilungen). Each section is labelled with its module's menu name for this, because the help writes paths from inside the module ("Settings → Renewals"); the help page's name ("LMS guide") is never a step.

The tool: search_help

{ "name": "search_help", "parameters": { "words": "English search words" } }

It runs helpRank() over the analyst's own sections (permissions still apply) and returns new numbered sections, continuing the numbering, so SOURCES: can cite them. This is how a language with no help files gets answered: the AI translates by searching in English itself. The design originally had a separate translation call; this replaced it for free.

For the same reason, a question where the search found nothing still goes to the AI - before that change, Hungarian and Polish questions never reached it.

Reading the reply

helpAskParse():

The AI wrote	Becomes
text + SOURCES: 2, 5	kind answer: the text, and "From the help" cards for sections 2 and 5 (numbers it was never given are ignored)
NOT_COVERED, alone or with an explanation after it	kind not_covered: "The help doesn't seem to cover that. These are the closest sections:" + three cards, and the question joins the unanswered list
nothing usable, an error, no credit	the plain search reply from part 2

Inline [1] markers and the SOURCES line are removed. The window builds the answer from text nodes (line breaks and **bold** only) - it is model output and never goes through innerHTML.

Measured

Claude Haiku, the 65-question set, answer citing a right section:

Plain (47)	Exact (6)	Vague (6)	Other language (6)
46	6	5	3 + 1 honest "not covered"

About 2.7 seconds and ~3,000 tokens in, ~150 out - around a third of a penny a question. Polish and Hungarian come back in Polish and Hungarian with the English menu names kept.

Known miss: a Spanish question about resetting an analyst's password is answered from Tickets → Users (requesters). System help is English only, so the Spanish words do not reach it, and the model did not search in English. It goes away when System help moves into the language files.

---

3. Speed and cost, end to end

Step	Time	Cost
Word search (722 sections, 1,853 passages)	~80 ms	free
Embed the question	~0.4 s	a tiny fraction of a penny
Compare with the candidate vectors	~16 ms	free
Written answer (Haiku)	~2.7 s	~0.3p
Rebuilding all vectors (722 sections, batches of 64)	a minute or two, in the background	about a penny (none at all for the shipped model)

The help is never read at question time - the index (part 1) is built ahead. The AI sees four sections, not 74 pages.

---

4. Permissions

Two layers:

1. Modules. helpSectionsFor() keeps only sections from modules the analyst can open (analystCanAccessModule()), plus pages that belong to no module. The AI's search_help searches the same filtered set. An analyst without Contracts can never be shown Contracts help - by the search or the AI.
2. Permissions inside a module. A section marked data-cap="tickets.manage" (the settings help of eleven modules) is still shown - knowing a setting exists is useful - but labelled for the AI, and rule 6 makes the answer "An administrator, or someone with Tickets management rights, can do this" rather than the steps. helpAskCapHeld() counts an umbrella permission as held when the analyst has any permission of that module.

api/help_assistant/ is exempt from the module-access coverage test (with a written reason): the assistant belongs to no module and filters per section instead.

---

5. Unanswered questions

The whole point of asking a help system questions is finding out where the help is thin. helpAssistantShouldRecord() records replies of kind none, weak and not_covered in help_assistant_unanswered, shown on System → Help assistant with Dealt with buttons.

The wide check

TRAP: help missing for this asker is not help missing. An analyst without Contracts asks about contract renewals; their search finds nothing - but the help exists. Recording it would send admins hunting for a gap that isn't there.

helpAssistantRecordUnanswered() asks the question again, search only, as an all-modules administrator. It skips recording only when that wider search finds a good answer in a module the asker cannot open. A not_covered from an AI that could see the right module is still recorded - the help is there but not good enough.

The bell

• Event help.unanswered, flagged admin_only - it is filtered out of effectivePreferences() for everyone else, so non-admins never even see it in their notification settings.
• Every entry uses entity_id 1, so they coalesce into one bell entry ("5 unanswered help questions") instead of one per question.
• The actor is the asker - and notifications never go to their own actor, so the asker is never told their question was flagged.
• It links to system/help-assistant/#unanswered (entity_links.php).

Names

By default the list says "an analyst". An admin can tick Record who asked per analyst (analysts.help_assistant_named); that analyst's window then shows a notice saying so. Privacy is the default, honesty when it is switched on.

Lifecycle: conversations are personal (deleted with the analyst); unanswered rows' analyst_id and dealt_by are history (kept, shown as a former analyst).

---

6. The window

assets/js/help-assistant.js, assets/css/help-assistant.css, injected on every analyst page by renderHeaderRight() → helpAssistantWidgetRender().

• Draggable on desktop, a bottom sheet on a phone. Position remembered in the browser.
• The conversation is stored on the server per analyst (help_assistant_messages, last 200 kept), so it survives moving between pages and devices, and goes only when Reset is clicked.
• Clicking a result on the same page scrolls the right panel to the section itself; on another page, it stores the target and that page scrolls once loaded.

TRAP: the browser's own jump to #anchor scrolls the whole document, pushing the fixed top bar off the screen (Ed spotted this in the first hour). reveal() puts the document back to the top and scrolls the panel that holds the section, 20 px below its top.

TRAP: a PHP patch script turned \n inside a JavaScript regex into a real line break - a syntax error that stopped help-assistant.js loading, so the whole window vanished. Every PHP test passed. A headless Chrome load of the real page caught it. Any JS edited by a script gets a browser check.

---

7. The trap list

Every trap found building this, in one place (details in the part named):

Trap	Part
Reading help files as text misses everything PHP adds - render them	1
A rendered page must be one process each; I18n::pinLocale() is CLI-only	1
A background worker on Windows must never print - the closed pipe kills it	1
Whole sections bury a card in a long section - score passages	2
A stemmer's rules must chain	2
Every question has a nearest vector - use a floor	2
Embed the question with the model in use, not the one chosen	2
Help quoting the test questions flatters the measurement	2
The end of the user message beats the system rules for format	3
A keen model searches every round and returns no text - one final call without tools	3
A model sometimes writes a tool call as text - keep what follows	3
A non-English question with no word matches must still go to the AI	3
Help missing for the asker is not help missing - the wide check	3
The browser's anchor jump hides the top bar	3
\n in JS from a PHP patch - browser-check	3
An English example in a rule is copied into other languages (Tickets → Settings → Abteilungen)	3
NOT_COVERED can come last, after narration - check every line	3
kind VARCHAR(10) truncated not_covered - widened to 20, DB Verify fixes old installs	3
Readiness must check every table, or code that arrives before DB Verify says "Something went wrong"	3
Tests written before Ask existed started calling the real AI on an install with it on - they force it off	3

---

8. Extending it

A new help page - nothing to do beyond the house style (and writing help it answers well): sections with ids and an h3, data-cap on settings sections. The out-of-date warning appears; Rebuild indexes it. Add a question for it to tests/help-search-questions.php.

A new embedding provider - add it to aiEmbeddingPayload(), aiEmbeddingUrl(), aiEmbeddingHeaders() and aiEmbeddingModels() in includes/ai_embeddings.php (see Embeddings developer guide). Knowledge and the help assistant both get it. Never a fifth copy of the embedding call - tests/ai-embeddings.php guards against it.

A new chat provider - nothing: help_ai uses the shared AI panel and request code.

Tuning a threshold - never by feel. Change the constant, rerun the question set (php tests/help-search.php for words; help-vectors.php for the merge, from cached vectors at no cost), and keep it only if the numbers say so. Record the before and after in the developer guide.

Adding a question to the set - a realistic wording a real person would type, every section that is a right answer, and its kind. Do not reuse a question's wording in help text (part 2's trap).

Shipping vectors - php scripts/help_vectors_export.php at every release (RELEASING step 3b).

---

See also: Every design question, answered · Help assistant developer guide · Design · User page