Help Assistant Internals 1 The Index

🧠 Help assistant internals, part 1: from help pages to an index

Part 1 of 3. Part 2: answering a question - the hybrid search · Part 3: writing the answer, and learning from misses · Every design question, answered

The help assistant answers "how do I...?" from FreeITSM's own in-app help. Everything it does rests on one thing built in advance: an index of every help section, in every language analysts use, with a vector for each. This page is how that index is made, kept fresh and shipped.

For what the feature does for users, see The help assistant. For the decisions behind it, the design. The step-by-step build notes are in the Developer Guide; these three pages explain how the whole thing fits together.

---

The shape of the whole thing

flowchart LR
    subgraph Built in advance - part 1
        P[75 help pages] -->|rendered, one process each| S[help_sections<br/>722 English + translations]
        S -->|embedding model| V[help_section_vectors<br/>one per section per model]
        R[Release file<br/>ready-made vectors] -.->|taken when the text matches| V
    end
    subgraph Every question - parts 2 and 3
        Q[Question] --> W[Word search<br/>BM25 over passages]
        Q --> M[Meaning<br/>question vector vs section vectors]
        W --> X[Merge<br/>reciprocal rank fusion]
        M --> X
        X --> A{Chat AI set up?}
        A -->|no| C[Canned line + help cards]
        A -->|yes| AI[AI writes the answer<br/>from those sections only]
    end
    S --> W
    V --> M

Loading

Everything on the left is a cache. The help pages and language files are the truth; delete the two tables and the next build makes them again. Nothing anyone writes is ever stored there.

---

1. Why the help pages are rendered, not read

The obvious way to index help is to read the t('...') keys out of each page's source. It does not work here:

• Many pages build their sections in loops through helpers - people/help.php draws ten sections from $h($id . '.title'), so the keys never appear in the source.
• The 35 System help pages are written in English in the page itself, with no keys at all.
• A key that no page shows would be indexed anyway, and could be "answered" with a link to nowhere.

So every page is rendered - executed exactly as a browser request would - and the sections are read from the HTML. That gives precisely what a reader sees, in their language, and nothing else.

One process per page

// includes/help_index.php - helpIndexRenderPage()
$cmd = escapeshellarg($php) . ' ' . escapeshellarg("$root/scripts/help_render_page.php") . ' '
     . escapeshellarg($page) . ' ' . escapeshellarg($locale) . ' ' . (int)$analystId;
exec($cmd . ' 2>NUL', $out, $code);   // the last line of output is JSON

Each render is its own PHP process because a help page is an ordinary page: it defines functions and constants, calls session_start() and may exit(). Two in one process would collide or never return. scripts/help_render_page.php:

I18n::pinLocale((string)$locale);                      // command line only - the page's own
                                                       // initFromSession() cannot change it
session_id('helprender' . bin2hex(random_bytes(8)));   // a throwaway session...
session_start();
$_SESSION['analyst_id'] = (int)$analystId;             // ...as an admin with every module, so
                                                       // each page's own module gate lets it in
parse_str($query, $_GET);                              // system/integrations/help.php?provider=jira
ob_start();
register_shutdown_function(function () {               // runs even if the page exit()s
    $html = '';
    while (ob_get_level() > 0) $html = ob_get_clean() . $html;
    @session_destroy();
    echo "\n" . json_encode(['ok' => true, 'sections' => helpRenderSections($html)]) . "\n";
});
chdir(dirname("$root/$page"));
require "$root/$page";

Three details that each cost a debugging session:

• I18n::pinLocale() exists only for this. A page calls I18n::initFromSession(), which reads the signed-in analyst's saved language; the pin makes that call leave the locale alone. It does nothing outside the command line.
• The page's own session_start() is ignored, because the renderer's session is already running - and PHP logs a notice per render (74 per language per build). The renderer silences exactly that notice.
• The analyst is only a key. Rendering writes nothing for them; help pages do not touch the Recent trail.

Which pages

helpIndexPages() finds every */help*.php and */*/help*.php that calls requireModuleAccess('…') (which also tells it the page's module), every system/help/*.php (module system), and a short HELP_INDEX_EXTRA list for what a folder scan cannot find - the integrations guide is one page per ?provider=. The self-service portal's guide and Help Centre are left out on purpose: the assistant is for analysts. Today that is 75 pages.

---

2. What a section is

The markup every help page already used is the contract (house style §7):

<div class="help-section" id="settings" data-cap="tickets.manage">
    <div class="help-section-header">
        <span class="help-section-num">10</span>
        <div><h3>Settings</h3><p>...</p></div>
    </div>
    <div class="help-card"><strong>Departments</strong><span>Create departments like IT, HR...</span></div>
</div>

helpRenderSections() (DOMDocument + XPath) turns each into:

Field	From
id	the id - the anchor an answer links to
title	the first <h3> (the number badge is dropped)
text	blocks, one per paragraph, card, step, note, list item or table row, joined by a blank line
cap	data-cap - the permission the section describes (part 3)
skip	data-help-search="off" - never indexed

Plus one extra entry per page for its introduction - the hero banner and any .help-lede above the first section - with no anchor, linking to the top of the page. It is often the best answer to "what is X?".

Why blocks, not one string: search scores passages of about 700 characters (part 2), and a passage must never cut a card in half. Keeping the blank-line boundaries lets helpPassages() pack whole blocks.

TRAP: textContent glues a card's title to its body - "DepartmentsCreate departments". Each block's text is joined from its text nodes with spaces (helpRenderText()).

---

3. help_sections - the index

CREATE TABLE help_sections (
    section_key   VARCHAR(191) NOT NULL,   -- 'tickets/help.php#settings'; a page intro is just the page
    locale        VARCHAR(10)  NOT NULL,   -- 'en', 'de', 'pt-BR'
    page, anchor, module,                  -- module drives the access filter
    cap           VARCHAR(80) NULL,        -- data-cap
    title, body,                           -- body keeps the blank-line block boundaries
    text_hash     CHAR(64) NOT NULL,       -- sha256(title + body): what it says now
    built_datetime DATETIME NOT NULL,
    UNIQUE KEY (section_key, locale)
);

helpIndexBuild($conn, $locales) renders every page in English, then in each extra language, and for each section:

Section	What happens
text unchanged (text_hash matches)	nothing - its row, and its vectors, stay
text changed	updated (its vectors become out of date - below)
gone from its page	deleted, its vectors with it (foreign key cascade)
another language, text identical to the English	not stored - an untranslated section would be a duplicate; English covers it
module switched off for the install	skipped, rows left alone - nobody can open it, so the search never returns them

An unchanged rebuild changes nothing. English takes about 25 seconds; each further language another 25.

Which languages

helpIndexLocalesInUse(): English, the install's default, and every language an analyst has chosen in Preferences. An index in a language nobody reads is wasted work.

The per-section language fallback

When a German analyst asks, helpSectionsFor() gives them, for each section, its German row if there is one, otherwise its English row:

$st = $conn->prepare("SELECT id, section_key, locale, ... FROM help_sections WHERE locale IN ('en', ?)");
$st->execute([$locale]);
foreach ($st->fetchAll(PDO::FETCH_ASSOC) as $r) {
    if (!isset($by[$r['section_key']]) || $r['locale'] === $locale) $by[$r['section_key']] = $r;
}

Per section, not per language: on an install where German is 22 % translated, a German analyst still searches 100 % of the help - 22 % of it in German.

Then modules the analyst cannot open are dropped (analystCanAccessModule(), which also says no to a module switched off for the install). On prtest that leaves an administrator 722 English sections; the German index adds 329 German ones.

---

4. Vectors - help_section_vectors

A vector is a list of numbers an embedding model produces from text, so that texts meaning similar things get similar lists. Part 2 explains how they are compared; here is how they are made and stored.

CREATE TABLE help_section_vectors (
    section_id  INT NOT NULL,            -- help_sections.id, ON DELETE CASCADE
    model       VARCHAR(120) NOT NULL,   -- 'openai:text-embedding-3-small@512'
    text_hash   CHAR(64) NOT NULL,       -- the section's text_hash when this was made
    vector      MEDIUMTEXT NOT NULL,     -- float32, base64
    source      VARCHAR(20) NOT NULL DEFAULT 'built',   -- or 'release'
    source_version VARCHAR(20) NULL,
    UNIQUE KEY (section_id, model)
);

One per section per model

🔴 Vectors from different models mean nothing to each other. A question must be embedded by exactly the model its candidate sections were.

So the table holds vectors for each model separately, and the model's identity travels with every vector:

// includes/ai_embeddings.php
function aiEmbeddingModelId(array $cfg): string
{
    $family = ($cfg['provider'] ?? 'openai') === 'openai_compatible' ? 'compatible' : 'openai';
    $id = $family . ':' . trim((string)($cfg['model'] ?? ''));
    return !empty($cfg['dimensions']) ? $id . '@' . (int)$cfg['dimensions'] : $id;
}

OpenAI and Azure OpenAI running the same model make the same vectors, so they share an identity (and the ready-made vectors suit both); a local server's models are compatible:<name>. The length is part of it: @512 vectors cannot be compared with full-length ones.

512 numbers, packed

OpenAI's text-embedding-3 models can return shorter vectors on request. Measured on the question set, 512 dimensions ranked exactly as well as the full 1,536 - and compare three times faster (16 ms against 50 ms over 720 sections in PHP). They are stored as float32, base64 - about a third of the size of JSON:

function aiVectorPack(array $v): string   { return base64_encode(pack('g*', ...$v)); }
function aiVectorUnpack(string $p): array  { return array_values(unpack('g*', base64_decode($p, true))); }

Building them

helpVectorsBuild() - run by the same build, after the sections:

1. If the configured model is the shipped one, take the ready-made vectors for every section whose text still matches (free - below).
2. Embed every section with no current vector (none for this model, or a text_hash that no longer matches), 64 to a request - aiEmbedBatch() sends many texts at once, so 720 sections are a dozen calls, not 720.
3. Only when none is missing, make this model the one in use and delete every other model's vectors.

That third step is what lets an admin change model without the assistant going dark - the old model keeps answering until the new set is complete. Part 2 shows the trap that made it necessary.

Out of date

A vector whose text_hash is not its section's current text_hash was made from old text. System → Help assistant counts them ("12 sections have changed since their vectors were made") and Rebuild re-embeds just those. The cheap first check - has anything changed at all? - is file dates: a help page or language file newer than the last build. The exact check is the hash, because dates lie both ways (a git pull re-dates unchanged files; a restore back-dates changed ones).

---

5. Ready-made vectors in each release

The help text is identical on every install of a version, so its vectors would be too. They ship:

data/help-vectors/openai-text-embedding-3-small-512.json.gz   (720 English sections, about 1 MB)
{ "model": "openai:text-embedding-3-small@512", "version": "5.0.0",
  "vectors": { "tickets/help.php#settings|en": ["<text_hash>", "<packed vector>"], ... } }

helpVectorsImportShipped() takes a vector only where the install's section still hashes the same - a section an install changed (an edited language file) is embedded there instead. On prtest a fresh build took 720 ready-made and embedded only the 659 German and Spanish sections.

Each release refreshes the file with php scripts/help_vectors_export.php X.Y.Z (RELEASING step 3b). Skipping it costs nothing but money on each install.

What shipping cannot save: the question still needs the embedding model, because it only exists when someone asks. An install with no embedding key never uses vectors - shipped or not - and searches by words.

---

6. Rebuilding from the web page

System → Help assistant → Rebuild spawns scripts/help_index_build.php in the background with the Intune worker's intuneSpawnGenericWorker() (reused, not copied), and the page polls help_index_build_state (running / done / failed, with stats). Saving the assistant on for the first time, or matching by meaning on for the first time, starts a build by itself.

TRAP: a background worker must not print. intuneSpawnGenericWorker() closes the worker's output straight away; on Windows the first write to that closed pipe killed the build after one page, leaving it "running" for ever. Started with the job id 0, the builder prints nothing and reports through the setting.

---

Next: Part 2 - answering a question: the hybrid search