Embeddings Developer Guide

🛠️ Embeddings - Developer Guide

One file turns text into a vector and compares two: includes/ai_embeddings.php. Every feature that searches by meaning goes through it - the Knowledge module's article embeddings, its AI chat, the web chat's answers, gap analysis, and (when it is built) the help assistant.

Until November 2026 there were four copies of the OpenAI call and two of the comparison. They were merged as step 1 of the help assistant (#2769), and merging them found a bug all four shared (#2770, below).

---

The shape of it

File	Uses
includes/ai_embeddings.php	The only code that calls the embeddings API or compares vectors
api/knowledge/generate_embedding.php	aiEmbedRequest() - embeds one article and reports why it failed
includes/services/knowledge.php	aiEmbed() - re-embeds an article when it is saved (a bonus, never a blocker)
includes/knowledge/kb_ai.php	aiEmbed() + aiCosineSimilarity() - finds articles for the web chat and Knowledge AI answers
api/knowledge/ai_chat.php	aiEmbed() + aiCosineSimilarity() - the Knowledge AI chat
includes/knowledge/gap_analysis.php	aiEmbed() + aiCosineSimilarity() - embeds closed tickets and clusters them

The file knows nothing about articles, tickets or help. Callers bring their own text, key and storage.

---

Providers (November 2026)

aiEmbedBatch($cfg, $texts) talks to OpenAI, Azure OpenAI (a deployment URL with an api-key header, or the newer v1 API with the model in the body) and any OpenAI-compatible server (Ollama, LM Studio, vLLM), many texts per request. aiEmbeddingUrl(), aiEmbeddingHeaders() and aiEmbeddingPayload() build the request and are tested without a network; aiEmbeddingModels() lists a provider's models for a dropdown; aiEmbeddingModelId() is the identity to store beside a vector (OpenAI and Azure on the same model share one). The Knowledge module still calls aiEmbed() / aiEmbedRequest() - OpenAI, its own key - unchanged. The help assistant is the first user of the rest: section 8 of its developer guide.

The functions

const AI_EMBEDDING_MODEL     = 'text-embedding-3-small';   // every stored vector so far
const AI_EMBEDDING_MAX_BYTES = 30000;                      // ~8k tokens at ~4 bytes a token

function aiEmbedRequest(string $text, string $apiKey): array   // ['embedding' => ?array, 'error' => ?string]
function aiEmbed(string $text, string $apiKey): ?array         // the vector, or null - when the reason does not matter
function aiEmbeddingRequestBody(string $text): ?string         // the JSON sent - testable without a network
function aiEmbeddingText(string $title, string $html): string  // title, blank line, body as plain text
function aiCosineSimilarity(array $a, array $b): float         // 1 same direction, 0 unrelated

aiEmbedRequest() - when the caller reports failures

generate_embedding.php shows the reason to the admin who clicked Generate, so it needs it:

$result = aiEmbedRequest(aiEmbeddingText((string)$article['title'], (string)$article['body']), $apiKey);
if ($result['error'] !== null) {
    echo json_encode(['success' => false, 'error' => $result['error']]);
    exit;
}
$embedding = $result['embedding'];

The reasons are the wording that endpoint always used: Connection error: ... (cURL), OpenAI API error: <OpenAI's message> (any non-200, falling back to API returned HTTP <code>), Invalid embedding response.

aiEmbed() - when an embedding is a bonus

Saving an article through KnowledgeService re-embeds it, but a failure must never stop the save:

$embedding = aiEmbed(aiEmbeddingText($title, $body), $apiKeyValue);
if ($embedding) {
    $conn->prepare("UPDATE knowledge_articles SET embedding = ?, embedding_updated = UTC_TIMESTAMP() WHERE id = ?")
         ->execute([json_encode($embedding), $articleId]);
    return true;
}

aiEmbeddingText() - what an article is embedded as

Title, a blank line, then the body with tags stripped, entities decoded and whitespace collapsed - exactly what the old copies built:

aiEmbeddingText('Reset a password', '<p>Open <b>People</b> &amp; choose <i>Reset</i>.</p>');
// "Reset a password\n\nOpen People & choose Reset."

TRAP: no /u on the whitespace regex. With /u, preg_replace() returns null on invalid UTF-8 - and the whole body would vanish from the embedding. Without it, a stray byte is just a byte.

aiCosineSimilarity() - comparing two vectors

$$\cos\theta = \frac{a \cdot b}{\lVert a\rVert,\lVert b\rVert}$$

$dot = 0.0; $na = 0.0; $nb = 0.0;
$len = min(count($a), count($b));
for ($i = 0; $i < $len; $i++) {
    $dot += $a[$i] * $b[$i];
    $na  += $a[$i] * $a[$i];
    $nb  += $b[$i] * $b[$i];
}
if ($na == 0.0 || $nb == 0.0) return 0.0;   // an all-zero vector matches nothing - never a division by zero
return $dot / (sqrt($na) * sqrt($nb));

Vectors are stored as JSON text (knowledge_articles.embedding, the gap-analysis ticket table) and decoded before comparing. Comparison is done in PHP over every candidate - fine at knowledge-base scale (hundreds to low thousands of rows).

Vectors from different models cannot be compared. Every stored vector so far was made with text-embedding-3-small. A second provider (Azure OpenAI, a local model) is a change in this one file - and switching model means re-embedding what is stored.

---

#2770: long articles in accented or non-Latin text never got an embedding

What you saw

Nothing - which was the problem. A long article (over about 30,000 bytes) written largely in accented or non-Latin text - Polish, Czech, Greek, Cyrillic, Hindi, Tamil - had no embedding. The Knowledge assistant, the web chat's answers and gap analysis could not find it by meaning; Generate reported an OpenAI error about the request rather than anything pointing at the article.

What was wrong

All four copies cut the text at the byte limit the same way:

if (strlen($text) > 30000) {
    $text = substr($text, 0, 30000);   // counts BYTES
}
$body = json_encode(['model' => 'text-embedding-3-small', 'input' => $text]);

é is two bytes, Devanagari three. Cutting at byte 30,000 can land in the middle of a character, leaving half of it at the end of the string. That is invalid UTF-8, and json_encode() refuses invalid UTF-8 by returning false - so the request went out with the body false, OpenAI rejected it, and the article was left without a vector. Short articles, and long ones in plain English (one byte a character), were never affected.

The fix

function aiEmbeddingRequestBody(string $text): ?string
{
    if (strlen($text) > AI_EMBEDDING_MAX_BYTES) {
        $text = mb_strcut($text, 0, AI_EMBEDDING_MAX_BYTES, 'UTF-8');   // a byte limit, cut on a character boundary
    }
    $body = json_encode(['model' => AI_EMBEDDING_MODEL, 'input' => $text], JSON_INVALID_UTF8_SUBSTITUTE);
    return $body === false ? null : $body;
}

mb_strcut() keeps the byte limit but backs off to the last whole character. JSON_INVALID_UTF8_SUBSTITUTE is a second guard: a bad byte already in the text becomes U+FFFD instead of sinking the request.

TRAP: never substr() text to a byte limit and then json_encode() it. Use mb_strcut(). The failure only shows on long non-English text, so English testing never finds it.

What it means for you: articles affected before the fix get their embedding the next time they are saved, or with Knowledge → Settings → Generate for all of them.

---

Tests

tests/ai-embeddings.php - 13 checks, no database, no network:

• the comparison: same direction 1, opposite -1, at right angles 0, an all-zero vector 0, unequal lengths compared on what they share
• aiEmbeddingText(): title then plain text; an invalid byte does not drop the body
• the request body: 20,000 é (40,000 bytes) encodes, is cut to between 29,997 and 30,000 bytes and is valid UTF-8 - with the control that the old substr() cut makes json_encode() return false
• a guard: no PHP file outside includes/ai_embeddings.php (and tests) contains v1/embeddings or defines a *cosine* function, so a fifth copy cannot creep back in. Control: the code before #2769 has four such files.

php tests/ai-embeddings.php

See also Developer Tests - Knowledge.

---

Related

• AI Providers - the chat side: provider, model and key per feature
• Knowledge Assistant - Developer Guide - the main user of embeddings
• The help assistant - design - the next one