Skip to content

Knowledge Migration API Developer Guide

Ed Mozley edited this page Nov 19, 2026 · 1 revision

πŸ› οΈ Knowledge migration API - Developer Guide

(PR #378 by @Dr4inder, #377, 5.7.0, #3013-#3015.) For someone changing the Knowledge REST endpoints, the folder or permissions screens, or the protected picture route.

User side: REST API: Knowledge - Migrating a knowledge base in. The review and its fixes: PR #378 write-up.

πŸšͺ Front door

Four ideas carry the whole thing:

  1. One write path. The REST endpoints and the folder and permissions screens go through KnowledgeService and KnowledgeTransferService, so a rule fixed in one place is fixed for both.
  2. Content and access commit together. An article created with an access list is never visible without it, even for a moment.
  3. A create can be replayed. client_reference + knowledge_api_references make "send it again after a timeout" safe.
  4. A picture is part of its article. Every read re-runs knowledgeCanRead() for that article and that reader; there is no other permission on a picture, and nobody without a session gets one.

πŸ“ Files

File What it does
🟣 includes/services/knowledge_transfer.php Actor checks, folders, setAccess(), revision(), createOnce(), transactions
🟣 includes/services/knowledge_images.php Upload, load() (the permission check), serve(), delete and purge
🟣 includes/services/knowledge.php saveArticle() learns folder_id, is_published on update, access, expected_revision, required_image_ids, processing; finishSave() runs search / AI / workflows after commit
βšͺ includes/knowledge/processing.php KnowledgeProcessing::suppresses() - reads processing
🟣 api/v1/resources/knowledge_transfer.php Folder, access and image endpoints
🟑 api/v1/resources/knowledge.php Article create/update use createOnce(), report processing, return revision
βšͺ api/v1/lib/routes.php, permissions.php, modules.php Nine routes, four permission groups, mapped to the Knowledge module
🟣 api/knowledge/image.php The picture URL that goes in article HTML, for analyst and portal sessions
🟑 api/knowledge/folders.php, permissions.php The screens, now thin adapters over the services
🟑 api/knowledge/knowledge_save.php The editor's save - an allow-list of the keys the screen sends
βšͺ uploads/knowledge/.htaccess, web.config Deny direct access (nginx: the shipped /uploads/ deny)
βšͺ includes/db_verify_schema.php, database/freeitsm.sql, includes/db_verify_indexes.php The two tables
🟒 tests/knowledge-migration-api.php 52 checks over HTTP, concurrent workers included
🟒 tests/knowledge-pr378-review-fixes.php 15 checks for the review fixes and Publish

🧱 The tables

CREATE TABLE knowledge_api_references (
    id INT NOT NULL AUTO_INCREMENT,
    analyst_id INT NOT NULL,                 -- the API key's analyst
    operation VARCHAR(16) NOT NULL,          -- 'folder', 'article', 'image'
    parent_id INT NOT NULL DEFAULT 0,        -- e.g. the article an image belongs to
    client_reference VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
    request_hash CHAR(64) NOT NULL,          -- sha256 of the canonical request
    object_id INT NOT NULL,
    created_datetime DATETIME NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    UNIQUE KEY uq_knowledge_reference (analyst_id, operation, parent_id, client_reference)
);

CREATE TABLE knowledge_images (
    id INT NOT NULL AUTO_INCREMENT,
    article_id INT NOT NULL,
    stored_name VARCHAR(64) NOT NULL,        -- 32 random hex + extension, nothing from the client
    original_name VARCHAR(255) NOT NULL,
    mime_type VARCHAR(100) NOT NULL,
    size_bytes INT UNSIGNED NOT NULL,
    sha256 CHAR(64) NOT NULL,
    created_by_id INT NULL,
    created_datetime DATETIME NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    KEY idx_knowledge_images_article (article_id)
);

client_reference is binary-collated, so ABC and abc are different references. Both analyst_id and created_by_id are in analystReferenceRegistry().

πŸ” createOnce() - a create you can send twice

$lock = 'kb:' . substr(hash('sha256', "$db:$ctx->actorId:$operation:$parent:$ref"), 0, 60);
// SELECT GET_LOCK(?, 10)  - else 409 reference_busy
...
$st = $conn->prepare('SELECT object_id, request_hash FROM knowledge_api_references
    WHERE analyst_id = ? AND operation = ? AND parent_id = ? AND client_reference = ? FOR UPDATE');
if ($prior = $st->fetch(PDO::FETCH_ASSOC)) {
    if (!hash_equals($prior['request_hash'], $hash)) throw new ServiceError('conflict', 'reference_conflict', ...);
    return ['id' => (int)$prior['object_id'], 'replayed' => true];
}
$result = $write();
// INSERT INTO knowledge_api_references ...
  • GET_LOCK serializes two workers sending the same reference at once; the row lock covers the rest.
  • A reference outlives its object. Delete the article and the reference stays, pointing at an id that no longer exists, so a stale retry resolves to a 404 instead of creating it again.
  • The caller re-authorizes the replayed id (apiLoadArticle(), object(), load()), so a replay never hands back something the key can no longer read.
  • A replay does not re-run search, AI or workflows: the response says not_repeated.

πŸ”’ setAccess() - the whole list, atomically

The REST PATCH .../access replaces the list; the screen's add, remove and mode actions read the current list, change it, and send the whole thing to the same function. Inside one transaction it updates is_restricted / inherit_permissions, keeps unchanged rows (their ids and authors - an open dialog removes by row id), deletes what is gone and inserts what is new.

TRAP: only a new entry has to name someone who exists. An entry already stored may point at a person, team or group deleted since; checking those too locked the list, because the screen sends every entry back on each change. Found in review, fixed before merge - see the write-up.

Gate: the key's knowledge_access.manage and the analyst's Cap::KNOWLEDGE_MANAGE (requireActor($conn, $ctx, true, true)).

βœ‹ revision() - optimistic locking

The row (minus view_count and the embedding columns, which change without an edit), its access list, its tag ids and its pictures' ids and hashes, JSON-encoded and HMAC'd with a key derived from the encryption key:

$key = hash_hmac('sha256', 'freeitsm/knowledge-revision/v1', getEncryptionKey(), true);
return hash_hmac('sha256', json_encode($row, JSON_THROW_ON_ERROR), $key);

Keyed because the input includes the admin-only access list. checkRevision() compares with hash_equals() and throws 409 revision_conflict. Every write locks the row (FOR UPDATE) before reading it, so the check and the write see the same row - even two saves in the same second.

πŸ–ΌοΈ Pictures

Upload goes through uploadStoreBytes() with UPLOAD_TYPES_IMAGE (magic bytes checked, SVG excluded, policy DROP) and a 10 MiB cap; the file gets a random name, and if the database insert fails the file is removed.

Read - the only permission check is the article's:

public static function load(PDO $conn, KnowledgeViewer $viewer, int $id, string $lifecycle = 'live'): array
{
    ...
    if (!$row || !knowledgeCanRead($conn, $viewer, (int)$row['article_id'], ['lifecycle' => $lifecycle])) {
        throw new ServiceError('not_found', 'not_found', 'Image not found.');
    }

image.php picks the viewer from the session - an analyst (lifecycle 'any', so a draft's pictures show in the editor), a portal user ('live' only) - and anybody else gets 404. Ed, 19 November 2026: no content of any sort without authentication.

Serve sends Content-Type from the allow-list (never the stored mime), nosniff, Content-Disposition: inline, Cache-Control: private, no-store and Content-Security-Policy: default-src 'none'; sandbox. No caching on purpose: revoking someone's access to an article must revoke its pictures on their next view, not when a cache expires.

stored_name is re-checked against ^[0-9a-f]{32}\.(png|jpe?g|gif|webp|bmp|heic|heif)$ before it is joined to a path.

πŸ“ Drafts, processing and the editor

  • is_published: false on a create makes a draft; is_published on an update is new in 5.7.0 and is what the screen's Publish button sends. Flipping it to true raises knowledge.published rather than knowledge.updated.
  • processing: "suppress" skips the AI embedding and workflows; the search index still runs after commit. REST requires knowledge_imports.manage.
  • saveArticle() only runs finishSave() when it owns the transaction. A caller already inside a transaction gets no post-commit processing and must call finishSave() itself after its own commit.

TRAP: api/knowledge/knowledge_save.php passes on an allow-list of the keys the screen sends. Passing the whole body would hand every analyst the REST-only keys above with none of their permissions. When the screen needs a new field, add it to the list - deliberately.

Adding to it

  • A new create endpoint that should be retry-safe: wrap the write in createOnce() with its own operation name, and re-authorize the replayed id in the resource.
  • A new reader of pictures: build the right KnowledgeViewer and call KnowledgeImageService::load(); never read knowledge_images and serve the file yourself.
  • A new screen field on articles: add it to the allow-list in knowledge_save.php.

Related: Knowledge folders and security - Developer Guide Β· REST API

FreeITSM

Getting Started

Modules

Multi-tenancy (planned)

Responses to feedback

Blue sky thinking

Bugs resolved

Pull requests

Links

Clone this wiki locally