Skip to content

API Uploads and Knowledge Documents Design

Ed Mozley edited this page Nov 21, 2026 · 2 revisions

Large files through the REST API, and documents on Knowledge articles - design

Status: design agreed by Ed, 21 November 2026 (decisions); waiting for @Dr4inder's view on #416; to be built in 5.10.0. Issues: #417 (picture size) and #416 (article documents), both from @Dr4inder. #417's first part - the size setting, GET /knowledge/images/constraints and 413 image_too_large - is already built for 5.9.0 (REST API - Knowledge); this page is what comes after it.

In one paragraph: files up to now reach the API in one request - a picture as base64 inside JSON, a project document as a multipart form. Both hit limits the integration cannot see or change: PHP's upload_max_filesize (2 MB by default), post_max_size (8 MB by default), PHP's memory, and any proxy in front (nginx refuses anything over 1 MB by default). This design adds one upload API for every file: start an upload, send it in pieces, then hand it to the record that wants it. Pieces fit under every one of those limits, a dropped connection resumes instead of starting again, and the server holds a piece at a time instead of the whole file. Article documents (#416) are the first new thing built on it.

Contents

  1. Why chunks
  2. The upload API
  3. Handing an upload to a record
  4. Documents on Knowledge articles (#416)
  5. Security and housekeeping
  6. What changes for existing clients
  7. Decisions
  8. How it will be tested

1. Why chunks

base64 JSON (pictures today) multipart form (project documents today) chunks (this design)
PHP upload_max_filesize not applied hard limit - 2 MB by default not applied
PHP post_max_size read anyway, but PHP warns on every one hard limit - 8 MB by default each piece is far below it
A proxy body limit (nginx: 1 MB) refused refused pieces sized to fit
Memory on the server ~5 x the file ~1 x one piece
Bytes on the wire +33% the file the file
Connection drops at 90% start again start again carry on from 90%
Calls 1 1 3 or more

FreeITSM already works this way in one place: the Files module sends a 2 GB file from the browser in 4 MB pieces (api/files/upload.php). This design gives the REST API the same mechanism, and the big platforms use the same shape - Microsoft Graph upload sessions, Google resumable uploads, S3 multipart, tus.

The one-call routes stay. Chunks are for files that are big, or servers whose limits are small; a 200 KB screenshot can still go up in one request.

2. The upload API

POST   /api/v1/uploads                    start
PUT    /api/v1/uploads/{upload_id}?offset=N   one piece (raw bytes)
GET    /api/v1/uploads/{upload_id}        how much has arrived - to resume
DELETE /api/v1/uploads/{upload_id}        give up
GET    /api/v1/uploads/limits             every purpose's size and type limits

Start

POST /api/v1/uploads
{ "purpose": "knowledge_document", "filename": "Network runbook.pdf", "size": 48318921,
  "sha256": "9f2c…" }
201 { "data": { "upload_id": "u_5e1f0c…", "purpose": "knowledge_document", "size": 48318921,
  "received": 0, "chunk_bytes": 4194304, "expires_at": "2026-11-22T14:05:00Z" } }
  • purpose says what the file is for: knowledge_image or knowledge_document to begin with (later ticket_attachment, project_document …). The purpose decides who may start (the key needs that purpose's create permission - no new permission to grant), how big (the purpose's own limit, checked now, before a byte is sent) and which types (the extension is checked now; the content again at the end).
  • sha256 is optional. When given, the assembled file must match it or the upload is refused.
  • chunk_bytes is the largest piece this server takes: 4 MB, or less where PHP's post_max_size is smaller. A client behind a stricter proxy simply sends smaller pieces - any size up to chunk_bytes is fine.

Send the pieces

PUT /api/v1/uploads/u_5e1f0c…?offset=0
Content-Type: application/octet-stream

<4194304 raw bytes>
200 { "data": { "upload_id": "u_5e1f0c…", "received": 4194304, "size": 48318921, "complete": false } }
  • Raw bytes, not multipart and not base64 - upload_max_filesize never applies, nothing is encoded, and the piece is streamed straight to disk.
  • In order. offset must equal what has arrived so far. Anything else is 409 offset_mismatch with received in the reply - so a client that lost track just asks, and carries on from there. A piece is never written twice.
  • The last piece brings received to size and the reply says "complete": true. Too many bytes is refused.

Resume, give up

GET /uploads/{upload_id} returns received, size and expires_at - after a dropped connection, send from received. DELETE throws the pieces away. An upload not finished within 24 hours of its last piece expires and its pieces are deleted.

Limits, without hard-coding them

GET /api/v1/uploads/limits
{ "data": { "chunk_bytes": 4194304, "purposes": {
    "knowledge_image":    { "max_bytes": 10485760, "formats": [ { "extension": "png", "mime_types": ["image/png"] }, "…" ] },
    "knowledge_document": { "max_bytes": 10485760, "formats": [ "… the install's attachment types …" ] } } } }

GET /knowledge/images/constraints (5.9.0) stays, and reports the same picture limits.

3. Handing an upload to a record

A finished upload is not a document or a picture yet - it is bytes waiting. The record that wants it takes it by id, in the same call that would otherwise carry the file:

POST /api/v1/knowledge/articles/812/images
{ "upload_id": "u_5e1f0c…", "filename": "diagram.png", "client_reference": "old-kb:img:881" }

At that moment, and only then, everything the one-call route checks is checked: the article exists and the key may write to it, the purpose matches, the size is within the limit now (an administrator may have lowered it meanwhile), the content really is an allowed type (magic bytes; never SVG), the SHA-256 if one was given. The file then moves into its store - the same random name, the same protected folder, the same audit - and the upload is used: it cannot be handed to a second record.

Retries are safe. An upload remembers what it became for 24 hours, so sending the same upload_id and client_reference again returns the same picture or document (HTTP 200, replayed: true) instead of an error. A different client_reference with a used upload is 409 upload_used. And the existing rule still holds across new uploads: the same client_reference with the same bytes is a replay; with different bytes it is 409 reference_conflict.

4. Documents on Knowledge articles (#416)

Article documents - the PDFs and Office files in an article's Documents panel, and the ones customers can download from the Help Centre - are not in the API today. They will be, reusing DocumentsService, the code behind the Documents panel and the project-documents API (#287), so the app and the API cannot drift.

Route Permission Does
GET /knowledge/articles/{id}/documents knowledge_documents.read Paged list, metadata only: id, kind (file / link), title, description, filename, MIME type, size, sha256, external URL, show_in_help_centre, client_reference, uploaded by and at, url (below) and content_url. Every document on the article, whether or not the article's HTML links to it. ?client_reference= finds an earlier upload without sending it again.
POST /knowledge/articles/{id}/documents knowledge_documents.create A file - upload_id (any size), or a multipart file as projects take it (small files) - or a link, external_url. With title, description, client_reference, show_in_help_centre. 201; 200 for a replay; 409 for a reused reference.
POST /knowledge/articles/{id}/documents/{document_id}/link knowledge_documents.create Attach a document that already exists - the key's analyst must already be able to see it.
PATCH /knowledge/articles/{id}/documents/{document_id} knowledge_documents.create title, description, show_in_help_centre.
GET /knowledge/articles/{id}/documents/{document_id}/content knowledge_documents.read and documents.download The bytes, through the download guard - counted, alerted, logged with the key - exactly as project documents. A link returns its URL as JSON.
DELETE /knowledge/articles/{id}/documents/{document_id} knowledge_documents.delete Removes this article's link. A document still attached elsewhere stays; the last link soft-deletes it, as in the app.

Access is the article's. A document on an article inherits every rule the article has - company, audience, access list, and whether it is published - through includes/knowledge/visibility.php, as it already does in the app. An article the key cannot reach is 404, and a document that is not on that article is 404: no document is ever served by id alone, and knowing a document's URL never gets round the article. Revoking someone's access to the article stops their downloads at once.

The url for article HTML is the app's own document link, which re-checks the reader on every request - an analyst's or portal user's session, never an API key - so a migrated article can link to its PDF the way a hand-written one does. Nobody signed in gets nothing, as with pictures.

Limits and types are the install's attachment settings (System β†’ Attachments: 10 MB and the usual list by default), reported by GET /uploads/limits.

What happens around an upload, stated so a migration can stage drafts predictably:

Search The title is searchable at once; the text of a PDF or Office file is extracted in the background, as in the app. Search shows a document only to people who can see its article.
Help Centre Off unless show_in_help_centre: true, and customers see it only once the article is published, customer-facing and theirs. Turning it on is the same "widens who can read it" act as attaching, so it needs knowledge_documents.create.
AI and embeddings Documents are not sent to an AI provider and do not change the article's embedding.
Workflows and notifications None. Attaching a document does not trigger a workflow or a notification.
Article revision Attaching, detaching and changing a document will change the article's revision, so a later PATCH with a stale expected_revision is refused. Today the revision covers the article's pictures but not its documents; this adds them.
Audit Recorded on the article, by the key's analyst, as pictures are.

5. Security and housekeeping

  • An upload belongs to the key that started it. Another key - even one acting as the same analyst - gets 404. upload_id is 32 random characters, never a counter.
  • Pieces are never reachable over the web. They collect in a protected store (uploads/api-incoming/, registered in includes/file_stores.php like every other store, denied by the web server and checked by tests/web-exposure-guard.php). Only the hand-over moves a file out.
  • No filling the disk. A key may have at most 10 unfinished uploads, each no bigger than its purpose allows, and expired ones are deleted - on every POST /uploads (so no scheduled job is needed) and by the nightly cron where there is one.
  • Rate limits count each piece like any other request. A 50 MB file in 4 MB pieces is 14 requests; the default per-minute limit leaves room, and a key moving thousands of files can be given a higher limit on System β†’ API.
  • Memory stays flat. A piece is streamed from php://input to the file, never held whole; the hand-over hashes the file in a stream.

6. What changes for existing clients

Nothing breaks. The base64 picture upload keeps working for small pictures; project documents keep taking a multipart file. Both routes also accept upload_id, so a client can move to chunks one endpoint at a time.

7. Decisions

Decided - Ed, 21 November 2026 ("all this sounds good"):

  1. Permissions - an upload needs its purpose's create permission; there is no separate uploads.create. A key that may add a picture may upload one, and nobody has another box to tick.
  2. Article documents' permissions - their own knowledge_documents.read/create/delete, as pictures have knowledge_images.* and as #416 asks ("narrowly scoped").
  3. Help Centre through the API - show_in_help_centre can be set on create and PATCH; a migration otherwise needs a person to click through every article.
  4. Expiry - 24 hours after the last piece, fixed; revisited if anyone asks.
  5. Release - 5.9.0 ships first, as it is; this goes in 5.10.0.

Not affected: the one-request limit. The size an administrator chooses (Knowledge β†’ Settings β†’ Images, up to 50 MB) applies to the one-request base64 route too - a 50 MB picture went up in one request, replayed and downloaded byte for byte on the test install (21 Nov 2026) once its PHP allowed it (memory_limit 512M, post_max_size 128M). Chunks are for when the server's limits are smaller, not a requirement.

8. How it will be tested

Against a throwaway install, with synthetic files, as tests/knowledge-migration-api.php already is:

  • a 30 MB file in pieces assembles byte for byte; a piece at the wrong offset is 409 with received; resuming after a "dropped" piece finishes the file; a wrong sha256 is refused at the end
  • a purpose's size and extension are refused at start, and the content type again at hand-over; SVG and a PHP file renamed .png never land
  • another key cannot read, send to or hand over someone else's upload; a used upload cannot be handed over twice; a replay with the same reference returns the same record
  • pieces are never served over HTTP; ten unfinished uploads per key and no more; an expired upload's pieces are gone
  • #416's acceptance: a restricted draft gets one file (via chunks) and one link; both appear in the native Documents panel; the article's revision moves; a retried upload makes no duplicate; another company's key and a reader without access get 404; revoking access stops the download; detaching from one article leaves the same document on another
  • a server with upload_max_filesize = 2M and post_max_size = 8M - the PHP defaults - takes a 30 MB document

FreeITSM

Getting Started

Modules

Multi-tenancy (planned)

Responses to feedback

Blue sky thinking

Bugs resolved

Pull requests

Links

Clone this wiki locally