Attached Documents Developer Guide

Attached documents — Developer Guide

The schema, the endpoints, and how to give a module documents.

Permissions and search get their own page: Attached documents — permissions and search. The user-facing guide is Attached documents.

Built for discussion #76.

---

1. 📁 The files involved

🟢 The core

File	Role
includes/documents.php	The entity registry, both permission functions, orphan collection, storage-key resolution. No module knows anything about documents; this knows nothing about any module.
includes/documents_panel.php	renderDocumentsPanel() for server-rendered records, documentsPanelAssets() for pages that mount it themselves

🔵 The API

File	Role
api/documents/save.php	Create a document (file upload or external link) and attach it
api/documents/find.php	Search documents you may see, for attaching elsewhere
api/documents/attach.php	Attach an existing document to another record
api/documents/list.php	What is attached to a record. Paged
api/documents/links.php	Where one document lives — powers the ⓘ
api/documents/download.php	The boundary. Serves the file, checks access, records it
api/documents/unlink.php	Detach; collects the document if that was its last link

🟠 Search

File	Role
includes/search/documents_index.php	Extraction queue, corpus indexing, the search visibility clause
api/system/global_search.php	Two ⌘K sources: documents by name, and by content
includes/search/search.php	Applies the document clause inside searchScopeToSql()

⚪ Front end, schema and jobs

File	Role
assets/js/documents.js	The panel component and the ⓘ dialogue
assets/css/documents.css	Panel and dialogue styling
scripts/documents_maintenance.php	Extraction and orphan collection, for cron
tests/document-permissions.php	22 assertions
tests/document-search-permissions.php	9 assertions

---

2. The schema, and the one decision that shaped it

documents                     document_links
  id                            id
  kind  ('file' | 'link')       document_id  ──→ documents.id  (FK, cascade)
  title, description            parent_type  ('contract', 'asset', …)
  storage_key                   parent_id     ⚠️ NO FK POSSIBLE
  original_name, mime_type      linked_by_id
  size_bytes, content_hash      created_datetime
  external_url                UNIQUE (document_id, parent_type, parent_id)
  tenant_id, uploaded_by_id
  created/updated/deleted     document_text            document_access_log
                                document_id (PK)         document_id
                                status, extractor        analyst_id, action, ip
                                extracted_text, chars    created_datetime

The parent is a ROW, not a column on documents. That single choice is what lets one document belong to several records without a migration, and it is why deleting a record removes the link rather than a file somebody else is still using. It costs one table and one JOIN.

Five decisions came with it, all cheap at the time and expensive to retrofit:

1. A storage key, not a path. documents.storage_key is opaque; documentStoragePath() resolves it. Moving to another disk or to object storage is a change to that one function, not a data migration.
2. One table for files and links, separated by kind. Splitting later is mechanical; merging later means reconciling two sets of habits.
3. No permission columns. Ever. See the permissions page for why storing them is a bug and not merely duplication.
4. Delete the link, not the document. Garbage-collect what that orphans.
5. content_hash so the same file attached eleven times can be recognised as one.

---

3. Adding a module

One entry in documentEntityRegistry():

'contract' => [
    'module' => 'contracts',              // key in allowed_modules — the read gate
    'table'  => 'contracts',
    'label'  => 'Contract',               // shown in the UI
    'url'    => 'contracts/view.php?id=%d',
    'title'  => 'title',                  // column to display as the record's name
    'alive'  => null,                     // extra SQL keeping deleted parents out
    'can'    => null,                     // fn(PDO,$analystId,$id):bool — exact check
    'filter' => null,                     // fn(PDO,$analystId,$alias):[sql,params]
],

can and filter are the same question in two shapes — one row and a set. Pass null when module membership is the whole rule (contracts and tasks have no tenant_id).

⚠️ Do not invent a filter for a column that does not exist. The clause builder DROPS an entity type whose filter throws, so a filter on a missing tenant_id does not fail loudly — it silently hides every document on that entity.

Then mount the panel. Which of the two ways depends on how the page renders, and this is the thing that caught us out:

Server-rendered record — one line:

require_once __DIR__ . '/../includes/documents_panel.php';
renderDocumentsPanel('contract', (int) $contractId, '../');

JS-rendered detail view — the page mounts it itself, because the container is rebuilt every time the user picks a different record:

documentsPanelAssets('../');            // once, in <head>

FreeITSMDocuments.mount(document.getElementById('chgDocuments'), {
    parentType: 'change',
    parentId:   c.id,
    apiBase:    '../api/documents/'
});

Assets, Knowledge, Problems and Changes all take the second form. Mount, do not re-point — setParent() exists for a container that persists while the record changes, which none of them are.

---

4. Extraction

document_text is its own table rather than a source_type column on attachment_text, for two reasons:

• attachment_text's primary key is the attachment id alone, and a document id is a different small integer from a different table. They would collide.
• The pipelines genuinely differ. An email attachment's text belongs to its ticket, so extraction ends by reindexing the ticket. A document has no ticket; it is its own corpus row.

What they share is attTextExtractFile(), which takes a path and returns text. That is the part worth sharing.

Two tiers, and mixing them up is a real bug we shipped and fixed:

Tier	Handles	Test
1, built in	plain text, OOXML (docx/xlsx/pptx)	attTextSupports($filename)
2, Tika	PDF, images, scans, everything else	tikaConfigured($conn) && tikaHandles($filename)

⚠️ attTextSupports() is tier 1 only. Using it as the whole test marks every PDF unsupported at upload — "we looked and there is nothing to read", permanently — when the truth is "we never asked the thing that could". Use documentTextReadable().

The statuses are not interchangeable

Status	Meaning
pending	We still owe this file. Will be retried
extracted	Done
unsupported	Nothing available can read this format
failed	We asked, and were answered badly

Writing failed when the extractor is merely down would blacklist every PDF that arrived during a five-minute outage, permanently and silently. This is why Ed's three PDFs sat on pending while Tika was stopped and extracted themselves the moment it started.

---

5. Orphan collection

document_links.parent_id is polymorphic, so no foreign key can protect it. Delete a contract and its links survive; nothing in the database objects.

The document is invisible from that instant — every permission check verifies the parent still exists — so nothing leaks. But without a sweep it is never collected: the row stays and the file sits on disk for ever.

documentsDetachParent($conn, 'contract', $id);   // tidy, at the moment of deletion
documentsCollectOrphans($conn, $limit);          // the net — correct on its own

The sweep is the primary mechanism, not the hook. Twelve delete paths, plus bulk deletes, plus anything that ever removes a row in SQL, is twelve chances to forget — and forgetting is silent. It runs bounded from list.php and from the maintenance script.

⚠️ A gotcha worth keeping. The first version used DELETE dl FROM document_links dl … LIMIT n. MySQL does not accept LIMIT on a multi-table delete; the syntax error was caught, logged and swallowed, and the sweep reported "0 removed" — indistinguishable from a clean run. It is a single-table DELETE now, and documentsCollectOrphans() returns its errors so a caller can tell "nothing to do" from "it broke".

---

6. Things that will bite

• The upload folder must be denied by the web server. uploads/documents/ ships an .htaccess and a web.config; nginx needs the rule in deploy/nginx/freeitsm.conf. Run D009 Guarded paths to find out what your server is actually honouring — a random stored filename is not a permission.
• window.t is guarded in documents.js. This component lands on a dozen pages, which is a dozen chances to hit one that forgot i18n.js.
• Nothing in the panel's layout is inherited. .fd-drop is a <label>; a page that does not make labels block-level left it inline and the dashed border wrapped the text instead of drawing a box.
• The command palette drops result types it does not know. Adding a type server-side is not enough — it must be in the array in command-palette.js or the results vanish with no error.
• renderDocumentsPanel() renders nothing when $parentId is 0. An unsaved record has nothing to attach to.

---

Related pages

• Attached documents — the user-facing guide
• Attached documents — permissions and search
• Attachment text extraction — Developer Guide — the email-attachment pipeline this borrows from
• Security — includes/uploads.php and the guarded folders