Docs: add any feature gaps

AGENTS.md

@@ -0,0 +1,9 @@
+# Documentation
+
+## What is generated vs hand-written
+
+Before modifying anything under `docs/`, read the scripts in the `scripts/` directory to determine whether the target file is generated. Generated files are overwritten on every script run; edits to them will be lost. If a file is generated, modify the relevant script or its data sources instead of editing the output directly, then re-run the script to rebuild.
+
+## Preview wrapper classes
+
+All generated component and page previews include `app-no-prose` on their wrapper `<div>` by default. This prevents Docusaurus prose CSS from interfering with GOV.UK component styles. Do not remove it.

docs/BUILDING_THE_PACKAGE.md

@@ -1,10 +1,3 @@
----
-layout: default
-title: Building the package
-render_with_liquid: false
-nav_order: 5
----
-
 # Building the package
 
 1. [Overview](#overview)

docs/contributing.md

@@ -56,3 +56,7 @@ If you would like to fix the bug yourself, contributions are accepted through pu
 ### Adding features
 
 Features should be discussed with the Defra Forms team prior to implementation. This is to prevent wasted effort if the Defra Forms team decides not to accept it, or if we suggest any significant amendments. Reach out to us on [#defra-forms-support](https://defra-digital-team.slack.com) to discuss your requirements. If accepted by the product owner, we welcome a pull request.
+
+## Building and publishing the package
+
+See the [Building the package](./BUILDING_THE_PACKAGE) guide for documentation on the build pipeline, path alias resolution, and the npm publish workflow.

docs/features/code-based/custom-services.md

@@ -1,90 +1,134 @@
 # Overriding forms-engine-plugin logic with custom services
 
-## Customising where forms are loaded from
+The `services` plugin option accepts three service objects — `formsService`, `formSubmissionService`, and `outputService` — which together cover where forms come from, where submission data goes, and how submission notifications are sent. Replace any or all of them to integrate with your own infrastructure.
 
-The engine plugin registers several [routes](https://hapi.dev/tutorials/routing/?lang=en_US) on the hapi server.
+## formsService
 
-They look like this:
+Responsible for loading form metadata and definitions. Called on every page request to check for definition changes and load the full definition when needed.
 
-```
-GET     /{slug}/{path}
-POST    /{slug}/{path}
-```
-
-A unique `slug` is used to route the user to the correct form, and the `path` used to identify the correct page within the form to show.
-
-The [plugin registration options](/plugin-options) have a `services` setting to provide a `formsService` that is responsible for returning `form definition` data.
-
-WARNING: This below is subject to change
-
-A `formsService` has two methods, one for returning `formMetadata` and another to return `formDefinition`s.
-
-```javascript
-const formsService = {
-  getFormMetadata: async function (slug) {
-    // Returns the metadata for the slug
-  },
-  getFormDefinition: async function (id, state) {
-    // Returns the form definition for the given id
-  }
+```ts
+interface FormsService {
+  getFormMetadata: (slug: string) => Promise<FormMetadata>
+  getFormMetadataById: (id: string) => Promise<FormMetadata>
+  getFormDefinition: (id: string, state: FormStatus) => Promise<FormDefinition | undefined>
+  getFormSecret: (formId: string, secretName: string) => Promise<string>
 }
 ```
 
-The reason for the two separate methods is caching.
-`formMetadata` is a lightweight record designed to give top level information about a form.
-This method is invoked for every page request.
+`getFormMetadata` is called on every request and should be fast. `getFormDefinition` is only called when the metadata signals the definition has changed, so it can be slower. `getFormMetadataById` is called by the status page to retrieve the submitted form's name by its ID — this allows the confirmation panel to display the correct form name even when the current URL belongs to a different form (for example, a shared feedback form).
 
-Only when the `formMetadata` indicates that the definition has changed is a call to `getFormDefinition` is made.
-The response from this can be quite big as it contains the entire form definition.
+`getFormSecret` retrieves a secret that belongs to a specific form — for example, a payment API key scoped to that form. This is distinct from global secrets such as `ordnanceSurveyApiKey` and `ordnanceSurveyApiSecret`, which are passed once at plugin registration and apply across all forms. If your forms do not use any components that require per-form secrets, this method will never be called and you can safely return `undefined` (or leave it unimplemented). See the documentation for each component to find out whether it requires secrets and what names it requests.
 
-## Loading forms from files
+### Loading forms from files
 
-To create a `formsService` from form config files that live on disk, you can use the `FileFormService` class.
-Form definition config files can be either `.json` or `.yaml`.
+For local or file-based forms, use the built-in `FileFormService`:
 
-Once created and files have been loaded using the `addForm` method,
-call the `toFormsService` method to return a `FormService` compliant interface which can be passed in to the `services` setting of the [plugin options](/plugin-options).
-
-```javascript
+```js
 import { FileFormService } from '@defra/forms-engine-plugin/file-form-service.js'
 
-// Create shared form metadata
 const now = new Date()
 const user = { id: 'user', displayName: 'Username' }
 const author = { createdAt: now, createdBy: user, updatedAt: now, updatedBy: user }
-const metadata = {
+
+const loader = new FileFormService()
+
+await loader.addForm('src/definitions/example-form.yaml', {
+  id: '95e92559-968d-44ae-8666-2b1ad3dffd31',
+  title: 'Example form',
+  slug: 'example-form',
   organisation: 'Defra',
   teamName: 'Team name',
   teamEmail: 'team@defra.gov.uk',
   submissionGuidance: "Thanks for your submission, we'll be in touch",
-  notificationEmail: 'email@domain.com',
+  notificationEmail: 'team@defra.gov.uk',
   ...author,
   live: author
+})
+
+const formsService = loader.toFormsService()
+```
+
+---
+
+## formSubmissionService
+
+Called during form submission to persist the submitted data and any uploaded files. The default implementation calls the Defra Forms submission API (`SUBMISSION_URL`), which is part of the Defra Forms hosting infrastructure. Teams not using that infrastructure must provide their own implementation.
+
+```ts
+interface FormSubmissionService {
+  persistFiles: (
+    files: { fileId: string; initiatedRetrievalKey: string }[],
+    persistedRetrievalKey: string
+  ) => Promise<object>
+  submit: (data: SubmitPayload) => Promise<SubmitResponsePayload | undefined>
 }
+```
 
-// Instantiate the file loader form service
-const loader = new FileFormService()
+`submit` is called first with the structured form payload. The `SubmitResponsePayload` it returns (including CSV file IDs) is then passed to `outputService.submit`. `persistFiles` is called by `FileUploadField` during submission to move uploaded files from temporary to permanent storage.
 
-// Add a Json form
-await loader.addForm(
-  'src/definitions/example-form.json', {
-    ...metadata,
-    id: '95e92559-968d-44ae-8666-2b1ad3dffd31',
-    title: 'Example Json',
-    slug: 'example-json'
-  }
-)
-
-// Add a Yaml form
-await loader.addForm(
-  'src/definitions/example-form.yaml', {
-    ...metadata,
-    id: '641aeafd-13dd-40fa-9186-001703800efb',
-    title: 'Example Yaml',
-    slug: 'example-yaml'
+Override this service if you are not using the Defra Forms hosting infrastructure, or if you handle file persistence differently.
+
+---
+
+## outputService
+
+Called after `formSubmissionService.submit` completes. Its job is to deliver the submission — by default, as a GOV.UK Notify email.
+
+```ts
+interface OutputService {
+  submit: (
+    context: FormContext,
+    request: FormRequestPayload,
+    model: FormModel,
+    emailAddress: string,
+    items: DetailItem[],
+    submitResponse: SubmitResponsePayload,
+    formMetadata?: FormMetadata
+  ) => Promise<void>
+}
+```
+
+The default implementation (`notifyService`) formats the submission using the [output formatter](#output-format) configured on the form definition and sends it to `emailAddress` via GOV.UK Notify.
+
+Override this service to deliver submissions differently — for example, publishing to an SNS topic, calling a webhook, or writing to a database. Your implementation receives the full `FormContext`, `FormModel`, `DetailItem[]` array, and the `SubmitResponsePayload` from `formSubmissionService`, giving you everything needed to format and route the submission however you need.
+
+```js
+await server.register({
+  plugin,
+  options: {
+    services: {
+      formsService,
+      formSubmissionService,
+      outputService: {
+        async submit(context, request, model, emailAddress, items, submitResponse, formMetadata) {
+          // publish to SNS, call a webhook, etc.
+        }
+      }
+    }
   }
-)
+})
+```
 
-// Get the forms service
-const formsService = loader.toFormsService()
+### Output format
+
+If you use the default `notifyService`, the format of the email body is controlled by the `output` field in the form definition:
+
+```json
+{
+  "output": {
+    "audience": "human",
+    "version": "1"
+  }
+}
 ```
+
+| Value                 | Description                                                                                                                                            |
+| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `audience: "human"`   | Formats the submission as readable Markdown for a GOV.UK Notify email template. Default.                                                               |
+| `audience: "machine"` | Formats the submission as a JSON payload, base64-encoded into the Notify email body. Useful when a downstream system reads the email programmatically. |
+
+`version` selects the formatter version within that audience. Currently `"1"` is the only stable version for `human`; `"1"` and `"2"` are available for `machine`. Defaults to `"1"` when omitted.
+
+If you provide a custom `outputService`, the `output` field has no effect — your service controls formatting entirely.
+
+> **Note:** Page events always use the `machine/v1` payload format regardless of the `output` setting. The `output` field only affects what the default `notifyService` sends.

docs/features/code-based/form-definition-formats.md

@@ -0,0 +1,102 @@
+# Form definition formats
+
+Form definitions are retrieved by `forms-engine-plugin` using the `formsService` plugin registration option. The plugin calls `getFormDefinition()` on every page request, which must return a JavaScript object matching the form definition schema.
+
+Two approaches are available:
+
+- **File-based loading** — store form definitions as YAML or JSON files in your repository and use the built-in `FileFormService`
+- **Custom service** — implement your own `formsService` to load definitions from an API, database, or any other source
+
+## File-based loading
+
+The built-in `FileFormService` loads form definitions from disk. YAML is recommended for forms with multi-line HTML content, as it natively supports block scalars. JSON is more portable but requires manually escaping quotes and line breaks in string values.
+
+### Registering forms
+
+```js
+import { FileFormService } from '@defra/forms-engine-plugin/file-form-service.js'
+
+const now = new Date()
+const user = { id: 'user', displayName: 'Username' }
+const author = { createdAt: now, createdBy: user, updatedAt: now, updatedBy: user }
+
+const loader = new FileFormService()
+
+await loader.addForm('src/definitions/example-form.yaml', {
+  id: '95e92559-968d-44ae-8666-2b1ad3dffd31',
+  title: 'Example form',
+  slug: 'example-form',
+  organisation: 'Defra',
+  teamName: 'Team name',
+  teamEmail: 'team@defra.gov.uk',
+  submissionGuidance: "Thanks for your submission, we'll be in touch",
+  notificationEmail: 'team@defra.gov.uk',
+  ...author,
+  live: author
+})
+
+const formsService = loader.toFormsService()
+```
+
+Pass the resulting `formsService` as a plugin registration option:
+
+```js
+await server.register({
+  plugin,
+  options: {
+    services: { formsService },
+    // ...
+  }
+})
+```
+
+Call `loader.addForm()` once per form definition file. The `slug` controls the URL path — a form with `slug: 'example-form'` is served at `/example-form/*`.
+
+### YAML vs JSON
+
+```yaml
+# example-form.yaml — YAML supports multi-line content natively
+name: "Form name"
+pages:
+  - title: "Page title"
+    components:
+      - type: "Html"
+        content: |
+          <h1 class="govuk-heading-l">Heading</h1>
+          <p class="govuk-body">Body text</p>
+```
+
+```jsonc
+// example-form.json — JSON requires escaped quotes and no newlines in strings
+{
+  "name": "Form name",
+  "pages": [
+    {
+      "title": "Page title",
+      "components": [
+        {
+          "type": "Html",
+          "content": "<h1 class=\"govuk-heading-l\">Heading</h1><p class=\"govuk-body\">Body text</p>"
+        }
+      ]
+    }
+  ]
+}
+```
+
+## Custom formsService
+
+To load form definitions from an API, database, or CMS, implement a custom `formsService` and pass it at plugin registration. The interface requires four methods:
+
+```ts
+interface FormsService {
+  getFormMetadata: (slug: string) => Promise<FormMetadata>
+  getFormMetadataById: (id: string) => Promise<FormMetadata>
+  getFormDefinition: (id: string, state: FormStatus) => Promise<FormDefinition | undefined>
+  getFormSecret: (formId: string, secretName: string) => Promise<string>
+}
+```
+
+`getFormMetadata` is called on every page request and should be fast. `getFormDefinition` is only called when the metadata signals the definition has changed, so it can do heavier lifting.
+
+See [Custom Services](./custom-services) for a full implementation guide.

docs/features/code-based/index.md

@@ -31,3 +31,15 @@ Automatically copy query string parameter values into hidden fields on first loa
 ## [Save and Exit](./code-based/save-and-exit)
 
 Show a secondary "Save and exit" button on question pages and handle the persisted session using a route handler you supply, enabling users to leave and resume their journey later.
+
+## [Template Extensions](./code-based/template-extensions)
+
+Add custom globals and filters to the Nunjucks template environment, making them available across all form page templates and LiquidJS page templates.
+
+## [Form Definition Formats](./code-based/form-definition-formats)
+
+Options for loading form definitions — file-based loading with the built-in `FileFormService`, or a custom `formsService` implementation for API and database sources.
+
+## [Session Cache](./code-based/session-cache)
+
+Configuring the server-side session store for production: named catbox cache or a custom `CacheService` subclass.