feat(datasource): runtime-UI-creatable datasources (ADR-0015 Addendum)

.objectui-sha

@@ -1 +1 @@
-0d47ceb6796c7f08bae9c3fdde978ce3f5834d3c
+1c8f7753898b6283ffc528d2fb2ee85f68886ca5

docs/plans/external-datasource-federation-impl.md

@@ -49,15 +49,16 @@ already exists and is reused:
 | Phase | Scope | Status |
 |:-----:|:--|:--|
 | **P1** | Spec changes (`schemaMode`, `object.external`, error classes) + DDL gate in `driver-sql` + tests | ✅ **Done** (this branch) |
-| **P2** | `IExternalDatasourceService` impl + type-compat matrix + CLI `introspect`/`validate` | 🟡 **Service core done** (matrix + contract + service); REST routes + CLI pending |
-| **P3** | Boot-validation plugin in `@objectstack/runtime` + `external_catalog` metadata type + caching | ⬜ Todo |
-| **P4** | `SchemaRetriever` annotation + agent prompt + AI safety nets (LIMIT injection, timeout) | ⬜ Todo |
+| **P2** | `IExternalDatasourceService` impl + type-compat matrix + CLI `introspect`/`validate` | ✅ **Done** (service + matrix; REST `/external/*` mounted in `rest-api-plugin`; CLI `datasource list-tables`/`introspect`/`validate`; `engine.introspectDatasource`) |
+| **P3** | Boot-validation plugin in `@objectstack/runtime` + `external_catalog` metadata type + caching | ✅ **Done** |
+| **P4** | `SchemaRetriever` annotation + agent prompt + AI safety nets (LIMIT injection, timeout) | ✅ **Done** (external badge in `SchemaRetriever.renderSnippet`; `query_data` injects LIMIT + per-query timeout for federated objects via `external.queryTimeoutMs`) |
 | **P5** | Studio UI in `../objectui` (wizard, schema browser, mapping editor, validation panel) | ⬜ Todo |
-| **P6** | Write gate + `allowWrites`/`writable` double opt-in + tests | ⬜ Todo |
+| **P6** | Write gate + `allowWrites`/`writable` double opt-in + tests | ✅ **Done** (`engine.assertWriteAllowed`, called from insert/update/delete; `external-write-gate.test.ts`) |
 | **P7** | Additional drivers (Snowflake / BigQuery / MySQL) | ⬜ Todo |
 
 **MVP = P1–P4**: connect a read-only Postgres replica, register a few
-tables, let AI Data Chat query them safely.
+tables, let AI Data Chat query them safely. ✅ **MVP complete** — P1–P4 + P6
+all landed; remaining work is P5 (Studio UI) and P7 (more drivers).
 
 ## P1 — delivered in this change
 
@@ -133,12 +134,87 @@ behaviour).
   `getDatasourceDriver(name)` / `introspectDatasource(name)` on the data
   engine so the plugin's default `introspect` works end-to-end.
 
+## P3 — delivered
+
+Gate 2 (boot validation) + remote-schema caching.
+
+1. **`external_catalog` metadata type** — registered in
+   `packages/spec/src/kernel/metadata-plugin.zod.ts`
+   (`allowRuntimeCreate`, `loadOrder: 6`, system domain) with its Zod schema
+   `packages/spec/src/data/external-catalog.zod.ts`
+   (`ExternalCatalogSchema` → `ExternalCatalog`: name / datasource /
+   `snapshotAt` / dialect / tables[columns]).
+
+2. **Boot-validation plugin** — `ExternalValidationPlugin`
+   (`packages/runtime/src/external-validation-plugin.ts`) subscribes to
+   `kernel:ready`, calls `external-datasource`'s `validateAll()`, and applies
+   each datasource's `external.validation.onMismatch` policy
+   (`fail` aborts boot via `ExternalSchemaMismatchError`, `warn` logs,
+   `ignore` no-ops). No-op when the service is absent. **Now registered into
+   the serve boot sequence** alongside the datasource plugins
+   (`packages/cli/src/commands/serve.ts`).
+
+3. **`schemaMode` → driver injection** — `createDefaultDatasourceDriverFactory`
+   (`packages/runtime/src/default-datasource-driver-factory.ts`) threads a
+   datasource's `schemaMode` into `SqlDriverConfig`, so the P1 DDL gate fires
+   for runtime-created external datasources too.
+
+4. **Catalog persistence** — `refreshCatalog` now parses the snapshot through
+   `ExternalCatalogSchema` and persists it as an `external_catalog` metadata
+   record via an injected `persistCatalog` dep (wired in `plugin.ts` to
+   `metadata.register`). Best-effort: a persist failure still returns the live
+   snapshot. Tests cover persistence, the read-only/throwing store, and the
+   canonicalised shape.
+
+5. **Background drift detection** — `ExternalValidationPlugin` now arms a
+   per-datasource `setInterval` for every federated datasource that declares
+   `external.validation.checkIntervalMs` (ADR §5.2). Each tick re-runs
+   `validateAll()` and emits one `external.schema.drift` event
+   (`{ datasource, object, diffs }`, type `ExternalSchemaDriftEvent`) on the
+   kernel bus per drifted object — observational, so it never throws or aborts
+   the process (unlike boot validation). Timers `unref()` and are cleared on
+   `stop()`; re-arming clears prior timers so intervals can't accumulate.
+   Consumed by `audit` / `notification` services. Tests cover event emission,
+   the validateAll-rejects no-op, selective scheduling, the firing interval,
+   re-arm idempotence, and the no-metadata no-op.
+
+## P-C — delivered (runtime "Import as Object", ADR-0015 Addendum)
+
+The runtime persona's create-in-UI bridge: turn a browsed remote table into a
+live, immediately-queryable federated object — no git commit (that stays the
+GitOps `os datasource introspect` path).
+
+1. **`IExternalDatasourceService.importObject(datasource, remoteName, opts?)`**
+   (`packages/spec/src/contracts/external-datasource-service.ts`) → `ImportObjectResult`
+   (`{ name, definition, review }`). `ImportObjectOpts` extends `GenerateDraftOpts`
+   with `name` (override) + `writable` (object.external.writable opt-in; still
+   gated by datasource `external.allowWrites`, ADR Gate 3).
+
+2. **Service impl** (`external-datasource-service.ts`) reuses the `generateObjectDraft`
+   pipeline (type mapping + review notes + external binding), applies the
+   name/writable overrides, and persists via an injected `persistObject`. Throws
+   a descriptive error when no writable metadata store is wired (GitOps-only
+   deployment) and when the remote table is missing (before any write).
+
+3. **Plugin wiring** (`plugin.ts`) supplies `persistObject` →
+   `metadata.register('object', name, definition)` (runtime origin), alongside
+   the existing `persistCatalog`.
+
+4. **REST** — `POST /api/v1/datasources/:name/external/tables/:remote/import`
+   (`packages/rest/src/external-datasource-routes.ts`): `201 { object }` on
+   success, `503` when the service is absent, `400 external_import_error` when
+   import is refused (read-only store / missing table). Body carries
+   `ImportObjectOpts`.
+
+5. **Tests** — service: persists read-only by default, name+writable overrides,
+   draft-option forwarding (include/rename), throws without a store, throws on
+   missing table without persisting (47 green).
+
 ### Follow-up notes / open items for later phases
 
-- **DDL gate plumbing (P3)**: the runtime must inject `Datasource.schemaMode`
-  into `SqlDriverConfig` when constructing drivers. P1 wires the driver
-  side and defaults to `'managed'`; the runtime wiring lands with the
-  boot-validation plugin.
+- **DDL gate plumbing (P3)**: ✅ done — `createDefaultDatasourceDriverFactory`
+  injects `Datasource.schemaMode` into `SqlDriverConfig`. P1 wired the driver
+  side and defaulted to `'managed'`.
 - **`applyMigrations` gate**: `ISchemaDiffService.applyMigrations` also
   needs the gate (per ADR §5.1) when the migration runner ships.
 - **Lint rule** preventing plugins from bypassing the gate via raw `knex`

packages/cli/package.json

@@ -68,6 +68,7 @@
     "@objectstack/service-analytics": "workspace:*",
     "@objectstack/service-automation": "workspace:*",
     "@objectstack/service-cache": "workspace:*",
+    "@objectstack/service-external-datasource": "workspace:*",
     "@objectstack/service-feed": "workspace:*",
     "@objectstack/service-job": "workspace:*",
     "@objectstack/service-package": "workspace:*",

packages/cli/src/commands/serve.ts

@@ -1537,6 +1537,80 @@ export default class Serve extends Command {
         }
       }
 
+      // ── Runtime-UI datasource lifecycle (ADR-0015 Addendum) ────────
+      // Always available so the Studio "Add Datasource" wizard has a
+      // working backend: list (code + runtime origins), test a connection,
+      // and create / update / remove runtime datasources. Code-origin
+      // datasources stay read-only. The connection probe + hot pool use a
+      // default driver factory (postgres / sqlite / mongodb / memory) and
+      // the secret is wrapped into `sys_secret` via a fail-closed binder —
+      // only a `credentialsRef` is ever persisted, never cleartext.
+      try {
+        const dsMod: any = await import('@objectstack/service-external-datasource');
+        const { ExternalDatasourceServicePlugin, DatasourceAdminServicePlugin } = dsMod;
+
+        if (
+          ExternalDatasourceServicePlugin &&
+          !hasPluginMatching(['service-external-datasource', 'ExternalDatasourceServicePlugin'])
+        ) {
+          await kernel.use(new ExternalDatasourceServicePlugin());
+          trackPlugin('ExternalDatasourceServicePlugin');
+        }
+
+        if (
+          DatasourceAdminServicePlugin &&
+          !hasPluginMatching(['service-datasource-admin', 'DatasourceAdminServicePlugin'])
+        ) {
+          const rtMod: any = await import('@objectstack/runtime');
+          const { createDefaultDatasourceDriverFactory, createDatasourceSecretBinder } = rtMod;
+
+          // Fail-closed secret binder: when no crypto provider is available
+          // the admin service still loads, but secret-bearing create/update
+          // throws rather than persist cleartext.
+          let secrets: any;
+          try {
+            const { InMemoryCryptoProvider } = await import('@objectstack/service-settings');
+            const cryptoProvider = new InMemoryCryptoProvider();
+            const lazyEngine = {
+              insert: (o: string, d: any, opt?: any) => (kernel.getService('data') as any).insert(o, d, opt),
+              delete: (o: string, opt: any) => (kernel.getService('data') as any).delete(o, opt),
+              // Read path for boot rehydration: the secret binder dereferences
+              // each runtime datasource's `credentialsRef` from `sys_secret`.
+              find: (o: string, q: any) => (kernel.getService('data') as any).find(o, q),
+            };
+            secrets = createDatasourceSecretBinder({ engine: lazyEngine, cryptoProvider });
+          } catch {
+            /* no crypto provider — admin service loads, secrets fail closed */
+          }
+
+          await kernel.use(
+            new DatasourceAdminServicePlugin({
+              driverFactory: createDefaultDatasourceDriverFactory(),
+              ...(secrets ? { secrets } : {}),
+            }),
+          );
+          trackPlugin('DatasourceAdminServicePlugin');
+        }
+
+        // Gate 2 (ADR-0015 §5.2): on kernel:ready, validate every federated
+        // object against its remote table and apply the datasource's
+        // `external.validation.onMismatch` policy. No-op when the
+        // `external-datasource` service isn't registered (federation unused).
+        const { createExternalValidationPlugin } = await import('@objectstack/runtime');
+        if (
+          createExternalValidationPlugin &&
+          !hasPluginMatching(['external-validation', 'ExternalValidationPlugin'])
+        ) {
+          await kernel.use(createExternalValidationPlugin());
+          trackPlugin('ExternalValidationPlugin');
+        }
+      } catch (err: unknown) {
+        const msg = err instanceof Error ? err.message : String(err);
+        if (!msg.includes('Cannot find module') && !msg.includes('ERR_MODULE_NOT_FOUND')) {
+          console.error(`[Datasource] runtime-UI lifecycle wiring failed: ${msg}`);
+        }
+      }
+
       // ── UI portals ────────────────────────────────────────────────
       // In dev mode, the bundled Console portal is enabled by default
       // (use --no-ui to disable). Always serve the pre-built `dist/` — no

packages/metadata/src/metadata-manager.ts

@@ -327,6 +327,23 @@ export class MetadataManager implements IMetadataService {
     }
   }
 
+  /**
+   * Register a metadata item into the in-memory registry ONLY, never persisting
+   * to a writable loader. Used for GitOps-managed artefacts that must be
+   * *listable* (so `list(type)` returns them) but must never leak into the
+   * runtime DB store — e.g. code-defined datasources (`origin:'code'`, ADR-0015
+   * Addendum) declared in `*.datasource.ts` and owned by source control. Writing
+   * them through `register()` would persist them to `sys_metadata` and create
+   * drift between the artefact and the DB; this method avoids that.
+   */
+  registerInMemory(type: string, name: string, data: unknown): void {
+    if (!this.registry.has(type)) {
+      this.registry.set(type, new Map());
+    }
+    this.registry.get(type)!.set(name, data);
+    this.invalidateListCache(type);
+  }
+
   /**
    * Get a metadata item by type and name.
    * Checks in-memory registry first, then falls back to loaders.

packages/objectql/src/overlay-precedence.test.ts

@@ -164,19 +164,19 @@ describe('overlay whitelist enforcement (shared-DB invariant)', () => {
     //     blocked only when overlaying an artifact-backed item. Brand-new
     //     (artifact-free) names succeed. Tested separately below.
     //
-    //  2. Types with `allowRuntimeCreate: false` (datasource/router/
-    //     function/service) — blocked for ANY write in project-kernel mode.
+    //  2. Types with `allowRuntimeCreate: false` (router/function/service) —
+    //     blocked for ANY write in project-kernel mode.
+    //
+    //     NOTE: `datasource` moved to cohort #1 with the ADR-0015 Addendum
+    //     (runtime-UI-creatable datasources). Brand-new runtime datasources
+    //     are now allowed; collision with a code-defined (artifact-backed)
+    //     datasource is still refused via the artifact provenance check.
     //     The error code surfaces as `not_creatable` when the item has no
     //     artifact (which the empty test-mock registry guarantees) and
     //     `not_overridable` when an artifact exists. Both carry status 403
     //     and the same underlying security guarantee.
     describe('denied — must throw 403 (not_overridable or not_creatable)', () => {
         const deniedTypeWide: Array<{ type: string; reason: string; item: any }> = [
-            {
-                type: 'datasource',
-                reason: 'wiring level; must be code, not per-org metadata',
-                item: { name: 'analytics', driver: 'sql' },
-            },
             {
                 type: 'router',
                 reason: 'API routing must be deterministic; per-org divergence creates invisible conflicts',
@@ -192,11 +192,6 @@ describe('overlay whitelist enforcement (shared-DB invariant)', () => {
                 reason: 'service definitions must be deployment-only, not per-org',
                 item: { name: 'notification_service' },
             },
-            {
-                type: 'datasources', // plural — `datasources` maps to `datasource` in PLURAL_TO_SINGULAR
-                reason: 'plural form of denied type must also be denied',
-                item: { name: 'analytics' },
-            },
         ];
 
         for (const { type, reason, item } of deniedTypeWide) {
@@ -248,6 +243,18 @@ describe('overlay whitelist enforcement (shared-DB invariant)', () => {
                 type: 'field',
                 item: { name: 'tenant_widget_color', type: 'text', label: 'Color' },
             },
+            // datasource/datasources became runtime-creatable with the
+            // ADR-0015 Addendum (UI "Add Datasource"). Brand-new runtime
+            // datasources succeed; code-defined collisions are refused via
+            // artifact provenance (exercised in protocol-meta.test.ts).
+            {
+                type: 'datasource',
+                item: { name: 'analytics', driver: 'sql', config: {} },
+            },
+            {
+                type: 'datasources', // plural — maps to `datasource` via PLURAL_TO_SINGULAR
+                item: { name: 'analytics2', driver: 'sql', config: {} },
+            },
         ];
 
         for (const { type, item } of runtimeCreatable) {

packages/plugins/plugin-hono-server/src/adapter.ts

@@ -108,6 +108,18 @@ export class HonoHttpServer implements IHttpServer {
             let streamEncoder: TextEncoder | null = null;
             let streamHeaders: Record<string, string> = {};
             let isStreaming = false;
+            let streamClosed = false;
+
+            // The unused stream is always created (see below) and may be closed
+            // from two places — `res.end()` and the post-handler cleanup — so
+            // guard against the double-close that crashes the event loop with
+            // `ERR_INVALID_STATE: Controller is already closed`.
+            const closeStream = () => {
+                if (streamController && !streamClosed) {
+                    streamClosed = true;
+                    try { streamController.close(); } catch { /* already closed */ }
+                }
+            };
 
             const res = {
                 json: (data: any) => { capturedResponse = c.json(data); },
@@ -133,9 +145,14 @@ export class HonoHttpServer implements IHttpServer {
                     }
                 },
                 end: () => {
-                    if (streamController) {
-                        streamController.close();
+                    // Body-less response (e.g. 204 No Content) honoring any
+                    // status already set via `res.status()`. A null body avoids
+                    // the undici "Invalid response status code 204" thrown when
+                    // an empty *string* body is paired with a null-body status.
+                    if (!isStreaming && capturedResponse === undefined) {
+                        capturedResponse = c.body(null);
                     }
+                    closeStream();
                 },
             };
 
@@ -160,11 +177,11 @@ export class HonoHttpServer implements IHttpServer {
                         }));
                     } else {
                         // Not streaming — close the unused stream and return null
-                        streamController?.close();
+                        closeStream();
                         resolve(null);
                     }
                 }).catch((err) => {
-                    streamController?.close();
+                    closeStream();
                     resolve(null);
                 });
             });