diff --git a/.objectui-sha b/.objectui-sha index 804bb436b..3e8ab0a40 100644 --- a/.objectui-sha +++ b/.objectui-sha @@ -1 +1 @@ -0d47ceb6796c7f08bae9c3fdde978ce3f5834d3c +1c8f7753898b6283ffc528d2fb2ee85f68886ca5 diff --git a/docs/plans/external-datasource-federation-impl.md b/docs/plans/external-datasource-federation-impl.md index 9879a9907..166a2725b 100644 --- a/docs/plans/external-datasource-federation-impl.md +++ b/docs/plans/external-datasource-federation-impl.md @@ -49,15 +49,16 @@ already exists and is reused: | Phase | Scope | Status | |:-----:|:--|:--| | **P1** | Spec changes (`schemaMode`, `object.external`, error classes) + DDL gate in `driver-sql` + tests | โœ… **Done** (this branch) | -| **P2** | `IExternalDatasourceService` impl + type-compat matrix + CLI `introspect`/`validate` | ๐ŸŸก **Service core done** (matrix + contract + service); REST routes + CLI pending | -| **P3** | Boot-validation plugin in `@objectstack/runtime` + `external_catalog` metadata type + caching | โฌœ Todo | -| **P4** | `SchemaRetriever` annotation + agent prompt + AI safety nets (LIMIT injection, timeout) | โฌœ Todo | +| **P2** | `IExternalDatasourceService` impl + type-compat matrix + CLI `introspect`/`validate` | โœ… **Done** (service + matrix; REST `/external/*` mounted in `rest-api-plugin`; CLI `datasource list-tables`/`introspect`/`validate`; `engine.introspectDatasource`) | +| **P3** | Boot-validation plugin in `@objectstack/runtime` + `external_catalog` metadata type + caching | โœ… **Done** | +| **P4** | `SchemaRetriever` annotation + agent prompt + AI safety nets (LIMIT injection, timeout) | โœ… **Done** (external badge in `SchemaRetriever.renderSnippet`; `query_data` injects LIMIT + per-query timeout for federated objects via `external.queryTimeoutMs`) | | **P5** | Studio UI in `../objectui` (wizard, schema browser, mapping editor, validation panel) | โฌœ Todo | -| **P6** | Write gate + `allowWrites`/`writable` double opt-in + tests | โฌœ Todo | +| **P6** | Write gate + `allowWrites`/`writable` double opt-in + tests | โœ… **Done** (`engine.assertWriteAllowed`, called from insert/update/delete; `external-write-gate.test.ts`) | | **P7** | Additional drivers (Snowflake / BigQuery / MySQL) | โฌœ Todo | **MVP = P1โ€“P4**: connect a read-only Postgres replica, register a few -tables, let AI Data Chat query them safely. +tables, let AI Data Chat query them safely. โœ… **MVP complete** โ€” P1โ€“P4 + P6 +all landed; remaining work is P5 (Studio UI) and P7 (more drivers). ## P1 โ€” delivered in this change @@ -133,12 +134,87 @@ behaviour). `getDatasourceDriver(name)` / `introspectDatasource(name)` on the data engine so the plugin's default `introspect` works end-to-end. +## P3 โ€” delivered + +Gate 2 (boot validation) + remote-schema caching. + +1. **`external_catalog` metadata type** โ€” registered in + `packages/spec/src/kernel/metadata-plugin.zod.ts` + (`allowRuntimeCreate`, `loadOrder: 6`, system domain) with its Zod schema + `packages/spec/src/data/external-catalog.zod.ts` + (`ExternalCatalogSchema` โ†’ `ExternalCatalog`: name / datasource / + `snapshotAt` / dialect / tables[columns]). + +2. **Boot-validation plugin** โ€” `ExternalValidationPlugin` + (`packages/runtime/src/external-validation-plugin.ts`) subscribes to + `kernel:ready`, calls `external-datasource`'s `validateAll()`, and applies + each datasource's `external.validation.onMismatch` policy + (`fail` aborts boot via `ExternalSchemaMismatchError`, `warn` logs, + `ignore` no-ops). No-op when the service is absent. **Now registered into + the serve boot sequence** alongside the datasource plugins + (`packages/cli/src/commands/serve.ts`). + +3. **`schemaMode` โ†’ driver injection** โ€” `createDefaultDatasourceDriverFactory` + (`packages/runtime/src/default-datasource-driver-factory.ts`) threads a + datasource's `schemaMode` into `SqlDriverConfig`, so the P1 DDL gate fires + for runtime-created external datasources too. + +4. **Catalog persistence** โ€” `refreshCatalog` now parses the snapshot through + `ExternalCatalogSchema` and persists it as an `external_catalog` metadata + record via an injected `persistCatalog` dep (wired in `plugin.ts` to + `metadata.register`). Best-effort: a persist failure still returns the live + snapshot. Tests cover persistence, the read-only/throwing store, and the + canonicalised shape. + +5. **Background drift detection** โ€” `ExternalValidationPlugin` now arms a + per-datasource `setInterval` for every federated datasource that declares + `external.validation.checkIntervalMs` (ADR ยง5.2). Each tick re-runs + `validateAll()` and emits one `external.schema.drift` event + (`{ datasource, object, diffs }`, type `ExternalSchemaDriftEvent`) on the + kernel bus per drifted object โ€” observational, so it never throws or aborts + the process (unlike boot validation). Timers `unref()` and are cleared on + `stop()`; re-arming clears prior timers so intervals can't accumulate. + Consumed by `audit` / `notification` services. Tests cover event emission, + the validateAll-rejects no-op, selective scheduling, the firing interval, + re-arm idempotence, and the no-metadata no-op. + +## P-C โ€” delivered (runtime "Import as Object", ADR-0015 Addendum) + +The runtime persona's create-in-UI bridge: turn a browsed remote table into a +live, immediately-queryable federated object โ€” no git commit (that stays the +GitOps `os datasource introspect` path). + +1. **`IExternalDatasourceService.importObject(datasource, remoteName, opts?)`** + (`packages/spec/src/contracts/external-datasource-service.ts`) โ†’ `ImportObjectResult` + (`{ name, definition, review }`). `ImportObjectOpts` extends `GenerateDraftOpts` + with `name` (override) + `writable` (object.external.writable opt-in; still + gated by datasource `external.allowWrites`, ADR Gate 3). + +2. **Service impl** (`external-datasource-service.ts`) reuses the `generateObjectDraft` + pipeline (type mapping + review notes + external binding), applies the + name/writable overrides, and persists via an injected `persistObject`. Throws + a descriptive error when no writable metadata store is wired (GitOps-only + deployment) and when the remote table is missing (before any write). + +3. **Plugin wiring** (`plugin.ts`) supplies `persistObject` โ†’ + `metadata.register('object', name, definition)` (runtime origin), alongside + the existing `persistCatalog`. + +4. **REST** โ€” `POST /api/v1/datasources/:name/external/tables/:remote/import` + (`packages/rest/src/external-datasource-routes.ts`): `201 { object }` on + success, `503` when the service is absent, `400 external_import_error` when + import is refused (read-only store / missing table). Body carries + `ImportObjectOpts`. + +5. **Tests** โ€” service: persists read-only by default, name+writable overrides, + draft-option forwarding (include/rename), throws without a store, throws on + missing table without persisting (47 green). + ### Follow-up notes / open items for later phases -- **DDL gate plumbing (P3)**: the runtime must inject `Datasource.schemaMode` - into `SqlDriverConfig` when constructing drivers. P1 wires the driver - side and defaults to `'managed'`; the runtime wiring lands with the - boot-validation plugin. +- **DDL gate plumbing (P3)**: โœ… done โ€” `createDefaultDatasourceDriverFactory` + injects `Datasource.schemaMode` into `SqlDriverConfig`. P1 wired the driver + side and defaulted to `'managed'`. - **`applyMigrations` gate**: `ISchemaDiffService.applyMigrations` also needs the gate (per ADR ยง5.1) when the migration runner ships. - **Lint rule** preventing plugins from bypassing the gate via raw `knex` diff --git a/packages/cli/package.json b/packages/cli/package.json index 0239b6f2d..80edae019 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -68,6 +68,7 @@ "@objectstack/service-analytics": "workspace:*", "@objectstack/service-automation": "workspace:*", "@objectstack/service-cache": "workspace:*", + "@objectstack/service-external-datasource": "workspace:*", "@objectstack/service-feed": "workspace:*", "@objectstack/service-job": "workspace:*", "@objectstack/service-package": "workspace:*", diff --git a/packages/cli/src/commands/serve.ts b/packages/cli/src/commands/serve.ts index 8d19272d9..37b3f1a8e 100644 --- a/packages/cli/src/commands/serve.ts +++ b/packages/cli/src/commands/serve.ts @@ -1537,6 +1537,80 @@ export default class Serve extends Command { } } + // โ”€โ”€ Runtime-UI datasource lifecycle (ADR-0015 Addendum) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + // Always available so the Studio "Add Datasource" wizard has a + // working backend: list (code + runtime origins), test a connection, + // and create / update / remove runtime datasources. Code-origin + // datasources stay read-only. The connection probe + hot pool use a + // default driver factory (postgres / sqlite / mongodb / memory) and + // the secret is wrapped into `sys_secret` via a fail-closed binder โ€” + // only a `credentialsRef` is ever persisted, never cleartext. + try { + const dsMod: any = await import('@objectstack/service-external-datasource'); + const { ExternalDatasourceServicePlugin, DatasourceAdminServicePlugin } = dsMod; + + if ( + ExternalDatasourceServicePlugin && + !hasPluginMatching(['service-external-datasource', 'ExternalDatasourceServicePlugin']) + ) { + await kernel.use(new ExternalDatasourceServicePlugin()); + trackPlugin('ExternalDatasourceServicePlugin'); + } + + if ( + DatasourceAdminServicePlugin && + !hasPluginMatching(['service-datasource-admin', 'DatasourceAdminServicePlugin']) + ) { + const rtMod: any = await import('@objectstack/runtime'); + const { createDefaultDatasourceDriverFactory, createDatasourceSecretBinder } = rtMod; + + // Fail-closed secret binder: when no crypto provider is available + // the admin service still loads, but secret-bearing create/update + // throws rather than persist cleartext. + let secrets: any; + try { + const { InMemoryCryptoProvider } = await import('@objectstack/service-settings'); + const cryptoProvider = new InMemoryCryptoProvider(); + const lazyEngine = { + insert: (o: string, d: any, opt?: any) => (kernel.getService('data') as any).insert(o, d, opt), + delete: (o: string, opt: any) => (kernel.getService('data') as any).delete(o, opt), + // Read path for boot rehydration: the secret binder dereferences + // each runtime datasource's `credentialsRef` from `sys_secret`. + find: (o: string, q: any) => (kernel.getService('data') as any).find(o, q), + }; + secrets = createDatasourceSecretBinder({ engine: lazyEngine, cryptoProvider }); + } catch { + /* no crypto provider โ€” admin service loads, secrets fail closed */ + } + + await kernel.use( + new DatasourceAdminServicePlugin({ + driverFactory: createDefaultDatasourceDriverFactory(), + ...(secrets ? { secrets } : {}), + }), + ); + trackPlugin('DatasourceAdminServicePlugin'); + } + + // Gate 2 (ADR-0015 ยง5.2): on kernel:ready, validate every federated + // object against its remote table and apply the datasource's + // `external.validation.onMismatch` policy. No-op when the + // `external-datasource` service isn't registered (federation unused). + const { createExternalValidationPlugin } = await import('@objectstack/runtime'); + if ( + createExternalValidationPlugin && + !hasPluginMatching(['external-validation', 'ExternalValidationPlugin']) + ) { + await kernel.use(createExternalValidationPlugin()); + trackPlugin('ExternalValidationPlugin'); + } + } catch (err: unknown) { + const msg = err instanceof Error ? err.message : String(err); + if (!msg.includes('Cannot find module') && !msg.includes('ERR_MODULE_NOT_FOUND')) { + console.error(`[Datasource] runtime-UI lifecycle wiring failed: ${msg}`); + } + } + // โ”€โ”€ UI portals โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ // In dev mode, the bundled Console portal is enabled by default // (use --no-ui to disable). Always serve the pre-built `dist/` โ€” no diff --git a/packages/metadata/src/metadata-manager.ts b/packages/metadata/src/metadata-manager.ts index 749598afc..9428b278f 100644 --- a/packages/metadata/src/metadata-manager.ts +++ b/packages/metadata/src/metadata-manager.ts @@ -327,6 +327,23 @@ export class MetadataManager implements IMetadataService { } } + /** + * Register a metadata item into the in-memory registry ONLY, never persisting + * to a writable loader. Used for GitOps-managed artefacts that must be + * *listable* (so `list(type)` returns them) but must never leak into the + * runtime DB store โ€” e.g. code-defined datasources (`origin:'code'`, ADR-0015 + * Addendum) declared in `*.datasource.ts` and owned by source control. Writing + * them through `register()` would persist them to `sys_metadata` and create + * drift between the artefact and the DB; this method avoids that. + */ + registerInMemory(type: string, name: string, data: unknown): void { + if (!this.registry.has(type)) { + this.registry.set(type, new Map()); + } + this.registry.get(type)!.set(name, data); + this.invalidateListCache(type); + } + /** * Get a metadata item by type and name. * Checks in-memory registry first, then falls back to loaders. diff --git a/packages/objectql/src/overlay-precedence.test.ts b/packages/objectql/src/overlay-precedence.test.ts index a13dce5e5..a57929cee 100644 --- a/packages/objectql/src/overlay-precedence.test.ts +++ b/packages/objectql/src/overlay-precedence.test.ts @@ -164,19 +164,19 @@ describe('overlay whitelist enforcement (shared-DB invariant)', () => { // blocked only when overlaying an artifact-backed item. Brand-new // (artifact-free) names succeed. Tested separately below. // - // 2. Types with `allowRuntimeCreate: false` (datasource/router/ - // function/service) โ€” blocked for ANY write in project-kernel mode. + // 2. Types with `allowRuntimeCreate: false` (router/function/service) โ€” + // blocked for ANY write in project-kernel mode. + // + // NOTE: `datasource` moved to cohort #1 with the ADR-0015 Addendum + // (runtime-UI-creatable datasources). Brand-new runtime datasources + // are now allowed; collision with a code-defined (artifact-backed) + // datasource is still refused via the artifact provenance check. // The error code surfaces as `not_creatable` when the item has no // artifact (which the empty test-mock registry guarantees) and // `not_overridable` when an artifact exists. Both carry status 403 // and the same underlying security guarantee. describe('denied โ€” must throw 403 (not_overridable or not_creatable)', () => { const deniedTypeWide: Array<{ type: string; reason: string; item: any }> = [ - { - type: 'datasource', - reason: 'wiring level; must be code, not per-org metadata', - item: { name: 'analytics', driver: 'sql' }, - }, { type: 'router', reason: 'API routing must be deterministic; per-org divergence creates invisible conflicts', @@ -192,11 +192,6 @@ describe('overlay whitelist enforcement (shared-DB invariant)', () => { reason: 'service definitions must be deployment-only, not per-org', item: { name: 'notification_service' }, }, - { - type: 'datasources', // plural โ€” `datasources` maps to `datasource` in PLURAL_TO_SINGULAR - reason: 'plural form of denied type must also be denied', - item: { name: 'analytics' }, - }, ]; for (const { type, reason, item } of deniedTypeWide) { @@ -248,6 +243,18 @@ describe('overlay whitelist enforcement (shared-DB invariant)', () => { type: 'field', item: { name: 'tenant_widget_color', type: 'text', label: 'Color' }, }, + // datasource/datasources became runtime-creatable with the + // ADR-0015 Addendum (UI "Add Datasource"). Brand-new runtime + // datasources succeed; code-defined collisions are refused via + // artifact provenance (exercised in protocol-meta.test.ts). + { + type: 'datasource', + item: { name: 'analytics', driver: 'sql', config: {} }, + }, + { + type: 'datasources', // plural โ€” maps to `datasource` via PLURAL_TO_SINGULAR + item: { name: 'analytics2', driver: 'sql', config: {} }, + }, ]; for (const { type, item } of runtimeCreatable) { diff --git a/packages/plugins/plugin-hono-server/src/adapter.ts b/packages/plugins/plugin-hono-server/src/adapter.ts index 5d5f06c6e..f57ee4c2b 100644 --- a/packages/plugins/plugin-hono-server/src/adapter.ts +++ b/packages/plugins/plugin-hono-server/src/adapter.ts @@ -108,6 +108,18 @@ export class HonoHttpServer implements IHttpServer { let streamEncoder: TextEncoder | null = null; let streamHeaders: Record = {}; let isStreaming = false; + let streamClosed = false; + + // The unused stream is always created (see below) and may be closed + // from two places โ€” `res.end()` and the post-handler cleanup โ€” so + // guard against the double-close that crashes the event loop with + // `ERR_INVALID_STATE: Controller is already closed`. + const closeStream = () => { + if (streamController && !streamClosed) { + streamClosed = true; + try { streamController.close(); } catch { /* already closed */ } + } + }; const res = { json: (data: any) => { capturedResponse = c.json(data); }, @@ -133,9 +145,14 @@ export class HonoHttpServer implements IHttpServer { } }, end: () => { - if (streamController) { - streamController.close(); + // Body-less response (e.g. 204 No Content) honoring any + // status already set via `res.status()`. A null body avoids + // the undici "Invalid response status code 204" thrown when + // an empty *string* body is paired with a null-body status. + if (!isStreaming && capturedResponse === undefined) { + capturedResponse = c.body(null); } + closeStream(); }, }; @@ -160,11 +177,11 @@ export class HonoHttpServer implements IHttpServer { })); } else { // Not streaming โ€” close the unused stream and return null - streamController?.close(); + closeStream(); resolve(null); } }).catch((err) => { - streamController?.close(); + closeStream(); resolve(null); }); }); diff --git a/packages/rest/src/external-datasource-routes.ts b/packages/rest/src/external-datasource-routes.ts index 9442401e7..52076f1a3 100644 --- a/packages/rest/src/external-datasource-routes.ts +++ b/packages/rest/src/external-datasource-routes.ts @@ -13,6 +13,7 @@ import type { IHttpServer } from '@objectstack/spec/contracts'; * * GET /datasources/:name/external/tables โ†’ listRemoteTables * POST /datasources/:name/external/tables/:remote/draft โ†’ generateObjectDraft + * POST /datasources/:name/external/tables/:remote/import โ†’ importObject * POST /datasources/:name/external/refresh-catalog โ†’ refreshCatalog * POST /datasources/:name/external/validate โ†’ validateAll (this ds) */ @@ -55,6 +56,28 @@ export function registerExternalDatasourceRoutes( res.json({ draft }); }); + // Import a remote table as a live (runtime-origin) federated object so it's + // immediately queryable โ€” the "Import as Object" action (ADR-0015 Addendum). + // 503 when the service is absent; 400 when import is refused (e.g. read-only + // metadata store) or the remote table is missing. + server.post(`${ext}/tables/:remote/import`, async (req: any, res: any) => { + const svc = externalService(); + if (!svc?.importObject) return unavailable(res); + try { + const result = await svc.importObject( + req.params.name, + req.params.remote, + (req.body as Record) ?? {}, + ); + res.status(201).json({ object: result }); + } catch (err) { + res.status(400).json({ + error: 'external_import_error', + message: err instanceof Error ? err.message : String(err), + }); + } + }); + // Refresh and return the cached catalog snapshot. server.post(`${ext}/refresh-catalog`, async (req: any, res: any) => { const svc = externalService(); @@ -72,3 +95,116 @@ export function registerExternalDatasourceRoutes( res.json({ ok: results.every((r: any) => r.ok), results }); }); } + +/** + * Datasource lifecycle REST routes (ADR-0015 Addendum ยง3.5). + * + * Mounted under `/api/v1/datasources` and served by the `datasource-admin` + * service. Like the federation routes, every route degrades gracefully + * (`503 datasource_admin_unavailable`) when the service is not wired in, and + * lifecycle/validation failures surface as `400` with the service's message. + * + * GET /datasources โ†’ listDatasources (provenance + health) + * POST /datasources/test โ†’ testConnection (no persistence) + * POST /datasources โ†’ createDatasource (origin: 'runtime') + * PATCH /datasources/:name โ†’ updateDatasource (runtime only) + * DELETE /datasources/:name โ†’ removeDatasource (runtime only) + * + * Request bodies carry the connection draft inline with an optional cleartext + * `secret` field; the route splits `secret` out so it never reaches the draft + * the service persists. + */ +export function registerDatasourceAdminRoutes( + server: IHttpServer, + ctx: PluginContext, + basePath = '/api/v1', +): void { + const root = `${basePath}/datasources`; + + const adminService = (): any => { + try { + return ctx.getService('datasource-admin'); + } catch { + return undefined; + } + }; + + const unavailable = (res: any) => + res.status(503).json({ error: 'datasource_admin_unavailable' }); + + const badRequest = (res: any, err: unknown) => + res.status(400).json({ error: 'datasource_admin_error', message: err instanceof Error ? err.message : String(err) }); + + /** Split an inline `{ secret, ...draft }` body into (draft, secret). */ + const splitSecret = (body: any): { draft: any; secret: any } => { + const { secret, ...draft } = (body as Record) ?? {}; + // Accept either a bare string or a `{ value, namespace?, key? }` object. + const normalised = + secret == null + ? undefined + : typeof secret === 'string' + ? { value: secret } + : secret; + return { draft, secret: normalised }; + }; + + // List all datasources with provenance + health. + server.get(root, async (_req: any, res: any) => { + const svc = adminService(); + if (!svc?.listDatasources) return unavailable(res); + const datasources = await svc.listDatasources(); + res.json({ datasources }); + }); + + // Probe a connection without persisting anything. Registered before the + // `:name` routes so the literal `test` segment is never captured as a name. + server.post(`${root}/test`, async (req: any, res: any) => { + const svc = adminService(); + if (!svc?.testConnection) return unavailable(res); + const { draft, secret } = splitSecret(req.body); + try { + const result = await svc.testConnection(draft, secret); + res.json({ result }); + } catch (err) { + badRequest(res, err); + } + }); + + // Create a runtime datasource. + server.post(root, async (req: any, res: any) => { + const svc = adminService(); + if (!svc?.createDatasource) return unavailable(res); + const { draft, secret } = splitSecret(req.body); + try { + const datasource = await svc.createDatasource(draft, secret); + res.status(201).json({ datasource }); + } catch (err) { + badRequest(res, err); + } + }); + + // Patch a runtime datasource. + server.patch(`${root}/:name`, async (req: any, res: any) => { + const svc = adminService(); + if (!svc?.updateDatasource) return unavailable(res); + const { draft, secret } = splitSecret(req.body); + try { + const datasource = await svc.updateDatasource(req.params.name, draft, secret); + res.json({ datasource }); + } catch (err) { + badRequest(res, err); + } + }); + + // Remove a runtime datasource. + server.delete(`${root}/:name`, async (req: any, res: any) => { + const svc = adminService(); + if (!svc?.removeDatasource) return unavailable(res); + try { + await svc.removeDatasource(req.params.name); + res.status(204).end(); + } catch (err) { + badRequest(res, err); + } + }); +} diff --git a/packages/rest/src/rest-api-plugin.ts b/packages/rest/src/rest-api-plugin.ts index 473d98953..0c4a2c0dd 100644 --- a/packages/rest/src/rest-api-plugin.ts +++ b/packages/rest/src/rest-api-plugin.ts @@ -4,7 +4,7 @@ import { Plugin, PluginContext, IHttpServer } from '@objectstack/core'; import { RestServer, RestKernelManager } from './rest-server.js'; import { ObjectStackProtocol, RestServerConfig } from '@objectstack/spec/api'; import { registerPackageRoutes } from './package-routes.js'; -import { registerExternalDatasourceRoutes } from './external-datasource-routes.js'; +import { registerExternalDatasourceRoutes, registerDatasourceAdminRoutes } from './external-datasource-routes.js'; import type { PackageService } from '@objectstack/service-package'; export interface RestApiPluginConfig { @@ -176,16 +176,16 @@ export function createRestApiPlugin(config: RestApiPluginConfig = {}): Plugin { throw err; } - // Register package management routes if service is available + const basePath = config.api?.api?.basePath || '/api'; + const version = config.api?.api?.version || 'v1'; + const versionedBase = `${basePath}/${version}`; + const enableProjectScoping = config.api?.api?.enableProjectScoping ?? false; + const projectResolution = config.api?.api?.projectResolution ?? 'auto'; + + // Register package management routes if the service is available. try { const packageService = ctx.getService('package'); if (packageService) { - const basePath = config.api?.api?.basePath || '/api'; - const version = config.api?.api?.version || 'v1'; - const versionedBase = `${basePath}/${version}`; - const enableProjectScoping = config.api?.api?.enableProjectScoping ?? false; - const projectResolution = config.api?.api?.projectResolution ?? 'auto'; - if (enableProjectScoping && projectResolution === 'required') { // Only register the scoped variant registerPackageRoutes(server, packageService, `${versionedBase}/environments/:environmentId`, { @@ -193,9 +193,6 @@ export function createRestApiPlugin(config: RestApiPluginConfig = {}): Plugin { }); } else { registerPackageRoutes(server, packageService, versionedBase, { protocol }); - // External Datasource Federation routes (ADR-0015) โ€” - // degrade gracefully when the service is not registered. - registerExternalDatasourceRoutes(server, ctx, versionedBase); if (enableProjectScoping) { registerPackageRoutes(server, packageService, `${versionedBase}/environments/:environmentId`, { protocol, @@ -208,6 +205,19 @@ export function createRestApiPlugin(config: RestApiPluginConfig = {}): Plugin { // Package service not available, skip ctx.logger.debug('Package service not available, package routes skipped'); } + + // Datasource routes do NOT depend on the package service โ€” register + // them unconditionally. Each degrades gracefully (503) when its + // backing service is absent. + // โ€ข External Datasource Federation (ADR-0015): catalog / draft / validate. + // โ€ข Datasource lifecycle (ADR-0015 Addendum): list / test / create / update / remove. + try { + registerExternalDatasourceRoutes(server, ctx, versionedBase); + registerDatasourceAdminRoutes(server, ctx, versionedBase); + ctx.logger.info('Datasource routes registered'); + } catch (e: any) { + ctx.logger.warn('Datasource routes registration failed', { error: e?.message }); + } } }; } diff --git a/packages/runtime/src/app-plugin.ts b/packages/runtime/src/app-plugin.ts index a29186572..ec35278f3 100644 --- a/packages/runtime/src/app-plugin.ts +++ b/packages/runtime/src/app-plugin.ts @@ -205,6 +205,41 @@ export class AppPlugin implements Plugin { ql.setDatasourceMapping(this.bundle.datasourceMapping); } + // Surface code-defined datasources (ADR-0015 Addendum) in the metadata + // registry so the datasource-admin list returns them alongside any + // UI-created (`origin:'runtime'`) ones. These are GitOps-managed + // (declared in `*.datasource.ts`), so they are registered IN MEMORY + // ONLY โ€” never persisted to the runtime DB store โ€” and stamped + // `origin:'code'` so the admin service enforces them as read-only. + // The engine already indexed them for the write gate via registerApp(). + try { + const dsDefs = this.bundle.datasources; + const dsList = Array.isArray(dsDefs) + ? dsDefs + : dsDefs && typeof dsDefs === 'object' + ? Object.entries(dsDefs).map(([name, def]) => ({ name, ...(def as any) })) + : []; + if (dsList.length > 0) { + const metadata = ctx.getService('metadata') as + | { registerInMemory?: (t: string, n: string, d: unknown) => void } + | undefined; + if (typeof metadata?.registerInMemory === 'function') { + for (const ds of dsList) { + if (!ds?.name) continue; + metadata.registerInMemory('datasource', ds.name, { ...ds, origin: 'code' }); + } + ctx.logger.info('Registered code-defined datasources in metadata registry', { + appId, + count: dsList.length, + }); + } + } + } catch (err) { + ctx.logger.warn('[AppPlugin] failed to register code-defined datasources', { + error: (err as Error)?.message ?? String(err), + }); + } + // Resolve the runtime hook owner. Modules that declare both a // `default` (defineStack(...)) export and a named `onEnable` export // hide the named export from `bundle.default`, so we fall back to the diff --git a/packages/runtime/src/datasource-secret-binder.test.ts b/packages/runtime/src/datasource-secret-binder.test.ts new file mode 100644 index 000000000..4cc2e5782 --- /dev/null +++ b/packages/runtime/src/datasource-secret-binder.test.ts @@ -0,0 +1,101 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import type { CryptoContext, CryptoHandle, ICryptoProvider } from '@objectstack/spec/contracts'; +import { + createDatasourceSecretBinder, + parseCredentialsRef, + toCredentialsRef, + type SecretStoreEngineLike, +} from './datasource-secret-binder.js'; + +/** + * Minimal AAD-binding crypto fake: ciphertext = base64(`${ns}|${key}::${plain}`). + * decrypt() verifies the (namespace,key) AAD matches what encrypt() sealed โ€” + * mirroring InMemoryCryptoProvider's guarantee without pulling in node:crypto. + */ +function fakeCrypto(): ICryptoProvider { + return { + async encrypt(plain: string, ctx: CryptoContext): Promise { + return { + id: 'sec_' + ctx.key, + kmsKeyId: 'local:test:v1', + alg: 'aes-256-gcm', + version: 1, + ciphertext: Buffer.from(`${ctx.namespace}|${ctx.key}::${plain}`, 'utf8').toString('base64'), + }; + }, + async decrypt(handle: CryptoHandle, ctx: CryptoContext): Promise { + const raw = Buffer.from(handle.ciphertext, 'base64').toString('utf8'); + const [aad, plain] = raw.split('::'); + if (aad !== `${ctx.namespace}|${ctx.key}`) throw new Error('AAD mismatch'); + return plain; + }, + async rotateKey(handle: CryptoHandle): Promise { + return handle; + }, + digest: (plain: string) => 'sha256:' + plain, + }; +} + +/** In-memory `sys_secret` store backing the engine surface. */ +function fakeEngine(): SecretStoreEngineLike & { rows: Map } { + const rows = new Map(); + return { + rows, + async insert(_object, data) { + rows.set(String(data.id), data); + return data; + }, + async delete(_object, options) { + rows.delete(String(options.where.id)); + return undefined; + }, + async find(_object, query) { + const id = String((query.where as any)?.id); + const row = rows.get(id); + return row ? [row] : []; + }, + }; +} + +describe('createDatasourceSecretBinder', () => { + it('round-trips: bind โ†’ credentialsRef โ†’ resolve back to cleartext', async () => { + const engine = fakeEngine(); + const binder = createDatasourceSecretBinder({ engine, cryptoProvider: fakeCrypto() }); + + const ref = await binder.bind({ value: 'super-secret-pw' }, { name: 'reporting' }); + expect(ref).toBe(toCredentialsRef('sec_reporting')); + + // The persisted row holds only ciphertext โ€” never the cleartext. + const row = engine.rows.get('sec_reporting'); + expect(row.namespace).toBe('datasource'); + expect(row.key).toBe('reporting'); + expect(JSON.stringify(row)).not.toContain('super-secret-pw'); + + expect(await binder.resolve(ref)).toBe('super-secret-pw'); + }); + + it('resolve() returns undefined after unbind (row gone)', async () => { + const engine = fakeEngine(); + const binder = createDatasourceSecretBinder({ engine, cryptoProvider: fakeCrypto() }); + const ref = await binder.bind({ value: 'pw' }, { name: 'ds1' }); + await binder.unbind(ref); + expect(await binder.resolve(ref)).toBeUndefined(); + }); + + it('resolve() returns undefined for a foreign / non-sys_secret ref', async () => { + const engine = fakeEngine(); + const binder = createDatasourceSecretBinder({ engine, cryptoProvider: fakeCrypto() }); + expect(parseCredentialsRef('vault://other/handle')).toBeUndefined(); + expect(await binder.resolve('vault://other/handle')).toBeUndefined(); + }); + + it('resolve() degrades to undefined when the engine cannot read', async () => { + const engine = fakeEngine(); + delete (engine as any).find; // older engine surface without a read path + const binder = createDatasourceSecretBinder({ engine, cryptoProvider: fakeCrypto() }); + const ref = await binder.bind({ value: 'pw' }, { name: 'ds1' }); + expect(await binder.resolve(ref)).toBeUndefined(); + }); +}); diff --git a/packages/runtime/src/datasource-secret-binder.ts b/packages/runtime/src/datasource-secret-binder.ts new file mode 100644 index 000000000..7b4f87581 --- /dev/null +++ b/packages/runtime/src/datasource-secret-binder.ts @@ -0,0 +1,144 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Default datasource SecretBinder โ€” persists a runtime datasource's cleartext + * credential into the `sys_secret` cipher store and returns an opaque + * `credentialsRef` handle (ADR-0015 Addendum, security invariant). + * + * Mirrors the SettingsService Phase-3 split: the cleartext is wrapped by an + * {@link ICryptoProvider} into a {@link CryptoHandle}, the ciphertext lands in a + * `sys_secret` row keyed by `handle.id`, and only the handle id (wrapped as + * `sys_secret:`) is ever stored on the datasource artefact. Cleartext never + * touches metadata. + * + * This is the dev/self-host wiring; production hosts swap the + * `InMemoryCryptoProvider` for a KMS-backed `ICryptoProvider` and pass it here. + */ + +import type { CryptoHandle, ICryptoProvider } from '@objectstack/spec/contracts'; + +/** Prefix used to recognise a datasource credential handle. */ +const REF_PREFIX = 'sys_secret:'; + +/** A persisted `sys_secret` row (subset used to reconstruct a {@link CryptoHandle}). */ +interface SecretRow { + id: string; + namespace: string; + key: string; + kms_key_id: string; + alg: string; + version: number; + ciphertext: string; +} + +/** Minimal data-engine surface used to read/write the `sys_secret` store. */ +export interface SecretStoreEngineLike { + insert(object: string, data: Record, options?: unknown): Promise; + delete(object: string, options: { where: Record }): Promise; + /** + * Read `sys_secret` rows for the `resolve()` path. Optional so existing + * callers that only bind/unbind keep working; `resolve()` no-ops when absent. + * Mirrors `IDataEngine.find` โ€” returns an array (or `{ data: [...] }`). + */ + find?(object: string, query: Record): Promise; +} + +export interface DatasourceSecretBinderDeps { + /** Data engine (ObjectQL) used to persist the `sys_secret` row. */ + engine: SecretStoreEngineLike; + /** Crypto provider that wraps cleartext into a {@link CryptoHandle}. */ + cryptoProvider: ICryptoProvider; + /** Settings namespace recorded on the secret row (default `'datasource'`). */ + namespace?: string; +} + +export interface DatasourceSecretBinder { + bind(input: { value: string; namespace?: string; key?: string }, hint: { name: string }): Promise; + unbind(credentialsRef: string): Promise; + /** + * Dereference a `credentialsRef` back to its cleartext credential by reading + * the `sys_secret` row and decrypting it. Used at boot to rebuild a runtime + * datasource's live connection pool (the cleartext is never persisted, so it + * must be recovered from the cipher store). Returns `undefined` when the ref + * isn't ours, the row is gone, the engine can't read, or decryption fails + * (e.g. an ephemeral dev key changed across restarts) โ€” callers degrade to + * skipping that pool rather than crashing boot. + */ + resolve(credentialsRef: string): Promise; +} + +/** Build a `credentialsRef` from a crypto handle id. */ +export function toCredentialsRef(handleId: string): string { + return `${REF_PREFIX}${handleId}`; +} + +/** Extract the `sys_secret` handle id from a credentialsRef, if it is one. */ +export function parseCredentialsRef(ref: string): string | undefined { + return ref?.startsWith(REF_PREFIX) ? ref.slice(REF_PREFIX.length) : undefined; +} + +/** + * Create the default datasource secret binder. Persists into `sys_secret` via + * the data engine and never returns or logs the cleartext. + */ +export function createDatasourceSecretBinder(deps: DatasourceSecretBinderDeps): DatasourceSecretBinder { + const { engine, cryptoProvider } = deps; + const defaultNamespace = deps.namespace ?? 'datasource'; + + return { + async bind(input, hint) { + const namespace = input.namespace ?? defaultNamespace; + const key = input.key ?? hint.name; + const handle: CryptoHandle = await cryptoProvider.encrypt(input.value, { namespace, key }); + await engine.insert('sys_secret', { + id: handle.id, + namespace, + key, + kms_key_id: handle.kmsKeyId, + alg: handle.alg, + version: handle.version, + ciphertext: handle.ciphertext, + }); + return toCredentialsRef(handle.id); + }, + + async unbind(credentialsRef) { + const id = parseCredentialsRef(credentialsRef); + if (!id) return; // not ours (or already cleared) โ€” nothing to do + await engine.delete('sys_secret', { where: { id } }); + }, + + async resolve(credentialsRef) { + const id = parseCredentialsRef(credentialsRef); + if (!id || typeof engine.find !== 'function') return undefined; + try { + const result = await engine.find('sys_secret', { + where: { id }, + limit: 1, + // Secrets are scoped through their owning datasource artefact, so + // skip the tenant-audit warning (mirrors SettingsService's store). + bypassTenantAudit: true, + }); + const rows = (Array.isArray(result) ? result : (result as { data?: unknown[] })?.data) ?? []; + const row = rows[0] as SecretRow | undefined; + if (!row?.ciphertext) return undefined; + // Reconstruct the handle and decrypt under the same (namespace,key) + // AAD the row was sealed with โ€” a mismatch fails authentication. + return await cryptoProvider.decrypt( + { + id: row.id, + kmsKeyId: row.kms_key_id, + alg: row.alg, + version: row.version, + ciphertext: row.ciphertext, + }, + { namespace: row.namespace, key: row.key }, + ); + } catch { + // Missing row / unreadable engine / decrypt failure (e.g. rotated dev + // key) โ€” never block boot; the pool is simply not rehydrated. + return undefined; + } + }, + }; +} diff --git a/packages/runtime/src/default-datasource-driver-factory.ts b/packages/runtime/src/default-datasource-driver-factory.ts new file mode 100644 index 000000000..8808d690b --- /dev/null +++ b/packages/runtime/src/default-datasource-driver-factory.ts @@ -0,0 +1,185 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Default (dev/self-host) implementation of {@link IDatasourceDriverFactory}. + * + * The framework ships no universal "driver-by-id" registry โ€” concrete drivers + * are constructed by the host stack (ADR-0015 Addendum ยง3.5). This factory is + * the host-side glue that lets the runtime-datasource lifecycle + * (`IDatasourceAdminService`) build a live driver from an *unsaved* draft so it + * can probe a connection before "Save" and hot-register a pool afterwards. + * + * Supported driver ids map onto the same open-core drivers the standalone + * stack auto-detects: + * - `postgres` / `pg` / `postgresql` โ†’ `@objectstack/driver-sql` (client `pg`) + * - `sqlite` / `sqlite3` โ†’ `@objectstack/driver-sql` (better-sqlite3) + * - `mongodb` / `mongo` โ†’ `@objectstack/driver-mongodb` (peer dep) + * - `memory` / `inmemory` โ†’ `@objectstack/driver-memory` + * + * Anything else returns `supports() === false`, so the admin service degrades + * gracefully (testConnection โ†’ `{ ok: false }`, create skips hot pool reg). + * + * SECURITY: the cleartext `spec.secret` is used only to open the connection and + * is never persisted or logged here. + */ + +import type { + IDatasourceDriverFactory, + DatasourceConnectionSpec, + DatasourceDriverHandle, +} from '@objectstack/spec/contracts'; + +type ResolvedKind = 'postgres' | 'sqlite' | 'mongodb' | 'memory'; + +const DRIVER_ID_ALIASES: Record = { + postgres: 'postgres', + postgresql: 'postgres', + pg: 'postgres', + sqlite: 'sqlite', + sqlite3: 'sqlite', + 'better-sqlite3': 'sqlite', + mongodb: 'mongodb', + mongo: 'mongodb', + memory: 'memory', + inmemory: 'memory', + 'in-memory': 'memory', +}; + +function resolveKind(driverId: string): ResolvedKind | undefined { + return DRIVER_ID_ALIASES[String(driverId ?? '').toLowerCase()]; +} + +/** + * Wrap a concrete engine driver in a probe handle. `ping`/`checkHealth` reuse + * the driver's own health check; `driver` is the escape hatch the admin service + * hands to `registerDriver()`. + */ +function toHandle(driver: any, serverVersion?: () => Promise): DatasourceDriverHandle { + return { + connect: typeof driver?.connect === 'function' ? () => driver.connect() : undefined, + disconnect: typeof driver?.disconnect === 'function' ? () => driver.disconnect() : undefined, + checkHealth: typeof driver?.checkHealth === 'function' ? () => driver.checkHealth() : undefined, + ping: typeof driver?.checkHealth === 'function' ? () => driver.checkHealth() : undefined, + ...(serverVersion ? { serverVersion } : {}), + driver, + }; +} + +/** Build the Knex `connection` for a SQL driver from a spec's config + secret. */ +function buildSqlConnection(spec: DatasourceConnectionSpec, client: 'pg' | 'better-sqlite3'): unknown { + const cfg = (spec.config ?? {}) as Record; + + if (client === 'better-sqlite3') { + const filename = + (cfg.filename as string | undefined) ?? + (cfg.file as string | undefined) ?? + (cfg.database as string | undefined) ?? + ':memory:'; + return { filename }; + } + + // pg โ€” accept either a connection string (`url`/`connectionString`) or + // discrete fields. The secret is the password and is never part of `config`. + const url = (cfg.url as string | undefined) ?? (cfg.connectionString as string | undefined); + if (url) { + // For a DSN, a separately-supplied secret overrides the embedded password. + return spec.secret ? { connectionString: url, password: spec.secret } : { connectionString: url }; + } + return { + host: cfg.host, + port: cfg.port, + database: cfg.database, + user: cfg.user ?? cfg.username, + ...(spec.secret ? { password: spec.secret } : cfg.password ? { password: cfg.password } : {}), + ...(cfg.ssl != null ? { ssl: cfg.ssl } : {}), + }; +} + +/** Build a mongodb connection URL from a spec's config + secret. */ +function buildMongoUrl(spec: DatasourceConnectionSpec): string { + const cfg = (spec.config ?? {}) as Record; + const explicit = (cfg.url as string | undefined) ?? (cfg.uri as string | undefined); + if (explicit) return explicit; + const host = (cfg.host as string | undefined) ?? 'localhost'; + const port = (cfg.port as number | string | undefined) ?? 27017; + const db = (cfg.database as string | undefined) ?? ''; + const user = (cfg.user as string | undefined) ?? (cfg.username as string | undefined); + const auth = user ? `${encodeURIComponent(user)}:${encodeURIComponent(spec.secret ?? '')}@` : ''; + return `mongodb://${auth}${host}:${port}/${db}`; +} + +/** + * Create the default datasource driver factory. Driver packages are imported + * lazily so a host that never builds (e.g.) a mongo connection doesn't pay for + * the mongo SDK. + */ +export function createDefaultDatasourceDriverFactory(): IDatasourceDriverFactory { + return { + supports(driverId: string): boolean { + return resolveKind(driverId) !== undefined; + }, + + async create(spec: DatasourceConnectionSpec): Promise { + const kind = resolveKind(spec.driver); + if (!kind) { + throw new Error(`Unsupported driver id '${spec.driver}'.`); + } + + const schemaMode = (spec.external as { schemaMode?: string } | undefined)?.schemaMode + ?? ((spec.config as Record | undefined)?.schemaMode as string | undefined); + + if (kind === 'postgres') { + const { SqlDriver } = await import('@objectstack/driver-sql'); + const driver = new SqlDriver({ + client: 'pg', + connection: buildSqlConnection(spec, 'pg') as any, + pool: { min: 0, max: 5 }, + ...(schemaMode ? { schemaMode: schemaMode as any } : {}), + } as any); + return toHandle(driver, () => sqlServerVersion(driver, 'pg')); + } + + if (kind === 'sqlite') { + const { SqlDriver } = await import('@objectstack/driver-sql'); + const driver = new SqlDriver({ + client: 'better-sqlite3', + connection: buildSqlConnection(spec, 'better-sqlite3') as any, + useNullAsDefault: true, + ...(schemaMode ? { schemaMode: schemaMode as any } : {}), + } as any); + return toHandle(driver, () => sqlServerVersion(driver, 'sqlite')); + } + + if (kind === 'mongodb') { + let MongoDBDriver: any; + try { + ({ MongoDBDriver } = await import('@objectstack/driver-mongodb' as any)); + } catch (err: any) { + throw new Error( + `mongodb driver requested but @objectstack/driver-mongodb is not installed (${err?.message ?? err}).`, + ); + } + const driver = new MongoDBDriver({ url: buildMongoUrl(spec) }); + return toHandle(driver); + } + + // memory + const { InMemoryDriver } = await import('@objectstack/driver-memory'); + return toHandle(new InMemoryDriver()); + }, + }; +} + +/** Best-effort server version via a raw query; swallows everything. */ +async function sqlServerVersion(driver: any, client: 'pg' | 'sqlite'): Promise { + if (typeof driver?.execute !== 'function') return undefined; + try { + const sql = client === 'pg' ? 'SELECT version() AS v' : 'SELECT sqlite_version() AS v'; + const rows: any = await driver.execute(sql); + const first = Array.isArray(rows) ? rows[0] : Array.isArray(rows?.rows) ? rows.rows[0] : rows; + const v = first?.v ?? first?.version ?? first?.['sqlite_version()']; + return typeof v === 'string' ? v : undefined; + } catch { + return undefined; + } +} diff --git a/packages/runtime/src/external-validation-plugin.test.ts b/packages/runtime/src/external-validation-plugin.test.ts index 0d82bf529..edbf10037 100644 --- a/packages/runtime/src/external-validation-plugin.test.ts +++ b/packages/runtime/src/external-validation-plugin.test.ts @@ -1,6 +1,6 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. -import { describe, it, expect, vi } from 'vitest'; +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; import { ExternalValidationPlugin } from './external-validation-plugin'; import { ExternalSchemaMismatchError, type SchemaDiffEntry } from '@objectstack/spec/shared'; @@ -81,3 +81,103 @@ describe('ExternalValidationPlugin (ADR-0015 Gate 2)', () => { await expect(new ExternalValidationPlugin().runValidation(ctx)).rejects.toBeInstanceOf(ExternalSchemaMismatchError); }); }); + +describe('ExternalValidationPlugin โ€” background drift detection (ADR-0015 ยง5.2)', () => { + beforeEach(() => vi.useFakeTimers()); + afterEach(() => vi.useRealTimers()); + + it('runDriftCheck emits one external.schema.drift event per drifted object', async () => { + const { ctx } = makeCtx({ + 'external-datasource': { + validateAll: async () => ({ + ok: false, + results: [ + { ok: false, datasource: 'warehouse', object: 'wh_order', diffs: sampleDiffs }, + { ok: true, datasource: 'warehouse', object: 'wh_ok', diffs: [] }, + // A failure on a *different* datasource must not bleed into warehouse's check. + { ok: false, datasource: 'other', object: 'x', diffs: sampleDiffs }, + ], + }), + }, + }); + const emitted = await new ExternalValidationPlugin().runDriftCheck(ctx, 'warehouse'); + expect(emitted).toBe(1); + expect(ctx.trigger).toHaveBeenCalledTimes(1); + expect(ctx.trigger).toHaveBeenCalledWith('external.schema.drift', { + datasource: 'warehouse', + object: 'wh_order', + diffs: sampleDiffs, + }); + }); + + it('runDriftCheck is a no-op (no throw) when validateAll rejects', async () => { + const { ctx, warnings } = makeCtx({ + 'external-datasource': { validateAll: async () => { throw new Error('remote unreachable'); } }, + }); + const emitted = await new ExternalValidationPlugin().runDriftCheck(ctx, 'warehouse'); + expect(emitted).toBe(0); + expect(ctx.trigger).not.toHaveBeenCalled(); + expect(warnings.length).toBeGreaterThan(0); + }); + + it('schedules a timer only for datasources declaring checkIntervalMs', async () => { + const { ctx } = makeCtx({ + 'external-datasource': { validateAll: async () => ({ ok: true, results: [] }) }, + metadata: { + list: async () => [ + { name: 'warehouse', external: { validation: { checkIntervalMs: 60_000 } } }, + { name: 'replica', external: { validation: {} } }, // no interval โ†’ skipped + { name: 'local' }, // not federated โ†’ skipped + ], + }, + }); + const plugin = new ExternalValidationPlugin(); + await plugin.scheduleDriftChecks(ctx); + expect(vi.getTimerCount()).toBe(1); + plugin.stop(); + expect(vi.getTimerCount()).toBe(0); + }); + + it('the armed timer fires runDriftCheck on its interval and emits drift', async () => { + const { ctx } = makeCtx({ + 'external-datasource': { + validateAll: async () => ({ ok: false, results: [{ ok: false, datasource: 'warehouse', object: 'wh_order', diffs: sampleDiffs }] }), + }, + metadata: { + list: async () => [{ name: 'warehouse', external: { validation: { checkIntervalMs: 1000 } } }], + }, + }); + const plugin = new ExternalValidationPlugin(); + await plugin.scheduleDriftChecks(ctx); + expect(ctx.trigger).not.toHaveBeenCalled(); + // Advance past one interval and flush the fire-and-forget async work. + await vi.advanceTimersByTimeAsync(1000); + expect(ctx.trigger).toHaveBeenCalledWith('external.schema.drift', expect.objectContaining({ + datasource: 'warehouse', + object: 'wh_order', + })); + plugin.stop(); + }); + + it('re-arming clears prior timers so intervals do not accumulate', async () => { + const { ctx } = makeCtx({ + 'external-datasource': { validateAll: async () => ({ ok: true, results: [] }) }, + metadata: { + list: async () => [{ name: 'warehouse', external: { validation: { checkIntervalMs: 1000 } } }], + }, + }); + const plugin = new ExternalValidationPlugin(); + await plugin.scheduleDriftChecks(ctx); + await plugin.scheduleDriftChecks(ctx); + expect(vi.getTimerCount()).toBe(1); + plugin.stop(); + }); + + it('is a no-op when metadata cannot enumerate datasources', async () => { + const { ctx } = makeCtx({ + 'external-datasource': { validateAll: async () => ({ ok: true, results: [] }) }, + }); + await expect(new ExternalValidationPlugin().scheduleDriftChecks(ctx)).resolves.toBeUndefined(); + expect(vi.getTimerCount()).toBe(0); + }); +}); diff --git a/packages/runtime/src/external-validation-plugin.ts b/packages/runtime/src/external-validation-plugin.ts index 6f6f06cae..4057bb970 100644 --- a/packages/runtime/src/external-validation-plugin.ts +++ b/packages/runtime/src/external-validation-plugin.ts @@ -19,11 +19,29 @@ interface ExternalDatasourceServiceLike { interface MetadataServiceLike { get?: (type: string, name: string) => Promise; + list?: (type: string) => Promise; } interface DatasourceDef { + name?: string; schemaMode?: string; - external?: { validation?: { onMismatch?: 'fail' | 'warn' | 'ignore' } }; + external?: { + validation?: { + onMismatch?: 'fail' | 'warn' | 'ignore'; + checkIntervalMs?: number; + }; + }; +} + +/** + * Payload of the `external.schema.drift` event emitted on the kernel bus by the + * background drift checker (ADR-0015 ยง5.2). Consumed by `audit` / `notification` + * services. One event per drifted federated object. + */ +export interface ExternalSchemaDriftEvent { + datasource: string; + object: string; + diffs: SchemaDiffEntry[]; } /** @@ -44,6 +62,9 @@ export class ExternalValidationPlugin implements Plugin { type = 'standard'; version = '1.0.0'; + /** Active background drift-check timers, keyed by datasource name. */ + private driftTimers = new Map>(); + init = (_ctx: PluginContext): void => { // Nothing to register; validation runs on kernel:ready (see start()). }; @@ -53,9 +74,17 @@ export class ExternalValidationPlugin implements Plugin { // services, manifests) has been registered. ctx.hook('kernel:ready', async () => { await this.runValidation(ctx); + // Boot validation done; arm any background drift checks (ADR-0015 ยง5.2). + await this.scheduleDriftChecks(ctx); }); }; + /** Tear down background drift-check timers (idempotent). */ + stop = (): void => { + for (const timer of this.driftTimers.values()) clearInterval(timer); + this.driftTimers.clear(); + }; + /** Exposed for testing; invoked from the kernel:ready handler. */ async runValidation(ctx: PluginContext): Promise { const svc = safeGet(ctx, 'external-datasource'); @@ -96,6 +125,96 @@ export class ExternalValidationPlugin implements Plugin { throw new ExternalSchemaMismatchError(r.datasource, r.object, r.diffs); } } + + /** + * Arm a background drift checker for every federated datasource that declares + * `external.validation.checkIntervalMs`. Each fires on its own interval and + * emits `external.schema.drift` events โ€” it never throws or aborts the + * process, since drift past boot is observational, not fatal. + * + * No-op when metadata can't be enumerated or no datasource opts in. Re-arming + * (e.g. a second `kernel:ready`) first clears existing timers so intervals + * don't accumulate. + */ + async scheduleDriftChecks(ctx: PluginContext): Promise { + this.stop(); + const metadata = safeGet(ctx, 'metadata'); + if (!metadata?.list) return; + + let datasources: unknown[]; + try { + datasources = await metadata.list('datasource'); + } catch (err) { + ctx.logger?.warn?.('[external-validation] could not list datasources for drift checks', { err }); + return; + } + + for (const def of datasources as DatasourceDef[]) { + const interval = def?.external?.validation?.checkIntervalMs; + const name = def?.name; + if (!name || typeof interval !== 'number' || interval <= 0) continue; + + const timer = setInterval(() => { + // Fire-and-forget: the checker swallows its own errors. + void this.runDriftCheck(ctx, name); + }, interval); + // Don't let the drift timer keep the process alive on its own. + (timer as { unref?: () => void }).unref?.(); + this.driftTimers.set(name, timer); + ctx.logger?.info?.('[external-validation] armed background drift check', { + datasource: name, + intervalMs: interval, + }); + } + } + + /** + * Re-validate one datasource's federated objects and emit an + * `external.schema.drift` event per mismatch. Exposed for testing; invoked + * from the interval armed by {@link scheduleDriftChecks}. Never throws. + * + * @returns the number of drift events emitted. + */ + async runDriftCheck(ctx: PluginContext, datasource: string): Promise { + const svc = safeGet(ctx, 'external-datasource'); + if (!svc?.validateAll) return 0; + + let report: Awaited>; + try { + report = await svc.validateAll(); + } catch (err) { + ctx.logger?.warn?.('[external-validation] drift check validateAll failed', { + datasource, + err, + }); + return 0; + } + + const drifted = report.results.filter((r) => !r.ok && r.datasource === datasource); + for (const r of drifted) { + const event: ExternalSchemaDriftEvent = { + datasource: r.datasource, + object: r.object, + diffs: r.diffs, + }; + try { + await ctx.trigger('external.schema.drift', event); + } catch (err) { + ctx.logger?.warn?.('[external-validation] failed to emit drift event', { + datasource, + object: r.object, + err, + }); + } + } + if (drifted.length > 0) { + ctx.logger?.warn?.('[external-validation] background drift detected', { + datasource, + objects: drifted.map((r) => r.object), + }); + } + return drifted.length; + } } /** Convenience factory mirroring the createXxxPlugin convention. */ diff --git a/packages/runtime/src/index.ts b/packages/runtime/src/index.ts index a7a0e47d1..1822138ec 100644 --- a/packages/runtime/src/index.ts +++ b/packages/runtime/src/index.ts @@ -21,6 +21,19 @@ export { AppPlugin, collectBundleHooks, collectBundleFunctions, collectBundleAct export { SeedLoaderService } from './seed-loader.js'; // External Datasource Federation โ€” boot-validation gate (ADR-0015, Gate 2) export { ExternalValidationPlugin, createExternalValidationPlugin } from './external-validation-plugin.js'; +export type { ExternalSchemaDriftEvent } from './external-validation-plugin.js'; +// Runtime-UI datasource lifecycle host glue (ADR-0015 Addendum) +export { createDefaultDatasourceDriverFactory } from './default-datasource-driver-factory.js'; +export { + createDatasourceSecretBinder, + toCredentialsRef, + parseCredentialsRef, +} from './datasource-secret-binder.js'; +export type { + DatasourceSecretBinder, + DatasourceSecretBinderDeps, + SecretStoreEngineLike, +} from './datasource-secret-binder.js'; export { createDispatcherPlugin } from './dispatcher-plugin.js'; export type { DispatcherPluginConfig } from './dispatcher-plugin.js'; export { createSystemEnvironmentPlugin, SYSTEM_ENVIRONMENT_ID } from './system-environment-plugin.js'; diff --git a/packages/services/service-ai/src/tools/query-data.tool.test.ts b/packages/services/service-ai/src/tools/query-data.tool.test.ts new file mode 100644 index 000000000..24598a111 --- /dev/null +++ b/packages/services/service-ai/src/tools/query-data.tool.test.ts @@ -0,0 +1,118 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { createQueryDataHandler, type QueryDataToolContext, type QueryPlan } from './query-data.tool.js'; + +/** + * P4 AI safety net (ADR-0015 ยง5.4): a federated (external) object hits a + * remote production DB, so the `query_data` tool must bound the wait. These + * tests exercise the timeout wrapper around `dataEngine.find` for external + * objects, and confirm managed objects are never wrapped. + */ + +/** Build a tool context with a single retrievable object + controllable find. */ +function makeCtx(opts: { + object: Record; + datasource?: Record; + find: () => Promise; + externalQueryTimeoutMs?: number; +}): { ctx: QueryDataToolContext; getCalls: string[] } { + const getCalls: string[] = []; + const ctx: QueryDataToolContext = { + ai: { + generateObject: async () => { + const plan: QueryPlan = { + objectName: opts.object.name as string, + whereJson: null, + fields: null, + orderBy: null, + limit: null, + }; + return { object: plan } as never; + }, + } as never, + metadata: { + listObjects: async () => [opts.object], + get: async (type: string, name: string) => { + getCalls.push(`${type}/${name}`); + return type === 'datasource' ? opts.datasource : undefined; + }, + } as never, + dataEngine: { + find: async () => opts.find(), + } as never, + ...(opts.externalQueryTimeoutMs !== undefined + ? { externalQueryTimeoutMs: opts.externalQueryTimeoutMs } + : {}), + }; + return { ctx, getCalls }; +} + +const externalObject = { + name: 'orders', + label: 'Orders', + datasource: 'warehouse', + external: { writable: false, remoteName: 'fact_orders' }, + fields: { id: { type: 'text' }, amount: { type: 'number' } }, +}; + +describe('query_data โ€” federated query timeout (P4)', () => { + it("times out a federated query using the datasource's queryTimeoutMs", async () => { + const { ctx, getCalls } = makeCtx({ + object: externalObject, + datasource: { name: 'warehouse', external: { queryTimeoutMs: 10 } }, + find: () => new Promise(() => {}), // never resolves + }); + const handler = createQueryDataHandler(ctx); + const out = JSON.parse((await handler({ request: 'show me orders' })) as string); + expect(out.error).toMatch(/exceeded the 10ms timeout/); + // The external branch resolved the datasource's declared timeout. + expect(getCalls).toContain('datasource/warehouse'); + }); + + it('falls back to externalQueryTimeoutMs when the datasource declares none', async () => { + const { ctx } = makeCtx({ + object: externalObject, + datasource: { name: 'warehouse' }, // no external.queryTimeoutMs + find: () => new Promise(() => {}), + externalQueryTimeoutMs: 15, + }); + const handler = createQueryDataHandler(ctx); + const out = JSON.parse((await handler({ request: 'show me orders' })) as string); + expect(out.error).toMatch(/exceeded the 15ms timeout/); + }); + + it('returns records when the federated query resolves before the timeout', async () => { + const { ctx } = makeCtx({ + object: externalObject, + datasource: { name: 'warehouse', external: { queryTimeoutMs: 1000 } }, + find: async () => [{ id: 'o1', amount: 42 }], + }); + const handler = createQueryDataHandler(ctx); + const out = JSON.parse((await handler({ request: 'show me orders' })) as string); + expect(out.error).toBeUndefined(); + expect(out.count).toBe(1); + expect(out.records[0].id).toBe('o1'); + }); + + it('does not wrap managed (non-external) objects in a timeout', async () => { + const managedObject = { + name: 'task', + label: 'Task', + fields: { id: { type: 'text' }, title: { type: 'text' } }, + }; + const { ctx, getCalls } = makeCtx({ + object: managedObject, + // A managed find that takes longer than any external fallback would โ€” + // it must still succeed because the managed path is never timed out. + find: () => new Promise((resolve) => setTimeout(() => resolve([{ id: 't1' }]), 40)), + externalQueryTimeoutMs: 5, + }); + const handler = createQueryDataHandler(ctx); + const out = JSON.parse((await handler({ request: 'show me task' })) as string); + expect(out.error).toBeUndefined(); + expect(out.count).toBe(1); + // Never consulted the datasource timeout โ€” the external branch wasn't taken. + expect(getCalls.some((c) => c.startsWith('datasource/'))).toBe(false); + }); +}); diff --git a/packages/services/service-ai/src/tools/query-data.tool.ts b/packages/services/service-ai/src/tools/query-data.tool.ts index a4488937d..62ebb2053 100644 --- a/packages/services/service-ai/src/tools/query-data.tool.ts +++ b/packages/services/service-ai/src/tools/query-data.tool.ts @@ -42,6 +42,15 @@ export interface QueryDataToolContext { dataEngine: IDataEngine; /** Maximum number of records returned per call (default: 100). */ maxLimit?: number; + /** + * Fallback hard cap (ms) on a single query against a *federated* (external) + * object, used when the datasource doesn't declare its own + * `external.queryTimeoutMs`. A slow remote warehouse must never hang the AI + * tool loop indefinitely (ADR-0015 ยง5.4 AI safety net). Default: 30_000. + * Managed (local) objects are never timed out here โ€” they're already bounded + * by the injected `LIMIT`. + */ + externalQueryTimeoutMs?: number; /** * Optional protocol shim for cross-source object enumeration. Mirrors the * fallback used by `list_objects`/`describe_object` โ€” without it the @@ -150,6 +159,20 @@ export const QUERY_DATA_TOOL: AIToolDefinition = { export function createQueryDataHandler(ctx: QueryDataToolContext): ToolHandler { const retriever = new SchemaRetriever(ctx.metadata, {}, ctx.protocol); const maxLimit = ctx.maxLimit ?? 100; + const externalTimeoutFallback = ctx.externalQueryTimeoutMs ?? 30_000; + + /** Resolve a federated object's per-query timeout (datasource-declared, + * else the tool fallback). Never throws โ€” degrades to the fallback. */ + const resolveExternalTimeout = async (datasource: string): Promise => { + try { + const ds = (await ctx.metadata.get?.('datasource', datasource)) as + | { external?: { queryTimeoutMs?: number } } + | undefined; + return ds?.external?.queryTimeoutMs ?? externalTimeoutFallback; + } catch { + return externalTimeoutFallback; + } + }; return async (args, execCtx) => { const { request } = args as { request: string }; @@ -244,14 +267,24 @@ export function createQueryDataHandler(ctx: QueryDataToolContext): ToolHandler { }); } } + // Federated objects hit a remote production DB โ€” bound the wait so a slow + // warehouse can't hang the tool loop. Managed objects skip this entirely. + const isExternal = matchedObject.external !== undefined; try { - const records = await ctx.dataEngine.find(plan.objectName, { + const findPromise = ctx.dataEngine.find(plan.objectName, { where, fields: plan.fields ?? undefined, orderBy: plan.orderBy ?? undefined, limit, context: buildAiEngineContext(execCtx), }); + const records = isExternal + ? await withTimeout( + findPromise, + await resolveExternalTimeout(matchedObject.datasource ?? 'default'), + plan.objectName, + ) + : await findPromise; return JSON.stringify({ plan: { ...plan, where }, count: records.length, @@ -266,6 +299,37 @@ export function createQueryDataHandler(ctx: QueryDataToolContext): ToolHandler { }; } +/** + * Bound a promise with a timeout. On expiry, rejects with a descriptive error + * (surfaced to the model as a query failure). The underlying `find` is not + * cancellable, so it may complete in the background โ€” that's acceptable for a + * safety net whose job is to return control to the tool loop promptly. + */ +function withTimeout(p: Promise, ms: number, object: string): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + reject( + new Error( + `query on external object '${object}' exceeded the ${ms}ms timeout. ` + + 'Narrow the filter or lower the limit.', + ), + ); + }, ms); + // Don't let the timer hold the event loop open on its own. + (timer as { unref?: () => void }).unref?.(); + p.then( + (v) => { + clearTimeout(timer); + resolve(v); + }, + (e) => { + clearTimeout(timer); + reject(e); + }, + ); + }); +} + /** * Register the `query_data` tool on the given {@link ToolRegistry}. * diff --git a/packages/services/service-external-datasource/src/__tests__/datasource-admin-plugin.test.ts b/packages/services/service-external-datasource/src/__tests__/datasource-admin-plugin.test.ts new file mode 100644 index 000000000..efc35e243 --- /dev/null +++ b/packages/services/service-external-datasource/src/__tests__/datasource-admin-plugin.test.ts @@ -0,0 +1,231 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import type { IDatasourceAdminService, IDatasourceDriverFactory } from '@objectstack/spec/contracts'; +import { + DatasourceAdminServicePlugin, + type DatasourceAdminServicePluginOptions, +} from '../datasource-admin-plugin.js'; + +/** + * Minimal PluginContext + in-memory metadata service. Boots the plugin and + * returns the registered `datasource-admin` service so we can exercise the + * plugin's glue (probe via factory, fail-closed secret) end to end. + */ +async function boot(opts: DatasourceAdminServicePluginOptions & { + services?: Record; +} = {}) { + const registry = new Map>(); + const metadata = { + get: async (t: string, n: string) => registry.get(t)?.get(n), + list: async (t: string) => [...(registry.get(t)?.values() ?? [])], + register: async (t: string, n: string, d: unknown) => { + if (!registry.has(t)) registry.set(t, new Map()); + registry.get(t)!.set(n, d); + }, + unregister: async (t: string, n: string) => { + registry.get(t)?.delete(n); + }, + listObjects: async () => [...(registry.get('object')?.values() ?? [])], + }; + + const services: Record = { metadata, ...(opts.services ?? {}) }; + let registered: IDatasourceAdminService | undefined; + const ctx: any = { + getService: (name: string) => { + if (name in services) return services[name]; + throw new Error(`no service ${name}`); + }, + registerService: (name: string, svc: unknown) => { + if (name === 'datasource-admin') registered = svc as IDatasourceAdminService; + }, + trigger: async () => {}, + logger: { warn() {}, info() {} }, + }; + + const { services: _omit, ...pluginOpts } = opts; + const plugin = new DatasourceAdminServicePlugin(pluginOpts); + await plugin.init(ctx); + return { service: registered!, registry, metadata, plugin, ctx }; +} + +/** A driver factory whose handle records connect/ping/disconnect calls. */ +function fakeFactory(over?: Partial & { onProbe?: () => void }): IDatasourceDriverFactory { + return { + supports: (id: string) => id === 'postgres', + create: async (spec) => ({ + connect: async () => {}, + ping: async () => { + over?.onProbe?.(); + // expose the secret the factory received for assertions + (globalThis as any).__lastProbeSecret = spec.secret; + }, + disconnect: async () => {}, + serverVersion: async () => 'PostgreSQL 16.1', + }), + ...over, + }; +} + +describe('DatasourceAdminServicePlugin: probe', () => { + it('tests a connection through the driver factory (latency + version)', async () => { + const { service } = await boot({ + driverFactory: fakeFactory(), + }); + const res = await service.testConnection( + { name: 'reporting', driver: 'postgres', config: { host: 'db' } }, + { value: 's3cret' }, + ); + expect(res.ok).toBe(true); + expect(res.serverVersion).toBe('PostgreSQL 16.1'); + expect(typeof res.latencyMs).toBe('number'); + expect((globalThis as any).__lastProbeSecret).toBe('s3cret'); + }); + + it('returns ok:false when no factory supports the driver', async () => { + const { service } = await boot({ driverFactory: fakeFactory() }); + const res = await service.testConnection({ name: 'x', driver: 'oracle', config: {} }); + expect(res.ok).toBe(false); + expect(res.error).toMatch(/no driver factory supports/i); + }); + + it('returns ok:false when no factory is registered at all', async () => { + const { service } = await boot(); + const res = await service.testConnection({ name: 'x', driver: 'postgres', config: {} }); + expect(res.ok).toBe(false); + expect(res.error).toMatch(/no driver factory is registered/i); + }); +}); + +describe('DatasourceAdminServicePlugin: secret fail-closed', () => { + it('refuses to create a secret-bearing datasource without a secret binder', async () => { + const { service, registry } = await boot({ driverFactory: fakeFactory() }); + await expect( + service.createDatasource({ name: 'reporting', driver: 'postgres', config: {} }, { value: 'pw' }), + ).rejects.toThrow(/no secret store configured/i); + // nothing persisted + expect(registry.get('datasource')?.size ?? 0).toBe(0); + }); + + it('persists a credentialsRef (not cleartext) when a binder is wired', async () => { + const bound: string[] = []; + const { service, registry } = await boot({ + driverFactory: fakeFactory(), + secrets: { + bind: async (input, hint) => { + bound.push(input.value); + return `sys_secret://datasource/${hint.name}#1`; + }, + }, + }); + await service.createDatasource({ name: 'reporting', driver: 'postgres', config: {} }, { value: 'pw' }); + const rec = registry.get('datasource')?.get('reporting') as any; + expect(rec.origin).toBe('runtime'); + expect(rec.external?.credentialsRef).toBe('sys_secret://datasource/reporting#1'); + expect(JSON.stringify(rec)).not.toContain('pw'); + expect(bound).toEqual(['pw']); + }); +}); + +describe('DatasourceAdminServicePlugin: boot rehydration', () => { + /** Fake engine ('data') that records hot-registered drivers. */ + function fakeEngine() { + const drivers: any[] = []; + return { + drivers, + registerDriver: (d: any) => drivers.push(d), + registerDatasourceDef: () => {}, + getDriverByName: (n: string) => drivers.find((d) => d.name === n), + }; + } + + /** Factory that records the spec (incl. resolved secret) of each create(). */ + function recordingFactory() { + const specs: any[] = []; + const factory: IDatasourceDriverFactory = { + supports: (id: string) => id === 'postgres', + create: async (spec) => { + specs.push(spec); + return { connect: async () => {}, disconnect: async () => {} }; + }, + }; + return { factory, specs }; + } + + it('rebuilds runtime pools at start(), decrypting the credentialsRef', async () => { + const engine = fakeEngine(); + const { factory, specs } = recordingFactory(); + const resolved: string[] = []; + + const { plugin, ctx, registry } = await boot({ + driverFactory: factory, + services: { data: engine }, + secrets: { + bind: async () => 'sys_secret:abc', + resolve: async (ref) => { + resolved.push(ref); + return ref === 'sys_secret:abc' ? 'super-secret-pw' : undefined; + }, + }, + }); + + // Simulate a persisted (DB-backed) runtime datasource that survived a restart. + registry.set( + 'datasource', + new Map([ + ['crm_primary', { name: 'crm_primary', driver: 'sqlite', origin: 'code' }], + [ + 'reporting', + { + name: 'reporting', + driver: 'postgres', + origin: 'runtime', + active: true, + config: { host: 'db' }, + external: { credentialsRef: 'sys_secret:abc' }, + }, + ], + [ + 'archived', + { name: 'archived', driver: 'postgres', origin: 'runtime', active: false }, + ], + ]), + ); + + await plugin.start(ctx); + + // Only the active runtime datasource is rehydrated โ€” not the code one, not the inactive one. + expect(engine.drivers.map((d) => d.name)).toEqual(['reporting']); + // The credentialsRef was dereferenced and the cleartext handed to the factory. + expect(resolved).toEqual(['sys_secret:abc']); + expect(specs).toHaveLength(1); + expect(specs[0].secret).toBe('super-secret-pw'); + expect(specs[0].name).toBe('reporting'); + }); + + it('does not block boot when nothing is persisted (dev: in-memory store)', async () => { + const engine = fakeEngine(); + const { factory } = recordingFactory(); + const { plugin, ctx } = await boot({ driverFactory: factory, services: { data: engine } }); + await expect(plugin.start(ctx)).resolves.toBeUndefined(); + expect(engine.drivers).toHaveLength(0); + }); +}); + +describe('DatasourceAdminServicePlugin: persistence + bound count', () => { + it('lists code (artefact) + runtime records with origin, blocks remove while bound', async () => { + const { service, registry } = await boot({ driverFactory: fakeFactory() }); + // seed an artefact (code) datasource lacking explicit origin + registry.set('datasource', new Map([['crm_primary', { name: 'crm_primary', driver: 'sqlite' }]])); + // seed an object bound to a runtime datasource + registry.set('object', new Map([['lead', { name: 'lead', datasource: 'reporting' }]])); + + await service.createDatasource({ name: 'reporting', driver: 'postgres', config: {} }); + + const list = await service.listDatasources(); + expect(list.find((d) => d.name === 'crm_primary')?.origin).toBe('code'); + expect(list.find((d) => d.name === 'reporting')?.origin).toBe('runtime'); + + await expect(service.removeDatasource('reporting')).rejects.toThrow(/1 object\(s\)/); + }); +}); diff --git a/packages/services/service-external-datasource/src/__tests__/datasource-admin-service.test.ts b/packages/services/service-external-datasource/src/__tests__/datasource-admin-service.test.ts new file mode 100644 index 000000000..c987b167b --- /dev/null +++ b/packages/services/service-external-datasource/src/__tests__/datasource-admin-service.test.ts @@ -0,0 +1,288 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, it, expect } from 'vitest'; +import { + DatasourceAdminService, + type DatasourceAdminServiceConfig, + type StoredDatasource, + type ProbeInput, +} from '../datasource-admin-service.js'; + +/** + * In-memory harness: an editable record store + secret store, with probe and + * bound-object count stubbable per test. Records what was probed/written so + * tests can assert credentials never leak into the persisted record. + */ +function makeHarness(opts?: { + seed?: StoredDatasource[]; + probe?: (input: ProbeInput) => Promise<{ ok: boolean; error?: string; latencyMs?: number }>; + boundCounts?: Record; +}) { + // Flat list, not a name-keyed map: in production `listDatasourceRecords` + // merges artefact (code) records with runtime-store records, so the same + // name can legitimately appear twice (a runtime row shadowed by a code one). + const records: StoredDatasource[] = (opts?.seed ?? []).map((r) => ({ ...r })); + /** Resolve the effective record for a name (code wins over runtime). */ + const findEffective = (n: string) => + records.find((r) => r.name === n && r.origin !== 'runtime') ?? + records.find((r) => r.name === n); + + const secrets = new Map(); + let secretSeq = 0; + const probed: ProbeInput[] = []; + const registered: string[] = []; + const unregistered: string[] = []; + const removedSecrets: string[] = []; + + const config: DatasourceAdminServiceConfig = { + probe: async (input) => { + probed.push(input); + return (opts?.probe ?? (async () => ({ ok: true, latencyMs: 3 })))(input); + }, + listDatasourceRecords: async () => records.map((r) => ({ ...r })), + getDatasourceRecord: async (n) => { + const r = findEffective(n); + return r ? { ...r } : undefined; + }, + putDatasourceRecord: async (record) => { + const idx = records.findIndex((r) => r.name === record.name && r.origin === 'runtime'); + if (idx >= 0) records[idx] = { ...record }; + else records.push({ ...record }); + }, + deleteDatasourceRecord: async (n) => { + const idx = records.findIndex((r) => r.name === n && r.origin === 'runtime'); + if (idx >= 0) records.splice(idx, 1); + }, + writeSecret: async (input, hint) => { + const ref = `sys_secret://datasource/${input.key ?? hint.name}#${++secretSeq}`; + secrets.set(ref, { value: input.value, namespace: input.namespace, key: input.key }); + return ref; + }, + removeSecret: async (ref) => { + removedSecrets.push(ref); + secrets.delete(ref); + }, + countBoundObjects: async (n) => opts?.boundCounts?.[n] ?? 0, + registerPool: (record) => { + registered.push(record.name); + }, + unregisterPool: (name) => { + unregistered.push(name); + }, + }; + + const service = new DatasourceAdminService(config); + // Thin accessor over the flat record list, runtime-preferring (tests assert + // on the persisted runtime row, e.g. after create/update). + const store = { + get: (n: string) => + records.find((r) => r.name === n && r.origin === 'runtime') ?? + records.find((r) => r.name === n), + has: (n: string) => records.some((r) => r.name === n), + get size() { + return records.length; + }, + }; + return { service, store, secrets, probed, registered, unregistered, removedSecrets }; +} + +describe('listDatasources', () => { + it('reports origin + dedupes by name (code wins, flags shadowed runtime)', async () => { + const { service } = makeHarness({ + seed: [ + { name: 'crm_primary', driver: 'sqlite', origin: 'code', definedIn: '@example/crm' }, + { name: 'crm_primary', driver: 'postgres', origin: 'runtime' }, + { name: 'reporting', driver: 'postgres', schemaMode: 'external', origin: 'runtime' }, + ], + }); + + const list = await service.listDatasources(); + const crm = list.find((d) => d.name === 'crm_primary')!; + const reporting = list.find((d) => d.name === 'reporting')!; + + expect(list).toHaveLength(2); + expect(crm.origin).toBe('code'); + expect(crm.driver).toBe('sqlite'); // code wins over the runtime row + expect(crm.definedIn).toBe('@example/crm'); + expect(crm.conflictsWithCode).toBe(true); + expect(reporting.origin).toBe('runtime'); + expect(reporting.schemaMode).toBe('external'); + expect(reporting.conflictsWithCode).toBeUndefined(); + }); +}); + +describe('testConnection', () => { + it('probes with the cleartext secret without persisting anything', async () => { + const { service, store, probed } = makeHarness(); + const res = await service.testConnection( + { name: 'tmp', driver: 'postgres', config: { host: 'db.internal' } }, + { value: 's3cret' }, + ); + expect(res.ok).toBe(true); + expect(probed[0].secret).toBe('s3cret'); + expect(store.size).toBe(0); // nothing saved + }); + + it('returns ok:false when no driver is supplied', async () => { + const { service } = makeHarness(); + const res = await service.testConnection({ name: 'x', driver: '' }); + expect(res.ok).toBe(false); + expect(res.error).toMatch(/driver is required/i); + }); + + it('captures a thrown probe error as ok:false', async () => { + const { service } = makeHarness({ + probe: async () => { + throw new Error('ECONNREFUSED'); + }, + }); + const res = await service.testConnection({ name: 'x', driver: 'postgres' }); + expect(res.ok).toBe(false); + expect(res.error).toMatch(/ECONNREFUSED/); + }); +}); + +describe('createDatasource', () => { + it('persists a runtime record and stores the secret as an opaque ref only', async () => { + const { service, store, secrets } = makeHarness(); + const summary = await service.createDatasource( + { + name: 'reporting', + driver: 'postgres', + schemaMode: 'external', + config: { host: 'db.internal', database: 'analytics' }, + external: { allowWrites: false }, + }, + { value: 'postgres://user:pw@db.internal/analytics' }, + ); + + expect(summary.origin).toBe('runtime'); + const rec = store.get('reporting')!; + expect(rec.origin).toBe('runtime'); + // credential is referenced, never inlined + expect(rec.external?.credentialsRef).toBeTruthy(); + expect(JSON.stringify(rec)).not.toContain('postgres://'); + expect(JSON.stringify(rec)).not.toContain('pw@'); + expect(secrets.size).toBe(1); + }); + + it('hot-registers the pool after create', async () => { + const { service, registered } = makeHarness(); + await service.createDatasource({ name: 'reporting', driver: 'postgres' }); + expect(registered).toContain('reporting'); + }); + + it('rejects a name owned by a code-defined datasource', async () => { + const { service } = makeHarness({ + seed: [{ name: 'crm_primary', driver: 'sqlite', origin: 'code' }], + }); + await expect( + service.createDatasource({ name: 'crm_primary', driver: 'postgres' }), + ).rejects.toThrow(/code-defined/i); + }); + + it('rejects a duplicate runtime name', async () => { + const { service } = makeHarness({ + seed: [{ name: 'reporting', driver: 'postgres', origin: 'runtime' }], + }); + await expect( + service.createDatasource({ name: 'reporting', driver: 'postgres' }), + ).rejects.toThrow(/already exists/i); + }); + + it('rejects an invalid name', async () => { + const { service } = makeHarness(); + await expect( + service.createDatasource({ name: 'Bad-Name', driver: 'postgres' }), + ).rejects.toThrow(/must match/i); + }); +}); + +describe('updateDatasource', () => { + it('patches a runtime record and rewraps the secret, removing the old ref', async () => { + const { service, store, secrets, removedSecrets } = makeHarness({ + seed: [ + { + name: 'reporting', + driver: 'postgres', + origin: 'runtime', + external: { credentialsRef: 'sys_secret://datasource/reporting#0' }, + }, + ], + }); + secrets.set('sys_secret://datasource/reporting#0', { value: 'old' }); + + const summary = await service.updateDatasource( + 'reporting', + { label: 'Reporting DB', active: false }, + { value: 'new-pw' }, + ); + + expect(summary.label).toBe('Reporting DB'); + expect(summary.active).toBe(false); + const rec = store.get('reporting')!; + expect(rec.external?.credentialsRef).not.toBe('sys_secret://datasource/reporting#0'); + expect(removedSecrets).toContain('sys_secret://datasource/reporting#0'); + }); + + it('preserves the existing credentialsRef when external is patched without a new secret', async () => { + const ref = 'sys_secret://datasource/reporting#0'; + const { service, store } = makeHarness({ + seed: [ + { name: 'reporting', driver: 'postgres', origin: 'runtime', external: { credentialsRef: ref } }, + ], + }); + await service.updateDatasource('reporting', { external: { allowWrites: true } }); + expect(store.get('reporting')!.external?.credentialsRef).toBe(ref); + }); + + it('rejects editing a code-defined datasource', async () => { + const { service } = makeHarness({ + seed: [{ name: 'crm_primary', driver: 'sqlite', origin: 'code' }], + }); + await expect( + service.updateDatasource('crm_primary', { label: 'x' }), + ).rejects.toThrow(/code-defined/i); + }); + + it('rejects updating a missing datasource', async () => { + const { service } = makeHarness(); + await expect(service.updateDatasource('nope', { label: 'x' })).rejects.toThrow(/not found/i); + }); +}); + +describe('removeDatasource', () => { + it('removes a runtime record, its secret, and the pool', async () => { + const ref = 'sys_secret://datasource/reporting#0'; + const { service, store, removedSecrets, unregistered } = makeHarness({ + seed: [ + { name: 'reporting', driver: 'postgres', origin: 'runtime', external: { credentialsRef: ref } }, + ], + }); + await service.removeDatasource('reporting'); + expect(store.has('reporting')).toBe(false); + expect(removedSecrets).toContain(ref); + expect(unregistered).toContain('reporting'); + }); + + it('refuses to remove while objects are still bound', async () => { + const { service, store } = makeHarness({ + seed: [{ name: 'reporting', driver: 'postgres', origin: 'runtime' }], + boundCounts: { reporting: 3 }, + }); + await expect(service.removeDatasource('reporting')).rejects.toThrow(/3 object\(s\)/); + expect(store.has('reporting')).toBe(true); + }); + + it('refuses to remove a code-defined datasource', async () => { + const { service } = makeHarness({ + seed: [{ name: 'crm_primary', driver: 'sqlite', origin: 'code' }], + }); + await expect(service.removeDatasource('crm_primary')).rejects.toThrow(/code-defined/i); + }); + + it('rejects removing a missing datasource', async () => { + const { service } = makeHarness(); + await expect(service.removeDatasource('nope')).rejects.toThrow(/not found/i); + }); +}); diff --git a/packages/services/service-external-datasource/src/__tests__/external-datasource-service.test.ts b/packages/services/service-external-datasource/src/__tests__/external-datasource-service.test.ts index 833553217..42f7e9e52 100644 --- a/packages/services/service-external-datasource/src/__tests__/external-datasource-service.test.ts +++ b/packages/services/service-external-datasource/src/__tests__/external-datasource-service.test.ts @@ -121,6 +121,80 @@ describe('generateObjectDraft', () => { }); }); +describe('importObject', () => { + /** Build a service with a recording persistObject (runtime metadata store). */ + function makeImporter(persistObject?: (name: string, def: Record) => Promise) { + const persisted: Array<{ name: string; def: Record }> = []; + const svc = new ExternalDatasourceService({ + introspect: async () => warehouseSchema(), + getDatasource: async () => ({ name: 'warehouse', schemaMode: 'external' }), + getObject: async () => undefined, + listObjects: async () => [], + persistObject: + persistObject ?? (async (name, def) => { persisted.push({ name, def }); }), + }); + return { svc, persisted }; + } + + it('persists a runtime federated object and returns name/definition/review', async () => { + const { svc, persisted } = makeImporter(); + const result = await svc.importObject('warehouse', 'fact_orders'); + + expect(result.name).toBe('fact_orders'); + expect(persisted).toHaveLength(1); + expect(persisted[0].name).toBe('fact_orders'); + const def = persisted[0].def as { datasource: string; external: { remoteName: string; writable?: boolean } }; + expect(def.datasource).toBe('warehouse'); + expect(def.external.remoteName).toBe('fact_orders'); + // Read-only by default โ€” no writable flag leaks in. + expect(def.external.writable).toBeUndefined(); + // The geography column surfaced a review note (carried over from the draft). + expect(result.review.some((r) => r.column === 'geo')).toBe(true); + }); + + it('applies the name override and writable opt-in', async () => { + const { svc, persisted } = makeImporter(); + const result = await svc.importObject('warehouse', 'fact_orders', { + name: 'wh_orders', + writable: true, + }); + expect(result.name).toBe('wh_orders'); + const def = persisted[0].def as { name: string; label: string; external: { writable?: boolean } }; + expect(def.name).toBe('wh_orders'); + expect(def.label).toBe('Wh Orders'); + expect(def.external.writable).toBe(true); + }); + + it('forwards draft options (include/rename) through to the persisted fields', async () => { + const { svc, persisted } = makeImporter(); + await svc.importObject('warehouse', 'fact_orders', { + includeColumns: ['order_id', 'amount'], + rename: { amount: 'total' }, + }); + const def = persisted[0].def as { fields: Record }; + expect(Object.keys(def.fields)).toEqual(['order_id', 'total']); + }); + + it('throws when no writable metadata store is wired', async () => { + const svc = new ExternalDatasourceService({ + introspect: async () => warehouseSchema(), + getDatasource: async () => ({ name: 'warehouse', schemaMode: 'external' }), + getObject: async () => undefined, + listObjects: async () => [], + // no persistObject + }); + await expect(svc.importObject('warehouse', 'fact_orders')).rejects.toThrow( + /writable metadata store/, + ); + }); + + it('throws when the remote table is missing (no persistence)', async () => { + const { svc, persisted } = makeImporter(); + await expect(svc.importObject('warehouse', 'ghost')).rejects.toThrow(/not found/); + expect(persisted).toHaveLength(0); + }); +}); + describe('validateObject', () => { const baseObject: ObjectLike = { name: 'wh_order', @@ -242,12 +316,45 @@ describe('validateAll', () => { describe('refreshCatalog', () => { it('produces a snapshot with suggested field types', async () => { const svc = makeService(); - const catalog = (await svc.refreshCatalog('warehouse')) as { - datasource: string; - tables: Array<{ remoteName: string; columns: Array<{ name: string; suggestedFieldType?: string }> }>; - }; + const catalog = await svc.refreshCatalog('warehouse'); expect(catalog.datasource).toBe('warehouse'); + expect(catalog.name).toBe('warehouse_catalog'); const orders = catalog.tables.find((t) => t.remoteName === 'fact_orders')!; expect(orders.columns.find((c) => c.name === 'amount')?.suggestedFieldType).toBe('number'); + // Canonicalised through the Zod schema: primaryKey default applied. + expect(orders.columns.find((c) => c.name === 'order_id')?.primaryKey).toBe(true); + expect(orders.columns.find((c) => c.name === 'amount')?.primaryKey).toBe(false); + }); + + it('persists the snapshot as an external_catalog record when a store is wired', async () => { + const persisted: unknown[] = []; + const svc = new ExternalDatasourceService({ + introspect: async () => warehouseSchema(), + getDatasource: async () => ({ name: 'warehouse', schemaMode: 'external' }), + getObject: async () => undefined, + listObjects: async () => [], + persistCatalog: async (c) => { + persisted.push(c); + }, + }); + const catalog = await svc.refreshCatalog('warehouse'); + expect(persisted).toHaveLength(1); + expect(persisted[0]).toBe(catalog); + expect((persisted[0] as { name: string }).name).toBe('warehouse_catalog'); + }); + + it('still returns the snapshot when persistence throws (best-effort cache)', async () => { + const svc = new ExternalDatasourceService({ + introspect: async () => warehouseSchema(), + getDatasource: async () => ({ name: 'warehouse', schemaMode: 'external' }), + getObject: async () => undefined, + listObjects: async () => [], + persistCatalog: async () => { + throw new Error('metadata store is read-only'); + }, + logger: { warn: () => {} }, + }); + const catalog = await svc.refreshCatalog('warehouse'); + expect(catalog.name).toBe('warehouse_catalog'); }); }); diff --git a/packages/services/service-external-datasource/src/datasource-admin-plugin.ts b/packages/services/service-external-datasource/src/datasource-admin-plugin.ts new file mode 100644 index 000000000..84a2413e4 --- /dev/null +++ b/packages/services/service-external-datasource/src/datasource-admin-plugin.ts @@ -0,0 +1,340 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +import type { Plugin, PluginContext } from '@objectstack/core'; +import type { + IDatasourceDriverFactory, + DatasourceConnectionSpec, + TestConnectionResult, +} from '@objectstack/spec/contracts'; +import { + DatasourceAdminService, + type DatasourceAdminServiceConfig, + type StoredDatasource, + type ProbeInput, +} from './datasource-admin-service.js'; +import type { Logger } from './external-datasource-service.js'; + +/** + * Minimal metadata-service surface used for datasource persistence + the + * bound-object count. Kept structural so the plugin doesn't hard-depend on the + * concrete `MetadataManager`. + */ +interface MetadataServiceLike { + get: (type: string, name: string) => Promise; + list: (type: string) => Promise; + register: (type: string, name: string, data: unknown) => Promise; + unregister: (type: string, name: string) => Promise; + listObjects?: () => Promise; +} + +/** Engine surface used for hot pool (de)registration. */ +interface DataEngineLike { + registerDriver?: (driver: unknown, isDefault?: boolean) => void; + registerDatasourceDef?: (def: { name: string; schemaMode?: string; external?: { allowWrites?: boolean } }) => void; + getDriverByName?: (name: string) => unknown; +} + +/** + * Host-provided secret binding. Encrypts a cleartext credential into the secret + * store and returns an opaque `credentialsRef`; `unbind` deletes it. Wired by + * the stack that owns the `ICryptoProvider` + `sys_secret` store. When absent, + * the plugin fails *closed*: creating/updating a datasource *with* a secret + * throws rather than risk persisting cleartext. + */ +export interface SecretBinder { + bind: (input: { value: string; namespace?: string; key?: string }, hint: { name: string }) => Promise; + unbind?: (credentialsRef: string) => Promise; + /** + * Dereference a `credentialsRef` back to cleartext for opening a live + * connection (boot rehydration + hot pool registration). Optional: when + * absent, pools for secret-bearing datasources are built without the + * credential (fine for credential-less drivers like sqlite/memory). + */ + resolve?: (credentialsRef: string) => Promise; +} + +export interface DatasourceAdminServicePluginOptions { + /** Secret binding backed by the host's crypto provider + `sys_secret`. */ + secrets?: SecretBinder; + /** Override the driver factory (defaults to the `'datasource-driver-factory'` service). */ + driverFactory?: IDatasourceDriverFactory; + logger?: Logger; +} + +/** + * DatasourceAdminServicePlugin โ€” registers `IDatasourceAdminService` into the + * kernel as the `'datasource-admin'` service (ADR-0015 Addendum). + * + * Bridges the decoupled {@link DatasourceAdminService} to live infrastructure: + * - persistence + bound-object count via the `'metadata'` service + * (`register`/`unregister` write through to the runtime DB loader), + * - connection probe + hot pool (de)registration via the + * `'datasource-driver-factory'` capability and the `'data'` engine, + * - secret encryption via a host-provided {@link SecretBinder} (fail-closed). + * + * Every dependency degrades gracefully: a missing driver factory turns + * `testConnection` into a clear `{ ok: false }` and skips hot pool registration + * (the driver is picked up at next boot); a missing secret binder makes + * secret-bearing create/update fail loudly instead of leaking cleartext. + */ +export class DatasourceAdminServicePlugin implements Plugin { + name = 'com.objectstack.service-datasource-admin'; + version = '1.0.0'; + type = 'standard' as const; + dependencies: string[] = []; + + private service?: DatasourceAdminService; + private config?: DatasourceAdminServiceConfig; + private readonly options: DatasourceAdminServicePluginOptions; + + constructor(options: DatasourceAdminServicePluginOptions = {}) { + this.options = options; + } + + async init(ctx: PluginContext): Promise { + const logger = this.options.logger; + + // Resolve infra services lazily, per call โ€” `init()` may run before the + // `data` / `metadata` plugins have registered their services (plugin start + // order is dependency- not registration-driven), and admin requests only + // arrive long after the full boot completes. + const metadataOf = (): MetadataServiceLike | undefined => + safeGetService(ctx, 'metadata'); + const engineOf = (): DataEngineLike | undefined => + safeGetService(ctx, 'data'); + + const factory = (): IDatasourceDriverFactory | undefined => + this.options.driverFactory ?? safeGetService(ctx, 'datasource-driver-factory'); + + const config: DatasourceAdminServiceConfig = { + probe: (input) => this.probe(factory(), input), + + listDatasourceRecords: async () => { + const rows = ((await metadataOf()?.list('datasource')) ?? []) as StoredDatasource[]; + // Artefact-loaded rows may omit `origin`; treat them as code-defined. + return rows.map((r) => ({ ...r, origin: r.origin ?? 'code' })); + }, + + getDatasourceRecord: async (name) => { + const row = (await metadataOf()?.get('datasource', name)) as StoredDatasource | undefined; + return row ? { ...row, origin: row.origin ?? 'code' } : undefined; + }, + + putDatasourceRecord: async (record) => { + const metadata = metadataOf(); + if (!metadata?.register) { + throw new Error('Metadata service is unavailable; cannot persist datasource.'); + } + await metadata.register('datasource', record.name, record); + }, + + deleteDatasourceRecord: async (name) => { + const metadata = metadataOf(); + if (!metadata?.unregister) { + throw new Error('Metadata service is unavailable; cannot remove datasource.'); + } + await metadata.unregister('datasource', name); + }, + + writeSecret: async (input, hint) => { + const binder = this.options.secrets; + if (!binder?.bind) { + throw new Error( + 'No secret store configured: refusing to persist a datasource credential in cleartext. ' + + 'Wire a SecretBinder (CryptoProvider + sys_secret) into DatasourceAdminServicePlugin.', + ); + } + return binder.bind(input, hint); + }, + + removeSecret: async (ref) => { + await this.options.secrets?.unbind?.(ref); + }, + + countBoundObjects: async (datasource) => { + const metadata = metadataOf(); + const objects = ((await metadata?.listObjects?.()) ?? + (await metadata?.list('object')) ?? + []) as Array<{ datasource?: string }>; + return objects.filter((o) => o?.datasource === datasource).length; + }, + + registerPool: async (record) => { + const f = factory(); + const engine = engineOf(); + if (!f || !engine?.registerDriver || !f.supports(record.driver)) return; + // Recover the cleartext credential from `sys_secret` so the pool opens + // with the real password. The cleartext is never persisted on the + // record (only `credentialsRef`), so it must be dereferenced here โ€” + // both on create/update and on boot rehydration. Credential-less + // drivers (sqlite/memory) simply have no ref and skip this. + const credentialsRef = record.external?.credentialsRef; + const secret = credentialsRef ? await this.options.secrets?.resolve?.(credentialsRef) : undefined; + const handle = await f.create({ ...this.toSpec(record), ...(secret ? { secret } : {}) }); + if (typeof handle?.connect === 'function') await handle.connect(); + // The engine routes a datasource to a driver by `driver.name === ` + // (see ObjectQL engine.getDriver). Prefer the factory's underlying engine + // driver (the `driver` escape hatch); fall back to the handle itself. Stamp + // the name so routing resolves to this pool. + const engineDriver = (handle.driver ?? handle) as { name?: string }; + try { + engineDriver.name = record.name; + } catch { + /* frozen driver โ€” registration may still work if name already matches */ + } + engine.registerDriver(engineDriver); + engine.registerDatasourceDef?.({ + name: record.name, + schemaMode: record.schemaMode, + external: record.external as { allowWrites?: boolean } | undefined, + }); + }, + + unregisterPool: async (name) => { + const driver = engineOf()?.getDriverByName?.(name) as { disconnect?: () => Promise } | undefined; + if (typeof driver?.disconnect === 'function') await driver.disconnect(); + }, + + logger, + }; + + this.config = config; + this.service = new DatasourceAdminService(config); + ctx.registerService('datasource-admin', this.service); + } + + async start(ctx: PluginContext): Promise { + // Rebuild live connection pools for persisted runtime datasources before + // announcing readiness โ€” a node restart otherwise leaves UI-created + // datasources with a record but no open pool until the next write. + await this.rehydratePools(); + if (this.service) await ctx.trigger('datasource-admin:ready', this.service); + } + + /** + * Boot-time rehydration: list persisted runtime datasources and re-register + * each one's connection pool (driver build โ†’ connect โ†’ registerDriver), + * decrypting its `sys_secret` credential on the way via the configured + * `registerPool` (which resolves `credentialsRef`). Code-defined datasources + * are owned by the host stack's own boot path and skipped here. Entirely + * best-effort: a missing factory/engine, an unpersisted dev store (nothing + * to rehydrate), or a single failing pool never blocks boot. + */ + private async rehydratePools(): Promise { + const cfg = this.config; + if (!cfg?.registerPool || !cfg.listDatasourceRecords) return; + + let records: StoredDatasource[]; + try { + records = await cfg.listDatasourceRecords(); + } catch (err) { + this.options.logger?.warn?.('datasource rehydrate: listing records failed', err); + return; + } + + const runtime = records.filter((r) => r.origin === 'runtime' && (r.active ?? true)); + if (runtime.length === 0) return; + + let registered = 0; + for (const record of runtime) { + try { + await cfg.registerPool(record); + registered++; + } catch (err) { + this.options.logger?.warn?.(`datasource rehydrate: pool '${record.name}' failed`, err); + } + } + this.options.logger?.info?.( + `Rehydrated ${registered}/${runtime.length} runtime datasource pool(s) on boot`, + ); + } + + async destroy(): Promise { + this.service = undefined; + } + + // --- internals ----------------------------------------------------------- + + private toSpec(record: StoredDatasource): DatasourceConnectionSpec { + return { + name: record.name, + driver: record.driver, + config: record.config ?? {}, + external: record.external, + pool: record.pool, + }; + } + + /** Probe a connection via the driver factory: build โ†’ connect โ†’ ping โ†’ close. */ + private async probe( + factory: IDatasourceDriverFactory | undefined, + input: ProbeInput, + ): Promise { + if (!factory) { + return { ok: false, error: 'No driver factory is registered to test connections.' }; + } + if (!factory.supports(input.driver)) { + return { ok: false, error: `No driver factory supports driver '${input.driver}'.` }; + } + + let driver: any; + try { + driver = await factory.create({ + driver: input.driver, + config: input.config, + secret: input.secret, + external: input.external, + }); + } catch (err) { + return { ok: false, error: `Failed to build driver: ${errMsg(err)}` }; + } + + const startedAt = monotonicNow(); + try { + if (typeof driver?.connect === 'function') await driver.connect(); + // Prefer a cheap ping; fall back to the engine driver's health check, then + // a schema introspection round-trip โ€” whichever the handle exposes. + if (typeof driver?.ping === 'function') await driver.ping(); + else if (typeof driver?.checkHealth === 'function') await driver.checkHealth(); + else if (typeof driver?.introspectSchema === 'function') await driver.introspectSchema(); + const latencyMs = elapsedSince(startedAt); + let serverVersion: string | undefined; + try { + serverVersion = typeof driver?.serverVersion === 'function' ? await driver.serverVersion() : undefined; + } catch { + /* version is best-effort */ + } + return { ok: true, latencyMs, ...(serverVersion ? { serverVersion } : {}) }; + } catch (err) { + return { ok: false, error: errMsg(err) }; + } finally { + try { + if (typeof driver?.disconnect === 'function') await driver.disconnect(); + } catch { + /* best-effort teardown */ + } + } + } +} + +function safeGetService(ctx: PluginContext, name: string): T | undefined { + try { + return ctx.getService(name); + } catch { + return undefined; + } +} + +function errMsg(err: unknown): string { + return err instanceof Error ? err.message : String(err); +} + +/** Monotonic clock when available (avoids wall-clock skew); falls back to 0. */ +function monotonicNow(): number { + const perf = (globalThis as { performance?: { now?: () => number } }).performance; + return typeof perf?.now === 'function' ? perf.now() : 0; +} + +function elapsedSince(startedAt: number): number { + return Math.max(0, Math.round(monotonicNow() - startedAt)); +} diff --git a/packages/services/service-external-datasource/src/datasource-admin-service.ts b/packages/services/service-external-datasource/src/datasource-admin-service.ts new file mode 100644 index 000000000..d54fabb6d --- /dev/null +++ b/packages/services/service-external-datasource/src/datasource-admin-service.ts @@ -0,0 +1,297 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * DatasourceAdminService โ€” implements {@link IDatasourceAdminService} + * (ADR-0015 Addendum) on top of injected persistence + secret + driver probe + * callbacks. + * + * Like its federation sibling {@link ExternalDatasourceService}, this service is + * intentionally decoupled from the kernel: every side effect (connection probe, + * metadata read/write, secret write, bound-object count, hot pool (de)register) + * is injected via {@link DatasourceAdminServiceConfig}, so the lifecycle rules + * (origin gating, secret indirection, removal safety) are pure and unit-testable. + * + * Invariants enforced here, independent of the wiring: + * - Code-defined datasources (`origin: 'code'`) are read-only โ€” update/remove + * reject them, and create refuses a name a code datasource already owns. + * - A runtime datasource never shadows a code one (code wins on collision). + * - Credentials never persist in cleartext: the cleartext {@link SecretInput} + * transits create/update/test only; create/update write it to the secret + * store and persist only the returned `credentialsRef`. + * - Removal is refused while objects are still bound to the datasource. + */ + +import type { + IDatasourceAdminService, + DatasourceDraft, + SecretInput, + TestConnectionResult, + DatasourceSummary, +} from '@objectstack/spec/contracts'; +import type { Logger } from './external-datasource-service.js'; + +/** Datasource name rule (mirrors `DatasourceSchema.name`). */ +const NAME_RE = /^[a-z_][a-z0-9_]*$/; + +/** + * A persisted datasource record (subset of `Datasource`). `origin` distinguishes + * code-defined from runtime; `external.credentialsRef` is the opaque secret + * handle โ€” never a cleartext credential. + */ +export interface StoredDatasource { + name: string; + label?: string; + driver: string; + schemaMode?: 'managed' | 'external' | 'validate-only'; + config?: Record; + external?: (Record & { credentialsRef?: string }) | undefined; + pool?: Record; + active?: boolean; + origin?: 'code' | 'runtime'; + /** Package that defines a code-origin datasource, when known. */ + definedIn?: string; +} + +/** What a connection probe needs (cleartext secret is transient, never stored). */ +export interface ProbeInput { + driver: string; + config: Record; + /** Cleartext secret used for this probe only (e.g. password / DSN). */ + secret?: string; + external?: Record; + timeoutMs?: number; +} + +/** + * Injected dependencies. The plugin supplies real implementations backed by the + * driver registry, `IMetadataService` (runtime store), and the secret store; + * tests supply fakes. + */ +export interface DatasourceAdminServiceConfig { + /** Probe a connection live (driver connect + cheap round-trip). */ + probe: (input: ProbeInput) => Promise; + /** Read every datasource record (code + runtime). */ + listDatasourceRecords: () => Promise; + /** Read one datasource record by name. */ + getDatasourceRecord: (name: string) => Promise; + /** Persist a runtime datasource record into the runtime metadata store. */ + putDatasourceRecord: (record: StoredDatasource) => Promise; + /** Remove a runtime datasource record from the runtime metadata store. */ + deleteDatasourceRecord: (name: string) => Promise; + /** Encrypt + store a secret, returning an opaque `credentialsRef`. */ + writeSecret: (input: SecretInput, hint: { name: string }) => Promise; + /** Best-effort delete of a stored secret by ref (cleanup on remove/rewrap). */ + removeSecret?: (credentialsRef: string) => Promise; + /** Count objects bound to a datasource (removal blocked while > 0). */ + countBoundObjects: (datasource: string) => Promise; + /** Hot-(re)register a runtime datasource's connection pool after write. */ + registerPool?: (record: StoredDatasource) => Promise | void; + /** Tear down a runtime datasource's pool on remove. */ + unregisterPool?: (name: string) => Promise | void; + logger?: Logger; +} + +export class DatasourceAdminService implements IDatasourceAdminService { + constructor(private readonly config: DatasourceAdminServiceConfig) {} + + private get logger(): Logger | undefined { + return this.config.logger; + } + + async listDatasources(): Promise { + const records = await this.config.listDatasourceRecords(); + + // Group by name; code wins on collision, and a shadowed runtime row marks + // the effective (code) entry as conflicting. + const byName = new Map(); + for (const rec of records) { + const slot = byName.get(rec.name) ?? {}; + if (rec.origin === 'runtime') slot.runtime = rec; + else slot.code = rec; + byName.set(rec.name, slot); + } + + const summaries: DatasourceSummary[] = []; + for (const [name, slot] of byName) { + const effective = slot.code ?? slot.runtime; + if (!effective) continue; + summaries.push({ + name, + label: effective.label, + driver: effective.driver, + schemaMode: effective.schemaMode ?? 'managed', + origin: slot.code ? 'code' : 'runtime', + active: effective.active ?? true, + status: 'unvalidated', + ...(slot.code?.definedIn ? { definedIn: slot.code.definedIn } : {}), + ...(slot.code && slot.runtime ? { conflictsWithCode: true } : {}), + }); + } + return summaries; + } + + async testConnection(input: DatasourceDraft, secret?: SecretInput): Promise { + if (!input?.driver) { + return { ok: false, error: 'A driver is required to test a connection.' }; + } + const queryTimeoutMs = (input.external as { queryTimeoutMs?: number } | undefined)?.queryTimeoutMs; + try { + return await this.config.probe({ + driver: input.driver, + config: input.config ?? {}, + secret: secret?.value, + external: input.external, + ...(typeof queryTimeoutMs === 'number' ? { timeoutMs: queryTimeoutMs } : {}), + }); + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : String(err) }; + } + } + + async createDatasource(input: DatasourceDraft, secret?: SecretInput): Promise { + this.assertValidName(input?.name); + if (!input.driver) throw new Error('A driver is required to create a datasource.'); + + const existing = await this.config.getDatasourceRecord(input.name); + if (existing) { + if (existing.origin === 'code' || existing.origin === undefined) { + throw new Error( + `Cannot create datasource '${input.name}': a code-defined datasource owns this name (read-only).`, + ); + } + throw new Error(`Datasource '${input.name}' already exists.`); + } + + const record: StoredDatasource = { + ...this.toRecord(input), + origin: 'runtime', + }; + + if (secret) { + const credentialsRef = await this.config.writeSecret(secret, { name: input.name }); + record.external = { ...(record.external ?? {}), credentialsRef }; + } + + await this.config.putDatasourceRecord(record); + await this.tryRegisterPool(record); + return this.toSummary(record); + } + + async updateDatasource( + name: string, + patch: Partial, + secret?: SecretInput, + ): Promise { + const existing = await this.config.getDatasourceRecord(name); + if (!existing) throw new Error(`Datasource '${name}' not found.`); + if (existing.origin !== 'runtime') { + throw new Error(`Datasource '${name}' is code-defined and cannot be edited at runtime.`); + } + + // Merge patch over the existing record; `name`/`origin` are never patched. + const merged: StoredDatasource = { + ...existing, + ...(patch.label !== undefined ? { label: patch.label } : {}), + ...(patch.driver !== undefined ? { driver: patch.driver } : {}), + ...(patch.schemaMode !== undefined ? { schemaMode: patch.schemaMode } : {}), + ...(patch.config !== undefined ? { config: patch.config } : {}), + ...(patch.pool !== undefined ? { pool: patch.pool } : {}), + ...(patch.active !== undefined ? { active: patch.active } : {}), + name: existing.name, + origin: 'runtime', + }; + if (patch.external !== undefined) { + // Preserve the existing credentialsRef unless a new secret rewraps it. + merged.external = { ...patch.external, credentialsRef: existing.external?.credentialsRef }; + } + + if (secret) { + const prevRef = existing.external?.credentialsRef; + const credentialsRef = await this.config.writeSecret(secret, { name }); + merged.external = { ...(merged.external ?? {}), credentialsRef }; + if (prevRef && prevRef !== credentialsRef) await this.tryRemoveSecret(prevRef); + } + + await this.config.putDatasourceRecord(merged); + await this.tryRegisterPool(merged); + return this.toSummary(merged); + } + + async removeDatasource(name: string): Promise { + const existing = await this.config.getDatasourceRecord(name); + if (!existing) throw new Error(`Datasource '${name}' not found.`); + if (existing.origin !== 'runtime') { + throw new Error(`Datasource '${name}' is code-defined and cannot be removed at runtime.`); + } + + const bound = await this.config.countBoundObjects(name); + if (bound > 0) { + throw new Error( + `Cannot remove datasource '${name}': ${bound} object(s) are still bound to it.`, + ); + } + + await this.config.deleteDatasourceRecord(name); + if (existing.external?.credentialsRef) await this.tryRemoveSecret(existing.external.credentialsRef); + await this.tryUnregisterPool(name); + } + + // --- internals ----------------------------------------------------------- + + private assertValidName(name: string | undefined): void { + if (!name || !NAME_RE.test(name)) { + throw new Error( + `Invalid datasource name '${name ?? ''}': must match /^[a-z_][a-z0-9_]*$/.`, + ); + } + } + + private toRecord(input: DatasourceDraft): StoredDatasource { + return { + name: input.name, + ...(input.label !== undefined ? { label: input.label } : {}), + driver: input.driver, + ...(input.schemaMode !== undefined ? { schemaMode: input.schemaMode } : {}), + ...(input.config !== undefined ? { config: input.config } : {}), + ...(input.external !== undefined ? { external: input.external } : {}), + ...(input.pool !== undefined ? { pool: input.pool } : {}), + ...(input.active !== undefined ? { active: input.active } : {}), + }; + } + + private toSummary(record: StoredDatasource): DatasourceSummary { + return { + name: record.name, + label: record.label, + driver: record.driver, + schemaMode: record.schemaMode ?? 'managed', + origin: record.origin ?? 'runtime', + active: record.active ?? true, + status: 'unvalidated', + }; + } + + private async tryRegisterPool(record: StoredDatasource): Promise { + try { + await this.config.registerPool?.(record); + } catch (err) { + this.logger?.warn(`registerPool('${record.name}') failed`, err); + } + } + + private async tryUnregisterPool(name: string): Promise { + try { + await this.config.unregisterPool?.(name); + } catch (err) { + this.logger?.warn(`unregisterPool('${name}') failed`, err); + } + } + + private async tryRemoveSecret(credentialsRef: string): Promise { + try { + await this.config.removeSecret?.(credentialsRef); + } catch (err) { + this.logger?.warn(`removeSecret('${credentialsRef}') failed`, err); + } + } +} diff --git a/packages/services/service-external-datasource/src/external-datasource-service.ts b/packages/services/service-external-datasource/src/external-datasource-service.ts index 7a7a5ac7a..98a1bfa20 100644 --- a/packages/services/service-external-datasource/src/external-datasource-service.ts +++ b/packages/services/service-external-datasource/src/external-datasource-service.ts @@ -16,6 +16,8 @@ import type { RemoteTable, GenerateDraftOpts, ObjectDraft, + ImportObjectOpts, + ImportObjectResult, SchemaValidationResult, SchemaValidationReport, IntrospectedSchema, @@ -25,6 +27,8 @@ import type { SchemaDiffEntry } from '@objectstack/spec/shared'; import { suggestFieldType, isCompatible, + ExternalCatalogSchema, + type ExternalCatalog, type SqlDialect, type FieldType, } from '@objectstack/spec/data'; @@ -71,6 +75,18 @@ export interface ExternalDatasourceServiceConfig { getObject: (name: string) => Promise; /** List all object definitions (for `validateAll`). */ listObjects: () => Promise; + /** + * Persist a refreshed catalog snapshot as an `external_catalog` metadata + * record. Optional: when absent, `refreshCatalog` still returns the snapshot + * but does not cache it (e.g. dev runs without a writable metadata store). + */ + persistCatalog?: (catalog: ExternalCatalog) => Promise; + /** + * Persist an imported object definition as a live (runtime-origin) `object` + * metadata record. Optional: when absent, {@link ExternalDatasourceService.importObject} + * throws (the deployment is GitOps-only / has no writable metadata store). + */ + persistObject?: (name: string, definition: Record) => Promise; logger?: Logger; } @@ -211,9 +227,50 @@ export class ExternalDatasourceService implements IExternalDatasourceService { }; } - async refreshCatalog(datasource: string): Promise { + async importObject( + datasource: string, + remoteName: string, + opts: ImportObjectOpts = {}, + ): Promise { + if (!this.config.persistObject) { + throw new Error( + `importObject requires a writable metadata store, but none is wired ` + + `(datasource '${datasource}'). This deployment may be GitOps-only โ€” ` + + `use 'os datasource introspect' and commit the generated *.object.ts instead.`, + ); + } + + // Reuse the draft pipeline (type mapping, review notes, external binding). + const draft = await this.generateObjectDraft(datasource, remoteName, opts); + + // Apply the runtime-persona overrides on top of the draft definition. + const name = opts.name ?? draft.name; + const external = { + ...(draft.definition.external as Record), + ...(opts.writable ? { writable: true } : {}), + }; + const definition: Record = { + ...draft.definition, + name, + label: toLabel(name), + external, + }; + + await this.config.persistObject(name, definition); + this.logger?.info?.(`importObject: persisted '${name}' from ${datasource}.${remoteName}`, { + writable: opts.writable === true, + review: draft.review.length, + }); + + return { name, definition, review: draft.review }; + } + + async refreshCatalog(datasource: string): Promise { const schema = await this.config.introspect(datasource); - return { + // Parse through the Zod schema so the persisted record is canonical + // (defaults applied, shape validated) and matches the `external_catalog` + // metadata type the boot gate + Studio read back. + const catalog = ExternalCatalogSchema.parse({ name: `${datasource}_catalog`, datasource, snapshotAt: new Date().toISOString(), @@ -232,7 +289,19 @@ export class ExternalDatasourceService implements IExternalDatasourceService { })), }; }), - }; + }) as ExternalCatalog; + + // Best-effort cache: a failure to persist must not fail the refresh โ€” the + // caller still gets the live snapshot back. + if (this.config.persistCatalog) { + try { + await this.config.persistCatalog(catalog); + } catch (err) { + this.logger?.warn?.(`refreshCatalog: failed to persist '${catalog.name}'`, err); + } + } + + return catalog; } async validateObject(objectName: string): Promise { diff --git a/packages/services/service-external-datasource/src/index.ts b/packages/services/service-external-datasource/src/index.ts index 0f38672f4..1366dbafe 100644 --- a/packages/services/service-external-datasource/src/index.ts +++ b/packages/services/service-external-datasource/src/index.ts @@ -9,6 +9,21 @@ export type { Logger, } from './external-datasource-service.js'; +// Datasource lifecycle service (ADR-0015 Addendum) +export { DatasourceAdminService } from './datasource-admin-service.js'; +export type { + DatasourceAdminServiceConfig, + StoredDatasource, + ProbeInput, +} from './datasource-admin-service.js'; + +// Datasource lifecycle kernel plugin +export { DatasourceAdminServicePlugin } from './datasource-admin-plugin.js'; +export type { + DatasourceAdminServicePluginOptions, + SecretBinder, +} from './datasource-admin-plugin.js'; + // Kernel plugin export { ExternalDatasourceServicePlugin } from './plugin.js'; export type { ExternalDatasourceServicePluginOptions } from './plugin.js'; diff --git a/packages/services/service-external-datasource/src/plugin.ts b/packages/services/service-external-datasource/src/plugin.ts index 54be8f431..67234ea1a 100644 --- a/packages/services/service-external-datasource/src/plugin.ts +++ b/packages/services/service-external-datasource/src/plugin.ts @@ -25,6 +25,7 @@ interface MetadataServiceLike { getObject?: (name: string) => Promise; listObjects?: () => Promise; list?: (type: string) => Promise; + register?: (type: string, name: string, data: unknown) => Promise | void; } export interface ExternalDatasourceServicePluginOptions { @@ -78,6 +79,21 @@ export class ExternalDatasourceServicePlugin implements Plugin { ((metadata?.listObjects ? await metadata.listObjects() : await metadata?.list?.('object')) ?? []) as ObjectLike[], + // Persist the refreshed snapshot as an `external_catalog` metadata record + // so the boot gate + Studio's schema browser can read it without + // re-introspecting. No-op when the metadata service can't write. + ...(metadata?.register + ? { + persistCatalog: async (catalog) => { + await metadata.register!('external_catalog', catalog.name, catalog); + }, + // Runtime "Import as Object": persist a federated object so it's + // immediately queryable, no git commit required (ADR-0015 Addendum). + persistObject: async (name, definition) => { + await metadata.register!('object', name, definition); + }, + } + : {}), logger: this.options.logger, }; diff --git a/packages/services/service-settings/src/index.ts b/packages/services/service-settings/src/index.ts index 486921103..560afc24b 100644 --- a/packages/services/service-settings/src/index.ts +++ b/packages/services/service-settings/src/index.ts @@ -10,6 +10,10 @@ export { type CryptoAdapter, NoopCryptoAdapter, } from './crypto-adapter.js'; +// Default ICryptoProvider for dev / self-host kernels (no KMS). Hosts swap in +// a KMS-backed provider for production; exported so other subsystems (e.g. the +// runtime-UI datasource secret binder) can reuse the same dev wrapping. +export { InMemoryCryptoProvider } from './in-memory-crypto-provider.js'; export { type SettingsActionHandler, type SettingsAuditSink, diff --git a/packages/spec/src/contracts/datasource-admin-service.ts b/packages/spec/src/contracts/datasource-admin-service.ts new file mode 100644 index 000000000..5614d351f --- /dev/null +++ b/packages/spec/src/contracts/datasource-admin-service.ts @@ -0,0 +1,119 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * IDatasourceAdminService โ€” runtime datasource lifecycle contract + * (ADR-0015 Addendum: Runtime UI-Created Datasources). + * + * Where {@link IExternalDatasourceService} covers *federation* (introspection, + * object drafting, schema validation) of datasources that already exist, this + * service covers their *lifecycle*: testing a connection before saving, + * creating / updating / removing a **runtime** datasource (`origin: 'runtime'`), + * and listing all datasources with their provenance + health. + * + * Code-defined datasources (`origin: 'code'`, authored as `*.datasource.ts`) + * are read-only here: `updateDatasource` / `removeDatasource` reject them, and + * a runtime datasource never shadows a code one of the same name (code wins). + * + * Credentials are never persisted in cleartext: callers pass a {@link SecretInput} + * separately from the connection `config`; the implementation encrypts it into + * the secret store (`sys_secret`) and persists only an opaque `credentialsRef`. + */ + +/** Provenance of a datasource definition. */ +export type DatasourceOrigin = 'code' | 'runtime'; + +/** + * A cleartext secret (password or full connection string) supplied for a + * create/update/test call. Never persisted as-is โ€” encrypted into the secret + * store, with only the returned handle (`credentialsRef`) kept on the record. + */ +export interface SecretInput { + /** The cleartext value to encrypt (e.g. password or connection string). */ + value: string; + /** Optional secret-store namespace (defaults to `'datasource'`). */ + namespace?: string; + /** Optional secret-store key (defaults to the datasource name). */ + key?: string; +} + +/** + * The connection definition a caller supplies to test/create/update. A subset + * of `Datasource` โ€” server-managed fields (`origin`) are never accepted from + * the client. + */ +export interface DatasourceDraft { + name: string; + label?: string; + driver: string; + schemaMode?: 'managed' | 'external' | 'validate-only'; + /** Driver-specific connection config (host, port, database, โ€ฆ). No secrets. */ + config?: Record; + /** External federation settings (required when schemaMode != 'managed'). */ + external?: Record; + pool?: Record; + active?: boolean; +} + +/** Result of probing a connection (live driver connect + cheap round-trip). */ +export interface TestConnectionResult { + ok: boolean; + /** Round-trip latency of the probe, when the connection succeeded. */ + latencyMs?: number; + /** Driver-reported server version, when available. */ + serverVersion?: string; + /** Human-readable failure reason, when `ok === false`. */ + error?: string; +} + +/** A datasource with its provenance and current health (no secrets). */ +export interface DatasourceSummary { + name: string; + label?: string; + driver: string; + schemaMode: 'managed' | 'external' | 'validate-only'; + origin: DatasourceOrigin; + active: boolean; + /** Validation health: `unvalidated` until the first validate/test runs. */ + status: 'ok' | 'error' | 'unvalidated'; + /** Package id that defines a code-origin datasource (omitted for runtime). */ + definedIn?: string; + /** True when a runtime row is shadowed by a code definition of the same name. */ + conflictsWithCode?: boolean; +} + +/** + * Runtime datasource lifecycle service. Registered into the kernel as the + * `'datasource-admin'` service; consumed by the REST layer and Studio wizard. + */ +export interface IDatasourceAdminService { + /** List every datasource (code + runtime) with provenance and health. */ + listDatasources(): Promise; + + /** + * Probe a connection without persisting anything. Accepts an unsaved draft + * so the wizard can validate credentials before "Save". + */ + testConnection(input: DatasourceDraft, secret?: SecretInput): Promise; + + /** + * Persist a new runtime datasource (`origin: 'runtime'`, environment-scoped). + * Rejects when a code-defined datasource of the same name exists. + */ + createDatasource(input: DatasourceDraft, secret?: SecretInput): Promise; + + /** + * Patch an existing runtime datasource. Rejects for code-defined datasources. + * Passing `secret` re-wraps the stored credential. + */ + updateDatasource( + name: string, + patch: Partial, + secret?: SecretInput, + ): Promise; + + /** + * Remove a runtime datasource. Rejects for code-defined ones and while + * objects are still bound to it. + */ + removeDatasource(name: string): Promise; +} diff --git a/packages/spec/src/contracts/datasource-driver-factory.ts b/packages/spec/src/contracts/datasource-driver-factory.ts new file mode 100644 index 000000000..aa17c05c0 --- /dev/null +++ b/packages/spec/src/contracts/datasource-driver-factory.ts @@ -0,0 +1,77 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * IDatasourceDriverFactory โ€” host-provided capability that builds a live driver + * from a connection spec (ADR-0015 Addendum ยง3.5). + * + * The framework deliberately ships no universal "driver-by-id" registry โ€” + * concrete drivers (`SqlDriver`, `MongoDBDriver`, `TursoDriver`, โ€ฆ) are + * constructed by the host stack and registered as live connections. The + * runtime-datasource lifecycle (`IDatasourceAdminService`) needs to build a + * driver from an *unsaved* draft โ€” to probe a connection before "Save", and to + * hot-register a pool after create/update โ€” so the host exposes this factory + * as the `'datasource-driver-factory'` service. + * + * When no factory is registered, or none `supports()` a given driver id, the + * admin service degrades gracefully: `testConnection` returns + * `{ ok: false, error }` and create/update skip hot pool registration (the + * driver is picked up on the next boot instead). + * + * Security: the cleartext `secret` on {@link DatasourceConnectionSpec} is used + * only to open the live connection. Factories MUST NOT persist or log it. + */ + +/** Everything needed to construct one live driver connection. */ +export interface DatasourceConnectionSpec { + /** Datasource name, when building for an existing/named datasource. */ + name?: string; + /** Driver id (e.g. `'postgres'`, `'sqlite'`, `'mongodb'`). */ + driver: string; + /** Driver-specific connection config (host, port, database, โ€ฆ). No secrets. */ + config: Record; + /** Cleartext secret (password / DSN) injected for this connection only. */ + secret?: string; + /** External federation settings (timeouts, allowed schemas, โ€ฆ). */ + external?: Record; + /** Connection pool settings. */ + pool?: Record; +} + +/** + * A live (or lazily-connecting) driver handle. Intentionally structural and + * fully optional so any concrete driver satisfies it โ€” the admin service uses + * whatever capabilities are present and skips the rest. + */ +export interface DatasourceDriverHandle { + /** Open the connection / pool. */ + connect?(): Promise; + /** Close the connection / pool. */ + disconnect?(): Promise; + /** Cheap liveness round-trip (preferred for probes). */ + ping?(): Promise; + /** Introspect the live schema (fallback probe when `ping` is absent). */ + introspectSchema?(): Promise; + /** Liveness check on the underlying engine driver (probe fallback). */ + checkHealth?(): Promise; + /** Driver-reported server version, when available. */ + serverVersion?(): Promise; + /** + * Escape hatch: the concrete engine driver to hand to + * `IDataEngine.registerDriver()` when hot-registering a pool. When present + * the admin service registers *this* (whose `.name` must equal the + * datasource name for routing) instead of the handle itself; absent โ‡’ the + * handle is assumed to be the driver. Never serialized. + */ + driver?: unknown; +} + +/** Host-provided factory that builds drivers from connection specs. */ +export interface IDatasourceDriverFactory { + /** True if this factory can build a driver for the given driver id. */ + supports(driverId: string): boolean; + /** + * Build a driver instance for the spec. Implementations may return a + * not-yet-connected handle; the caller calls `connect()` when needed. + */ + create(spec: DatasourceConnectionSpec): Promise | DatasourceDriverHandle; +} diff --git a/packages/spec/src/contracts/external-datasource-service.ts b/packages/spec/src/contracts/external-datasource-service.ts index b964ffdd2..ee2f5d50f 100644 --- a/packages/spec/src/contracts/external-datasource-service.ts +++ b/packages/spec/src/contracts/external-datasource-service.ts @@ -14,6 +14,7 @@ */ import type { SchemaDiffEntry } from '../shared/external-errors'; +import type { ExternalCatalog } from '../data/external-catalog.zod'; /** * A remote table discovered via introspection, filtered by the datasource's @@ -64,6 +65,32 @@ export interface ObjectDraft { review: Array<{ column: string; remoteType: string; note: string }>; } +/** + * Options for {@link IExternalDatasourceService.importObject}: a superset of + * the draft options, plus the runtime-persona choices the Studio "Import as + * Object" action exposes. + */ +export interface ImportObjectOpts extends GenerateDraftOpts { + /** Override the auto-derived object name (snake_case). */ + name?: string; + /** + * Mark the imported object writable (`object.external.writable`). Default + * `false` โ€” federated objects are read-only unless explicitly opted in (and + * the datasource must also set `external.allowWrites`, ADR-0015 Gate 3). + */ + writable?: boolean; +} + +/** Outcome of importing a remote table as a live federated object. */ +export interface ImportObjectResult { + /** The object name as persisted. */ + name: string; + /** The persisted object definition (parseable by `ObjectSchema`). */ + definition: Record; + /** Review notes carried over from the draft (lossy/unknown column mappings). */ + review: ObjectDraft['review']; +} + /** Per-object validation outcome. */ export interface SchemaValidationResult { ok: boolean; @@ -103,11 +130,26 @@ export interface IExternalDatasourceService { ): Promise; /** - * Refresh and persist the cached remote schema snapshot - * (`external_catalog`). Returns the snapshot. (Persistence lands with the - * `external_catalog` metadata type.) + * Persist a remote table as a live, runtime-origin federated `Object` so it + * is immediately queryable โ€” the backend of the Studio "Import as Object" + * action (ADR-0015 ยง6.4, Addendum runtime persona). Builds the draft via + * {@link generateObjectDraft}, applies the import overrides, and writes it + * through the metadata store. Requires a writable metadata store: throws when + * none is wired (e.g. a GitOps-only / read-only deployment). + */ + importObject( + datasource: string, + remoteName: string, + opts?: ImportObjectOpts, + ): Promise; + + /** + * Refresh and persist the cached remote schema snapshot as an + * `external_catalog` metadata record (conventionally `_catalog`). + * Returns the snapshot. Persistence is best-effort: when no catalog store is + * wired the snapshot is still returned, just not cached. */ - refreshCatalog(datasource: string): Promise; + refreshCatalog(datasource: string): Promise; /** Validate one federated object against the live remote table. */ validateObject(objectName: string): Promise; diff --git a/packages/spec/src/contracts/index.ts b/packages/spec/src/contracts/index.ts index 0448fc76c..97ee1ffb0 100644 --- a/packages/spec/src/contracts/index.ts +++ b/packages/spec/src/contracts/index.ts @@ -49,6 +49,8 @@ export * from './embedder.js'; export * from './provisioning-service.js'; export * from './schema-diff-service.js'; export * from './external-datasource-service.js'; +export * from './datasource-admin-service.js'; +export * from './datasource-driver-factory.js'; export * from './deploy-pipeline-service.js'; export * from './tenant-router.js'; export * from './app-lifecycle-service.js'; diff --git a/packages/spec/src/data/datasource.zod.ts b/packages/spec/src/data/datasource.zod.ts index b6221a85e..ee5c59228 100644 --- a/packages/spec/src/data/datasource.zod.ts +++ b/packages/spec/src/data/datasource.zod.ts @@ -224,6 +224,21 @@ export const DatasourceSchema = lazySchema(() => z.object({ * Required when `schemaMode !== 'managed'`; forbidden otherwise. */ external: ExternalDatasourceSettingsSchema.optional(), + + /** + * Provenance (ADR-0015 Addendum) + * + * Server-managed, read-only. Distinguishes code-defined datasources + * (`code` โ€” authored as `*.datasource.ts`, GitOps-owned, read-only in the + * UI) from runtime datasources (`runtime` โ€” created via the Studio wizard, + * persisted in the runtime metadata store, environment-scoped, editable). + * + * Never accepted from client input: the runtime stamps `code` on artefact + * load and `runtime` on UI create. Defaults to `code` for artefact-defined + * datasources that predate this field. + */ + origin: z.enum(['code', 'runtime']).default('code') + .describe('Datasource provenance (server-managed, read-only)'), }).superRefine((ds, ctx) => { if (ds.schemaMode !== 'managed' && !ds.external) { ctx.addIssue({ diff --git a/packages/spec/src/kernel/metadata-plugin.zod.ts b/packages/spec/src/kernel/metadata-plugin.zod.ts index df6b194a9..24fea657d 100644 --- a/packages/spec/src/kernel/metadata-plugin.zod.ts +++ b/packages/spec/src/kernel/metadata-plugin.zod.ts @@ -587,7 +587,12 @@ export const DEFAULT_METADATA_TYPE_REGISTRY: MetadataTypeRegistryEntry[] = [ { type: 'job', label: 'Background Job', filePatterns: ['**/*.job.ts', '**/*.job.yml', '**/*.job.json'], supportsOverlay: false, allowOrgOverride: true, allowRuntimeCreate: true, supportsVersioning: false, executionPinned: false, loadOrder: 80, domain: 'automation' }, // System Protocol - { type: 'datasource', label: 'Datasource', filePatterns: ['**/*.datasource.ts', '**/*.datasource.yml'], supportsOverlay: false, allowOrgOverride: false, allowRuntimeCreate: false, supportsVersioning: false, executionPinned: false, loadOrder: 5, domain: 'system' }, + // `datasource`: runtime-creatable (ADR-0015 Addendum) โ€” the Studio wizard + // persists `origin: 'runtime'` datasources into the runtime metadata store. + // Code-defined (`origin: 'code'`) datasources remain read-only and win on + // name collision; record-level read-only gating is enforced by origin, not + // by this flag. No per-org overlay (a datasource = one physical connection). + { type: 'datasource', label: 'Datasource', filePatterns: ['**/*.datasource.ts', '**/*.datasource.yml'], supportsOverlay: false, allowOrgOverride: false, allowRuntimeCreate: true, supportsVersioning: false, executionPinned: false, loadOrder: 5, domain: 'system' }, { type: 'external_catalog', label: 'External Catalog', filePatterns: ['**/*.external-catalog.ts', '**/*.external-catalog.yml', '**/*.external-catalog.json'], supportsOverlay: false, allowOrgOverride: false, allowRuntimeCreate: true, supportsVersioning: false, executionPinned: false, loadOrder: 6, domain: 'system' }, { type: 'translation', label: 'Translation', filePatterns: ['**/*.translation.ts', '**/*.translation.yml', '**/*.translation.json'], supportsOverlay: true, allowOrgOverride: true, allowRuntimeCreate: true, supportsVersioning: false, executionPinned: false, loadOrder: 90, domain: 'system' }, { type: 'router', label: 'Router', filePatterns: ['**/*.router.ts'], supportsOverlay: false, allowOrgOverride: false, allowRuntimeCreate: false, supportsVersioning: false, executionPinned: false, loadOrder: 40, domain: 'system' }, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a09a3261a..46e8f0148 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -501,6 +501,9 @@ importers: '@objectstack/service-cache': specifier: workspace:* version: link:../services/service-cache + '@objectstack/service-external-datasource': + specifier: workspace:* + version: link:../services/service-external-datasource '@objectstack/service-feed': specifier: workspace:* version: link:../services/service-feed