feat(datasource): runtime-UI-creatable datasources (ADR-0015 Addendum)#1396
Merged
Conversation
Enable adding a datasource from the UI/API at runtime alongside
code-defined ones, with a two-tier model:
- origin:'code' — declared in *.datasource.ts, GitOps-managed, read-only
- origin:'runtime'— created via API, editable
Spec/contracts:
- DatasourceAdminService contract: list/test/create/update/remove
- DatasourceDriverFactory contract (host-provided; no universal registry),
driver escape-hatch + checkHealth on the handle
Service (decoupled + kernel plugin):
- DatasourceAdminService: origin gating (code read-only, code wins on
name collision), removal refused while objects bound, secret indirection
- DatasourceAdminServicePlugin: lazy per-call resolution of metadata/data,
probe (build→connect→ping→close), hot pool (de)register, fail-closed
SecretBinder (secret-bearing create/update throws when absent)
Runtime host glue:
- default datasource driver factory (postgres/sqlite/mongodb/memory)
- datasource secret binder (encrypt→sys_secret, persist only credentialsRef)
- AppPlugin registers code datasources in-memory only (never persisted)
Metadata:
- registerInMemory(): in-memory-only registration so GitOps datasources
are listable without leaking into the runtime DB store
REST:
- register datasource routes unconditionally (no longer gated behind the
package service); 204 on remove
Security invariant: credentials never persisted in cleartext — only an
opaque credentialsRef→sys_secret handle; cleartext transits create/update/
test only; secret unbound on remove.
Fix(hono-server): guard the response stream controller against double-close
and return a null-body response on res.end(), fixing an ERR_INVALID_STATE
process crash on 204 routes (also affected 5 latent DELETE routes in REST).
Add External Datasource Studio and API objectui@1c8f7753898b6283ffc528d2fb2ee85f68886ca5
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
dependencies
Runtime (UI-created) datasources persist only a credentialsRef handle, so after a restart their live connection pools must be rebuilt from the cipher store. Add a resolve() path that dereferences sys_secret -> cleartext and a boot rehydration pass that rebuilds every active runtime pool. - runtime binder: resolve(credentialsRef) reads the sys_secret row and decrypts under the same (namespace,key) AAD; degrades to undefined on a missing row / unreadable engine / decrypt failure so boot never crashes - plugin: registerPool now resolves the credentialsRef before create() so password DBs get a working pool; start() runs rehydratePools() over all active origin:'runtime' records (skips code-defined and inactive) - serve: lazyEngine gains find() so the binder can read sys_secret - tests: binder round-trip/unbind/foreign-ref/no-find; plugin rehydration rebuilds only active runtime pools and hands decrypted secret to factory Security invariant preserved: cleartext only transits create/update/test; only sys_secret:<id> is persisted on the artefact.
The Gate-2 boot-validation plugin and the external_catalog metadata type already existed but were not connected end to end. - serve: register ExternalValidationPlugin into the boot sequence so Gate 2 (kernel:ready -> validateAll -> onMismatch policy) actually runs; no-op when the external-datasource service is absent - service: refreshCatalog now parses its snapshot through ExternalCatalogSchema and persists it as an external_catalog metadata record via an injected persistCatalog dep; best-effort so a persist failure still returns the live snapshot. Plugin wires persistCatalog -> metadata.register - spec: tighten IExternalDatasourceService.refreshCatalog return to ExternalCatalog and refresh the stale 'persistence lands later' note - tests: catalog persistence, read-only/throwing store, canonicalised shape - docs: mark P3 done in the federation impl plan Schema-mode -> driver injection (DDL gate for runtime external datasources) was already wired in createDefaultDatasourceDriverFactory.
github-actions
Bot
added
the
documentation
The Addendum flipped the 'datasource' metadata type to allowRuntimeCreate:true (UI 'Add Datasource'), but overlay-precedence.test.ts still asserted it was denied for any per-org write. Move datasource/datasources from the denied cohort to the runtime-creatable cohort: brand-new runtime datasources now succeed (code-defined collisions are still refused via artifact provenance, covered in protocol-meta.test.ts). Fixes 2 stale failures on this branch.
The query_data tool already injected a LIMIT and SchemaRetriever already
badges federated objects, but a slow remote warehouse could hang the tool
loop indefinitely. Add a per-query timeout that applies only to federated
(external) objects, resolved from the datasource's external.queryTimeoutMs
(fallback externalQueryTimeoutMs, default 30s). Managed/local objects are
untouched. On expiry the model gets a clear 'exceeded the Nms timeout' error.
- query-data.tool: withTimeout() wrapper + resolveExternalTimeout() over
metadata.get('datasource', ...); only the external branch is wrapped
- tests: datasource-declared timeout, fallback timeout, fast success, and
managed objects never wrapped (datasource timeout never consulted)
- docs: mark P4 done; MVP (P1-P4 + P6) complete in the federation plan
ExternalValidationPlugin now arms a per-datasource setInterval for every federated datasource declaring external.validation.checkIntervalMs (ADR-0015 §5.2). Each tick re-runs validateAll() and emits an external.schema.drift event per drifted object on the kernel bus (ExternalSchemaDriftEvent), for audit/notification consumers. Drift past boot is observational: the checker never throws or aborts the process (unlike boot validation). Timers unref() and are cleared on stop(); re-arming clears prior timers so intervals can't accumulate. Tests: event emission, validateAll-rejects no-op, selective scheduling, firing interval, re-arm idempotence, no-metadata no-op (runtime 324 green).
Bridge the runtime persona: turn a browsed remote table into a live,
immediately-queryable federated object without a git commit (the GitOps
introspect->commit path stays separate).
- spec: IExternalDatasourceService.importObject(datasource, remoteName, opts?)
-> ImportObjectResult; ImportObjectOpts adds name override + writable opt-in
(still gated by datasource external.allowWrites, Gate 3).
- service: reuses the generateObjectDraft pipeline, applies overrides, persists
via injected persistObject. Throws when no writable metadata store is wired
(GitOps-only) and when the remote table is missing (before any write).
- plugin: persistObject -> metadata.register("object", ...) (runtime origin).
- rest: POST /datasources/:name/external/tables/:remote/import (201 / 503 / 400).
Tests: read-only default, name+writable overrides, draft-option forwarding,
no-store + missing-table guards (service suite 47 green).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Enable adding a datasource from the UI/API at runtime, alongside code-defined ones — the low-code persona's "connect to a remote business DB in Studio" goal (ADR-0015 Addendum). Two-tier model:
origin:'code'— declared in*.datasource.ts, GitOps-managed, read-onlyorigin:'runtime'— created via API, editableChanges
Spec / contracts
DatasourceAdminServicecontract —list/test/create/update/removeDatasourceDriverFactorycontract — host-provided (no universal driver-by-id registry); driver escape-hatch +checkHealthon the handleService (decoupled core + kernel plugin)
DatasourceAdminService— origin gating (code read-only, code wins on name collision), removal refused while objects are bound, secret indirection. Pure + unit-tested.DatasourceAdminServicePlugin— lazy per-call resolution ofmetadata/data(plugin init order safe), connection probe (build→connect→ping→close), hot pool (de)register, fail-closedSecretBinder(secret-bearing create/update throws when no secret store is wired)Runtime host glue
sys_secret, persist onlycredentialsRef)AppPluginregisters code datasources in-memory only (never persisted)Metadata
registerInMemory()— in-memory-only registration so GitOps datasources are listable without leaking into the runtime DB storeREST
Fix (hono-server adapter)
res.end(), fixing anERR_INVALID_STATEprocess crash on 204 routes (also fixes 5 latent.status(204).end()DELETE routes in REST)Security invariant
Credentials are never persisted in cleartext — only an opaque
credentialsRef → sys_secrethandle is stored. Cleartext transits create/update/test only; the secret is unbound on remove. Verified by DB inspection:sys_secretholds aes-256-gcm ciphertext, no cleartext leak.Verification
app-crmdev server:GET /datasources→ code (crm_primary/crm_analytics) + runtime entriesPOST /datasources/test(sqlite) →ok:true, serverVersion:3.53.1POST /datasources(+secret) → 201,origin:runtimePATCH/DELETEon code → 400 read-only;DELETEruntime → 204 (server survives), secret unboundsys_metadatadatasource rows = 0 (code datasources never persisted)Known dev limitation (not a defect)
In dev there is no writable
datasource:loader, so runtime datasources live in-memory and don't survive restart; DB-backed deployments persist them. Boot-time pool rehydration with secret decryption is a later phase (P-D).