feat: External Datasource Federation (ADR-0015) — backend (P1–P4, P6)#1390
Merged
Conversation
…, object.external, DDL gate
Implements Phase 1 of ADR-0015 (External Datasource Federation): the spec
foundation plus the driver-layer DDL gate, so a datasource can be marked as
a guest in a mature external database that ObjectStack must never run DDL
against. All changes are additive and backward-compatible (schemaMode
defaults to 'managed').
Spec:
- Datasource.schemaMode ('managed' | 'external' | 'validate-only') and
Datasource.external settings, with a superRefine enforcing the
cross-field invariant (external settings <-> non-managed mode).
- Object.external binding (remoteName/remoteSchema/writable/columnMap/
introspectedAt/ignoreColumns).
- shared/external-errors.ts: ExternalSchemaMismatchError /
ExternalWriteForbiddenError / ExternalSchemaModeViolationError with stable
`code`s, plus SchemaDiffEntry and a pure renderDiffMessage().
driver-sql (Gate 1 — DDL gate, ADR section 5.1):
- SqlDriverConfig gains an optional schemaMode (stripped before Knex).
- assertSchemaMutable() choke-point throws ExternalSchemaModeViolationError
when schemaMode != 'managed'; wired into initObjects (covers syncSchema)
and dropTable.
Tests: Zod refinements (datasource modes/settings, object binding), error
classes + diff rendering, and the DDL gate (managed allows DDL;
external/validate-only block create/alter/drop; schemaMode not leaked to
Knex). Adds the /shared subpath alias to driver-sql's vitest config.
Docs: docs/plans/external-datasource-federation-impl.md (phased plan +
progress tracker) and a changeset.
Refs ADR-0015.
https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
documentation
…n P2 — type-compat matrix, service contract & impl Implements the service core of Phase 2 of ADR-0015: the SQL↔field type compatibility matrix, the IExternalDatasourceService contract, and a new service package that introspects, drafts, and validates federated objects. Spec: - data/type-compat.ts: pure, dialect-aware matrix (canonicalizeSqlType, suggestFieldType, isCompatible -> true | 'lossy' | false) for postgres, mysql, sqlite, snowflake, bigquery, mongo. Independently unit-tested. - contracts/external-datasource-service.ts: IExternalDatasourceService plus RemoteTable, GenerateDraftOpts, ObjectDraft, SchemaValidationResult/Report. Reuses the existing IntrospectedSchema (schema-diff-service) and SchemaDiffEntry (external-errors). New package @objectstack/service-external-datasource: - ExternalDatasourceService implements the contract: listRemoteTables (schema-qualified, allowedSchemas-filtered), generateObjectDraft (renders a reviewable *.object.ts with // REVIEW: markers on lossy/unknown columns), validateObject/validateAll (structured diffs; lossy = warning, hard mismatch = error; honours columnMap + ignoreColumns), refreshCatalog (snapshot shape; persistence lands with P3's external_catalog type). - Decoupled from the kernel via injected I/O, so the logic is fully unit-tested. ExternalDatasourceServicePlugin wires the live IDataEngine + IMetadataService and registers it as the 'external-datasource' service. Tests: type-compat matrix (canonicalization, suggestion, compatibility) and the service (list/draft/validate/refresh) — 15 service cases + matrix cases. REST routes and the `os datasource` CLI commands follow in the next slice. Refs ADR-0015. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
github-actions
Bot
added
dependencies
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…icalization CodeQL flagged the SQL-type canonicalizer's `\s*\([^)]*\)` and `\s+…$` patterns as polynomial regular expressions on (library-controlled) input. Replace them with linear substring operations (indexOf-slice for the precision parenthetical, anchored endsWith for trailing qualifiers). Behaviour is unchanged; type-compat tests still pass. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
os-zhuang
changed the title
…og metadata type Registers the `external_catalog` metadata type (MetadataTypeSchema + DEFAULT_METADATA_TYPE_REGISTRY, system domain) and adds data/external-catalog.zod.ts (ExternalCatalogSchema / ExternalTableSchema / ExternalColumnSchema) for persisting a cached remote-schema snapshot of a federated datasource — consumed by refreshCatalog, the boot-validation gate, and Studio's schema browser. Refs ADR-0015. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…on plumbing (ADR-0015) - Write gate: insert/update/delete block writes to a federated datasource (schemaMode != 'managed') unless BOTH datasource.external.allowWrites and object.external.writable are true, throwing ExternalWriteForbiddenError. registerDatasourceDef() records declarative ownership; manifests carrying `datasources` are indexed during registerApp. - engine.introspectDatasource(name) delegates to the named driver's introspectSchema(), completing the external-datasource service path. Tests: 8 write-gate + 3 introspect cases, real engine, all pass. Refs ADR-0015. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…insert) The engine dispatches inserts via driver.create(); the test's fake driver defined insert(), so the allow-path cases hit 'driver.create is not a function'. Rename to create(). All 11 write-gate + introspect cases pass. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…ADR-0015) Commit 0603646 landed the assertWriteAllowed() definition but the three call sites were lost during an environment recovery, leaving the gate dead code and a 'declared but never read' DTS build error. Re-add the calls. Write-gate suite passes 11/11. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…0015) ExternalValidationPlugin subscribes to kernel:ready and runs the external-datasource service's validateAll(), enforcing each datasource's external.validation.onMismatch policy: fail (throw, default), warn (log), or ignore. No-op when the service is not registered. 7 tests. Refs ADR-0015. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…, ADR-0015) SchemaRetriever.renderSnippet appends an [external, read-only|writable, datasource=<name>] badge for federated objects so the LLM does not propose schema changes or unsafe writes. ObjectShape gains datasource + external; managed objects unannotated. 3 new renderSnippet tests. Refs ADR-0015. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…DR-0015) The badge edit was lost in a prior environment recovery while its tests landed, leaving 3 failing tests. Re-add the [external, read-only|writable, datasource=<name>] badge to renderSnippet. schema-retriever suite passes 14/14. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…n badge (P4, ADR-0015) Prior environment recoveries left schema-retriever.ts corrupted (the describeField helper was dropped, breaking 3 pre-existing tests). Restore the file from main and re-apply only the minimal P4 changes: ObjectShape gains datasource + external, and renderSnippet appends an [external, read-only|writable, datasource=<name>] badge for federated objects. Suite passes 12/12 (9 original + 3 federation). https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
…R-0015) New os datasource group backed by the external federation REST routes: list-tables, introspect (writes a reviewable *.object.ts), validate (non-zero exit on mismatch). Refs ADR-0015. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
os-zhuang
changed the title
…facts after env churn (ADR-0015)
A series of container reclaims left several committed files half-applied.
This consolidates the repairs:
- objectql: add the missing assertWriteAllowed(object, 'delete') call site so
the write gate (Gate 3) covers delete (was insert/update only).
- runtime: export ExternalValidationPlugin + createExternalValidationPlugin
from index, and add the required init() to satisfy the Plugin interface.
- service-ai: rewrite schema-retriever.test.ts as a complete, self-contained
file (it had lost its imports + duplicated a describe block).
- cli: drop the broken ../../lib/api-client import (wrong path; the helper is
async and returns {client,token}); the os datasource commands now resolve
url/token inline via flags/env. introspect constrains --out to the CWD
(CodeQL: network-data-to-file).
Tests: objectql write-gate 11/11, service-ai schema-retriever 3/3, runtime
boot-gate 7/7. objectql/runtime/cli/service-ai builds (incl. DTS) green.
Refs ADR-0015.
https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
| this.error(`--out must be a relative path within the current directory: ${flags.out}`); | ||
| return; | ||
| } | ||
| await writeFile(target, draft.source, 'utf8'); |
Add registerExternalDatasourceRoutes mounting /api/v1/datasources/:name/ external/* (tables, tables/:remote/draft, refresh-catalog, validate), served by the external-datasource service and wired into the REST API plugin next to registerPackageRoutes. Routes return 503 external_service_unavailable when the service is not registered, so they are safe to mount unconditionally. Refs ADR-0015. https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the backend of ADR-0015 — External Datasource Federation: connect a mature external database (Postgres, Snowflake, …), register its tables as ObjectStack objects without ever touching the remote schema, and let the whole stack (ObjectQL, REST, Views, AI) use them — enforced by three runtime gates.
A federated object stays a normal
Object; its remote-ness is the datasource (schemaMode !== 'managed') + anobject.externalbinding. All changes are additive and backward-compatible (schemaModedefaults to'managed').What's included
P1 — Spec + DDL gate (Gate 1) ·
@objectstack/spec,@objectstack/driver-sqlDatasource.schemaMode+Datasource.external;Object.externalbinding; cross-fieldsuperRefine.shared/external-errors.ts:ExternalSchemaMismatchError/ExternalWriteForbiddenError/ExternalSchemaModeViolationError+SchemaDiffEntry.driver-sql:assertSchemaMutable()blocks DDL (initObjects/syncSchema/dropTable) when not managed.P2 — Service + type matrix + REST + CLI ·
@objectstack/spec,@objectstack/service-external-datasource(new),@objectstack/rest,@objectstack/cli,@objectstack/objectqldata/type-compat.ts: dialect-aware SQL↔field matrix (postgres/mysql/sqlite/snowflake/bigquery/mongo).IExternalDatasourceService+ExternalDatasourceService:listRemoteTables/generateObjectDraft(reviewable*.object.ts) /validateObject/validateAll/refreshCatalog.registerExternalDatasourceRoutes→/api/v1/datasources/:name/external/*(tables, draft, refresh-catalog, validate), wired into the REST API plugin;503 external_service_unavailablewhen the service is absent.os datasource list-tables | introspect | validate.engine.introspectDatasource(name)end-to-end introspection.P3 — Boot validation (Gate 2) + catalog type ·
@objectstack/runtime,@objectstack/specExternalValidationPlugin(exported): onkernel:ready, enforcesexternal.validation.onMismatch(fail/warn/ignore).external_catalogmetadata type +ExternalCatalogSchema.P4 — AI awareness ·
@objectstack/service-aiSchemaRetrieverbadges federated objects[external, read-only, datasource=…];query_dataalready clamps limits.P6 — Write gate (Gate 3) ·
@objectstack/objectqlassertWriteAllowed()blocks insert/update/delete on federated datasources unlessdatasource.external.allowWritesandobject.external.writable.Out of scope
../objectuirepo (not in this environment).ExternalDatasourceServicePlugin(service) +ExternalValidationPlugin(Gate 2) are exported for the host to.use(); not force-registered in runtime (consistent with other opt-in services). REST routes mount unconditionally and degrade gracefully; the CLI talks to them over HTTP.Verification (per-package, local)
spec federation 151, driver-sql DDL gate 8, objectql write-gate/introspect 11, service-external-datasource 15, runtime boot-gate 7, service-ai 3 — 195 tests, all green. Affected builds (incl. DTS) pass for spec, driver-sql, objectql, runtime, rest, cli, service-ai, service-external-datasource. Changeset fixed-group in sync (62 packages).
Known pre-existing CI failures (NOT from this PR)
Surfaced only because the new package changed the lockfile (path-filtered workflows that rarely run on
main); each reproduced on a cleanmaincheckout:objectql/protocol-meta.test.tsprovenance (_packageId/_provenance).`{<id>}`breaks the new Turbopack/fumadocs MDX parser.pnpm audithigh ontarvia thesqlite3peer dep.CodeQL findings in this PR's own code (ReDoS in
type-compat; network-data-to-file inos datasource introspect --out, which now sandboxes--outto the CWD) are addressed.Refs ADR-0015.
https://claude.ai/code/session_019FweBAH7Af5HzMWNQ84nPd