Skip to content

docs(design): RFC — multi-platform extensions (Fabric + Databricks)#328

Draft
padak wants to merge 1 commit into
mainfrom
design/multi-platform-extensions
Draft

docs(design): RFC — multi-platform extensions (Fabric + Databricks)#328
padak wants to merge 1 commit into
mainfrom
design/multi-platform-extensions

Conversation

@padak
Copy link
Copy Markdown
Member

@padak padak commented May 22, 2026

What

A design RFC (doc-only, no code) for making kbagent a single interface to Keboola + Microsoft Fabric + Databricks, with the non-Keboola platforms shipping as toggleable, read-first extensions exposed through the existing kbagent serve HTTP API.

File: docs/design/multi-platform-extensions.md.

Why

Two named use cases drive it: a cross-platform pipeline view and a cross-platform failed-job view ("all errors in one place"), scriptable through one HTTP API (e.g. kbagent http get "/runs?status=error").

What's inside

  • 5-Whys → root need = a unified, normalized, read-first operational view (not write parity).
  • Six candidate approaches with KISS/YAGNI verdicts.
  • Devil's-advocate review (leaky-abstraction trap, auth as the real cost, serve-router explosion, lineage-stitching trap, product identity/naming).
  • A concrete design grounded in the current architecture: the 3-layer design already is an adapter pattern; a connections discriminated union; a PlatformProvider Protocol; a parallel aggregator; one normalized serve router set; pip extras + lazy imports; OpenLineage as the vocabulary.
  • A phased plan: Phase 0 builds the platform-agnostic seam over Keboola-only to de-risk the abstraction; Phase 1 Databricks; Phase 2 Fabric; Phase 3 opt-in (writes / MCP federation / lineage).

Recommendation (accepted)

  • Approach: in-process PlatformProvider adapter + kind discriminator + pip extras.
  • v1 scope: read-only.
  • First external platform: Databricks (official SDK → lower risk; validates the normalized envelope before paying Fabric's Entra-OAuth cost).

Next step

Review the RFC. On approval, implement Phase 0 (the platform-agnostic seam, Keboola-only) before any vendor code.

Proposal for adding Microsoft Fabric and Databricks to kbagent as toggleable, read-first extensions exposed through the existing 'serve' HTTP API. Recommends an in-process PlatformProvider adapter selected by a connection 'kind' discriminator, with optional pip extras and lazy imports, and a unified normalized model (connections/pipelines/runs) aligned to the OpenLineage vocabulary.

Includes a 5-Whys analysis, six candidate approaches with KISS/YAGNI verdicts, a devil's-advocate review, a concrete code-level design grounded in the current architecture, and a phased plan. Accepted direction: read-only v1, Databricks first.
@padak padak marked this pull request as draft May 22, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant