Skip to content

feat: add Delta Lake connector + docs + test#8916

Merged
royendo merged 14 commits intomainfrom
royendo/add-delta-connector
Mar 13, 2026
Merged

feat: add Delta Lake connector + docs + test#8916
royendo merged 14 commits intomainfrom
royendo/add-delta-connector

Conversation

@royendo
Copy link
Copy Markdown
Contributor

@royendo royendo commented Feb 23, 2026

Adds a Delta Lake connector that lets users query Delta tables stored on S3, Azure Blob Storage, or local filesystem, powered by DuckDB's delta extension.

  • Backend: installs and loads the delta DuckDB extension at startup; adds project schema validation for Delta sources
  • Frontend: multi-step form schema with storage type selector (S3 / Azure / local), conditional field visibility via x-grouped-fields, and disabled-option enforcement when cloud connectors are missing
  • Source rewriting: maybeRewriteToDuckDb() normalizes Delta form data into a DuckDB model with delta_scan() SQL, bridging cloud credentials via create_secrets_from_connectors
  • Icons: Delta Lake logo and icon Svelte components
  • Docs: new Delta Lake connector page with storage backend matrix, UI walkthrough, and manual YAML configuration examples
Screenshot 2026-03-02 at 16 06 33

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

Developed in collaboration with Claude Code

@royendo royendo changed the title draft: feat/add Delta Lake connector draft: feat: add Delta Lake connector Feb 23, 2026
@royendo royendo changed the title draft: feat: add Delta Lake connector feat: add Delta Lake connector Mar 2, 2026
@royendo royendo changed the title feat: add Delta Lake connector feat: add Delta Lake connector - Mar 2, 2026
@royendo royendo changed the title feat: add Delta Lake connector - feat: add Delta Lake connector + docs + test Mar 2, 2026
@royendo royendo requested a review from ericpgreen2 March 2, 2026 20:59
@royendo royendo marked this pull request as ready for review March 2, 2026 22:08
@ericpgreen2 ericpgreen2 requested review from AdityaHegde and removed request for ericpgreen2 March 9, 2026 09:59
royendo and others added 5 commits March 12, 2026 10:48
Resolve conflicts with iceberg connector (PR #8902) that merged to main.
Keep both delta and iceberg connectors side by side.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@royendo
Copy link
Copy Markdown
Contributor Author

royendo commented Mar 12, 2026

@AdityaHegde , pulled changes from iceberg PR :)

@royendo royendo merged commit 3db8f51 into main Mar 13, 2026
13 checks passed
@royendo royendo deleted the royendo/add-delta-connector branch March 13, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants