feat(examples): Transpile virtual dataset SQL on import #37311
Conversation
When importing example datasets, automatically transpile virtual dataset SQL to the target database dialect using SQLGlot. This ensures that virtual datasets exported from one database type (e.g., PostgreSQL) can be loaded into a different database type (e.g., MySQL, DuckDB, SQLite). - Add transpile_virtual_dataset_sql() helper function - Integrate transpilation into ImportExamplesCommand._import() - Fall back gracefully to original SQL if transpilation fails - Add comprehensive unit tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
CodeAnt AI is reviewing your PR. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
dosubot
bot
added
the
data:connect
Sequence DiagramWhen importing example datasets, the importer automatically transpiles a dataset's SQL to the target database dialect (via SQLGlot) before creating the dataset, ensuring cross-dialect compatibility. The diagram shows the main success path added in this PR. sequenceDiagram
participant Importer as ImportExamplesCommand
participant DB as Database (ORM)
participant Transpiler as transpile_to_dialect (SQLGlot)
participant DatasetService as import_dataset
Importer->>Importer: Iterate dataset config (has SQL)
Importer->>DB: Query Database by database_id
DB-->>Importer: Database (with engine)
Importer->>Transpiler: transpile_to_dialect(sql, target_engine)
Transpiler-->>Importer: transpiled_sql
Importer->>DatasetService: import_dataset(config with transpiled SQL)
DatasetService-->>Importer: created dataset
Generated by CodeAnt AI |
Address review feedback: 1. transpile_to_dialect now accepts source_engine parameter to properly parse SQL from specific dialects (not just generic SQL) 2. Export stores source_db_engine in dataset YAML for virtual datasets 3. Import uses source_db_engine when transpiling to target dialect 4. Added comprehensive tests for PostgreSQL -> MySQL/DuckDB/ClickHouse/SQLite Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
✅ Deploy Preview for superset-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![bito-code-review[bot]](https://avatars.githubusercontent.com/in/1061978?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review Agent Run #aebbb4
Actionable Suggestions - 1
-
tests/unit_tests/sql/transpile_to_dialect_test.py - 1
- Parametrize decorator expects tuple not string · Line 351-352
Review Details
-
Files reviewed - 6 · Commit Range:
119b45f..0cd1b48- superset/commands/dashboard/export_example.py
- superset/commands/importers/v1/examples.py
- superset/datasets/schemas.py
- superset/sql/parse.py
- tests/unit_tests/commands/importers/v1/examples_test.py
- tests/unit_tests/sql/transpile_to_dialect_test.py
-
Files skipped - 0
-
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful
Bito Usage Guide
Commands
Type the following command in the pull request comment and save the comment.
-
/review- Manually triggers a full AI review. -
/pause- Pauses automatic reviews on this pull request. -
/resume- Resumes automatic reviews. -
/resolve- Marks all Bito-posted review comments as resolved. -
/abort- Cancels all in-progress reviews.
Refer to the documentation for additional commands.
Configuration
This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.
Documentation & Help
AI Code Review powered by
Co-authored-by: Beto Dealmeida <roberto@dealmeida.net>
|
Bot and human suggestions applied :D |
Co-authored-by: bito-code-review[bot] <188872107+bito-code-review[bot]@users.noreply.github.com>
Code Review Agent Run #9274c1Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
2 participants
SUMMARY
When importing example datasets, this PR automatically transpiles virtual dataset SQL to the target database dialect using SQLGlot. This ensures that virtual datasets exported from one database type (e.g., PostgreSQL) can be loaded into a different database type (e.g., MySQL, DuckDB, SQLite).
Problem: Virtual datasets contain raw SQL that is dialect-specific. If someone exports a dashboard with virtual datasets from PostgreSQL (using syntax like
DATE_TRUNC,::casting, etc.) and someone else tries to load it with MySQL or SQLite, the SQL would fail.Solution: Leverage Superset's existing
transpile_to_dialect()function (which uses SQLGlot) to automatically convert the SQL to the target database's dialect during import.Changes:
transpile_virtual_dataset_sql()helper function insuperset/commands/importers/v1/examples.pyImportExamplesCommand._import()before dataset importThis builds on the work from #36538 which modernized example data loading with Parquet and YAML configs.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
N/A - Backend change only
TESTING INSTRUCTIONS
superset load-examplesUnit tests:
ADDITIONAL INFORMATION
🤖 Generated with Claude Code