Skip to content

Release v0.14.0

Choose a tag to compare

View all tags
@gueniai gueniai released this 12 Jun 22:14
· 62 commits to main since this release
v0.14.0
e969dff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release Notes — Lakebridge v0.14.0

Highlights

  • Five new profiler sources. Snowflake, Redshift, BigQuery, Oracle, and Legacy SQL DW are now all supported profiler targets, significantly expanding the set of platforms Lakebridge can assess ahead of a migration.

  • Switch now supports SAS. A new built-in prompt converts SAS programs to PySpark, adding SAS to the growing list of languages Switch can migrate automatically.

  • Reconcile now supports Teradata. Teradata can now be used as a source platform for data quality reconciliation, enabling validation for customers migrating from Teradata to Databricks.

  • Automatic reconciliation configuration. A new auto-configure-recon-tables command discovers source and target tables and generates the initial reconcile configuration automatically, replacing a previously manual setup step.

Profilers

New Sources

  • Redshift (#2305, #2304, #2306, #2408, #2501)
    Amazon Redshift is now a fully supported profiler source, covering all three deployment variants: provisioned, provisioned multi-AZ, and serverless. This includes credential flows (database password, federated user, AWS Secrets Manager ARN, temporary credentials with db user or IAM, with optional SSL), dedicated extraction queries and validation schemas for each variant, full CLI wiring, and user documentation. A bug in the serverless managed-storage aggregation was also fixed: the previous query summed hourly sys_serverless_usage snapshots, causing reported storage to grow linearly with the lookback window rather than reflecting the actual allocated amount.

  • Snowflake (#2420, #2499)
    Adds Snowflake as a profiler source. The interactive configurator prompts for connection details and a Programmatic Access Token (PAT), then extracts warehouse usage, query history, storage, user activity, account info, and optional credits (pipe, autoclustering, materialized-view refresh) from SNOWFLAKE.ACCOUNT_USAGE into a timestamped DuckDB file. Post-extraction computations produce TCO summaries. A supplementary rate_sheet extract pulls the effective per-credit rate (90-day average) and account service tier from SNOWFLAKE.ORGANIZATION_USAGE.RATE_SHEET_DAILY, providing the inputs needed by the downstream TCO value model.

  • BigQuery (#2472)
    Adds BigQuery as a profiler source. Running configure-database-profiler followed by execute-database-profiler --source-tech bigquery executes 16 region-qualified INFORMATION_SCHEMA queries against the customer's configured BigQuery project(s) and writes 12 analysis tables into a local DuckDB file at ~/.databricks/labs/lakebridge_profilers/bigquery_assessment/profiler_extract.db. The pipeline structure mirrors existing source-techs.

  • Oracle (#2187)
    Adds Oracle as a profiler source. Covers the interactive configuration dialog, extraction SQL scripts, and local DuckDB population from Oracle Database, with unit tests for all three components.

  • Legacy SQL DW (#2441)
    Extends profiler coverage to legacy Azure SQL DW (pre-Synapse) deployments, adding the necessary extraction queries to assess this platform.

Enhancements

  • SQL Server profiler switched to SQL scripts and shared DatabaseManager (#2482)
    The MSSQL profiler's activity and info extraction steps have been converted from per-step Python virtualenvs to in-process sql/ddl steps using the shared DatabaseManager connector. Thirteen SQL query/DDL pairs replace the previous Python scripts, eliminating the duplicated get_sqlserver_reader connector and aligning SQL Server with the architecture used by other profiler sources.

  • User-configurable output folder (#2488)
    The profiler's DuckDB extract path is now exposed as a CLI flag (--output-folder) instead of being hardcoded per source in pipeline_config.yml. The default remains ~/.databricks/labs/lakebridge_profilers/<source>_assessment. Output filenames now include a timestamp (profiler_extract_<YYYYMMDD_HHMMSS>.db) to prevent overwrites on repeated runs, and the absolute path to the extract is logged on successful completion.

  • Custom credentials file path for execute-database-profiler (#2494)
    The execute-database-profiler command now accepts an optional --cred-file-path argument, letting users supply a non-default credentials file rather than the one written by configure-database-profiler. This makes it easier to manage multiple credential configurations or run the profiler in scripted, non-interactive environments.

  • SQL Server: TrustServerCertificate support (#2498)
    The configure-database-profiler command for SQL Server now surfaces a TrustServerCertificate connection property, addressing a common customer request for environments where the server certificate cannot be validated.

  • Fix: profiler steps no longer fail with ModuleNotFoundError (#2485)
    The Synapse and SQL Server profilers previously created a fresh virtualenv per pipeline step, installing only each step's declared dependencies. After database_manager.py began importing redshift_connector at module scope, every clean run of those profilers failed with ModuleNotFoundError. Profiler Python steps now run with the parent interpreter, inheriting all installed packages—faster, more robust, and immune to this class of transitive-import failures.

Converters

Morpheus

T-SQL Improvements

  • Better date and number formatting from T-SQL CONVERT
    T-SQL's CONVERT function accepts a style code to format dates and numbers as strings (e.g. CONVERT(VARCHAR, myDate, 103) for dd/MM/yyyy). Morpheus now correctly translates these style codes to the equivalent Databricks SQL expressions, unblocking a large class of previously untranslatable queries.

  • Fix integer-as-date behavior from T-SQL
    T-SQL allows using 0 (or any integer) where a date is expected, treating it as "N days after 1 Jan 1900". Morpheus now replicates this behavior, preventing runtime type errors when running translated queries on Databricks.

  • Fix variable declarations with VARCHAR(N) / CHAR(N) types
    T-SQL local variables declared as VARCHAR(N) or CHAR(N) were being passed through verbatim and failing at runtime in Databricks. They are now automatically translated to STRING, the correct equivalent type for variable declarations.

  • Fix T-SQL variable assignment queries with ORDER BY
    T-SQL queries that assign a value to a variable (e.g. SELECT @var = col FROM t ORDER BY col) were being incorrectly structured during translation. The ORDER BY and similar clauses are now placed correctly in the output.

  • Support HASH JOIN query hint in T-SQL
    T-SQL's OPTION(HASH JOIN) query hint, which instructs the database to use a specific join strategy, is now correctly parsed and handled during translation.

Snowflake Improvements

  • Translate REGEXP_SUBSTR_ALL to REGEXP_EXTRACT_ALL
    Snowflake's REGEXP_SUBSTR_ALL (returns all regex matches as an array) is now translated to its Databricks SQL equivalent REGEXP_EXTRACT_ALL.

  • Translate binary hash functions (MD5_BINARY, SHA1_BINARY, SHA2_BINARY)
    Snowflake's binary digest functions are now translated to their Databricks SQL equivalents by wrapping the hex output with UNHEX().

  • Translate hex hash function synonyms (MD5_HEX, SHA1_HEX, SHA2_HEX)
    Snowflake's *_HEX hash function aliases are now directly mapped to their identically-behaved Databricks SQL counterparts.

  • Translate UNICODE and TRY_TO_DOUBLE
    Snowflake's UNICODE() (returns the code point of the first character) is now mapped to ASCII() in Databricks SQL, and TRY_TO_DOUBLE() is mapped to TRY_CAST(_ AS DOUBLE). The UNICODE fix also applies to T-SQL.

  • Translate type-check functions (IS_DATE, IS_DOUBLE, IS_REAL, etc.)
    Snowflake functions that test whether a value inside a semi-structured (VARIANT) column holds a specific type are now translated where possible (e.g. IS_DATE → TRY_CAST(v AS DATE) IS NOT NULL). Those with no equivalent in Databricks (IS_TIME, IS_TIMESTAMP_TZ) are flagged with a clear migration note.

  • Flag unsupported functions (CHECK_XML, PARSE_XML, IS_ROLE_IN_SESSION, etc.) with migration notes
    Seven Snowflake-specific functions with no Databricks equivalent now produce a clear "FIXME" comment in the output instead of silently passing through and failing at runtime. IS_NULL_VALUE is also correctly translated to IS_VARIANT_NULL.

  • Flag 12 more Snowflake-only functions with migration notes
    Additional Snowflake admin, statistical, and VARIANT-inspection functions (including COMPRESS, NORMAL, ZIPF, INVOKER_ROLE, IS_BOOLEAN) now produce clear FIXME annotations instead of failing silently at runtime.

  • Flag Snowflake INFORMATION_SCHEMA metadata functions with migration notes
    Snowflake monitoring/metadata functions called via INFORMATION_SCHEMA (like PIPE_USAGE_HISTORY, MATERIALIZED_VIEW_REFRESH_HISTORY) no longer cause an UNRESOLVED_ROUTINE crash — they are now flagged with a clear migration note explaining that no equivalent exists.

  • Translate Snowflake's row generator pattern to RANGE()
    Snowflake's TABLE(GENERATOR(ROWCOUNT => N)) pattern (used to generate N rows, often with SEQ4() to get row numbers) is now automatically translated to Databricks SQL's RANGE(0, N).

  • Improve parsing of COPY INTO commands
    The parser now correctly handles both forms of Snowflake's COPY INTO — loading data into a table and unloading data to an external location — laying groundwork for future full translation support.

Cross-dialect Improvements

  • Support for cursor-based SQL across all dialects
    Cursor statements (DECLARE, OPEN, FETCH, CLOSE, DEALLOCATE) are now supported for migration across T-SQL, Snowflake, and Redshift.

  • Clearer error messages for untranslatable statements
    When a SQL statement can't be automatically translated, the output now includes the original SQL text and a meaningful explanation in the FIXME comment, rather than a generic confusing error message.

Switch

New Source Format Support

  • SAS code conversion
    New built-in prompt converts SAS programs (both inline DATALINES and external file patterns) to PySpark equivalents, with example input/output pairs included.

  • Informatica ETL conversion
    New built-in prompt for migrating Informatica workflows to Lakeflow Spark Declarative Pipelines (SDP). Covers Source Qualifier, Expression, Router, Joiner, Lookup (connected and unconnected), Aggregator, Normalizer, Sequence Generator, and Update Strategy transformations to PySpark/Spark SQL.

  • Custom ETL conversion
    A flexible companion template lets users define their own input/output specs and conversion logic for any ETL tool not covered by a dedicated prompt.

New Reference Prompts

  • Teradata stored procedures reference prompt
    Introduces a new category of reference prompts — field-tested, opt-in prompts users can point Switch at via conversion_prompt_yaml. This first entry converts Teradata stored procedures directly to Databricks SQL (no Python-notebook wrapping) and covers: CREATE VOLATILE TABLE + multiple INSERT → CREATE OR REPLACE TEMPORARY VIEW ... UNION ALL; UPDATE...FROM → MERGE INTO; multi-column IN deletes → EXISTS; epoch/timezone conversions; and Teradata-specific functions (SYSLIB.OREPLACE, DELIMITEDCOUNT, DELIMITEDITEM, etc.).

Improved Built-in Prompts

  • Snowflake
    Added conversion rules discovered during a customer dbt migration — IFF, TRY_TO_*, PARSE_JSON, SELECT * EXCLUDE, QUALIFY, WITH RECURSIVE, colon (:) JSON access, TIMESTAMP_NTZ, trailing comma removal, and a new date-format pattern mapping section (e.g. HH24 → HH, MI → mm).

  • Teradata
    Major expansion with rules found during BTEQ customer migrations. New coverage includes: BTEQ control commands (.LOGON, .QUIT, FastLoad, MultiLoad, FastExport statements); shell variable substitution via dbutils.widgets and Python f-strings; (FORMAT)(CHAR) shorthand cast patterns; REPLACE VIEW with column list, RENAME TABLE, COLLECT STATISTICS; UPDATE...FROM → MERGE INTO; extended DDL clause removal (NO FALLBACK, COMPRESS, CASESPECIFIC, PARTITION BY RANGE_N, etc.); and two new few-shot examples for BTEQ scripts with dynamic identifiers.

  • Redshift
    Substantial update validated against ~30,000 real queries, producing more accurate and performant Databricks SQL output.

Bug Fixes

  • Serverless export failures
    Fixed a bug where exporting files or notebooks to subdirectories would fail on serverless compute with ResourceDoesNotExist: The parent folder does not exist. The root cause was os.makedirs only writing to the FUSE layer without materializing a real Workspace object. Replaced with ws_client.workspace.mkdirs in export_to_file.py and export_to_notebook.py, matching the pattern already used elsewhere in the codebase.

Enhancements

  • #2256 — Custom Switch configuration path for llm-transpile
    The llm-transpile CLI command now accepts an optional --switch-config-path parameter, allowing users to point Switch at a custom configuration file stored in their Databricks workspace (the path must start with /Workspace/). When omitted, Switch uses its default configuration as before.

Reconcile

  • #2456 — Teradata support
    Reconciliation now supports Teradata as a source platform, enabling data quality validation for customers migrating from Teradata to Databricks.

  • #2465 — Unified sampling query across all dialects
    The reconcile sampling query is now implemented as a single derived-table-join shape that works across all supported dialects. The previous implementation used a CTE for non-T-SQL paths and a VALUES derived table for T-SQL; the latter failed at runtime on Azure Synapse Dedicated SQL Pool, which does not accept VALUES as a derived table.

  • #2392 — Auto-discovery of source/target tables
    A new CLI command, databricks labs lakebridge auto-configure-recon-tables, discovers tables in source and target systems and automatically generates column mappings for the reconcile configuration. Users are guided through two interactive stages: table discovery, then auto-configuration of the discovered tables.

  • #2504 — Source system logging in recon commands
    Reconciliation commands now include the source system and report type in user-agent extras, improving observability and making it easier to trace usage across different source platforms.

Documentation

  • #2457 — Document PyPI and Maven mirrors during installation
    The installation documentation now covers how to configure package mirrors and proxies when direct access to GitHub, PyPI, or Maven Central is unavailable — a common requirement in air-gapped or enterprise firewall environments.

General

  • Python 3.14 support (#2333)
    The CLI and its dependencies now support Python 3.14. This required updating mypy, pylint, and pytest to versions compatible with the new release, along with minor code changes to satisfy the updated tooling. Test coverage now runs across all supported Python versions. Downstream dependencies (Blueprint, Bladebridge) had already released Python 3.14 support.

  • Remove explicit cryptography dependency (#2458)
    The explicit dependency on the cryptography package has been dropped. It was previously used directly for Snowflake connection setup but has not been needed since that code was revised. The package remains available as a transitive dependency.

New Contributors

Full Changelog: v0.13.0...v0.14.0

Contributors

ysmx-github, take60, and 3 other contributors
Assets 6
Loading