Skip to content

Release v0.15.3

Choose a tag to compare

View all tags
@gueniai gueniai released this 16 Oct 21:08
· 6 commits to main since this release
v0.15.3
44af4f4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Release v0.15.3

Highlights

  • Choose to skip unredacted profiler steps. The new --excluded unredacted option on execute-database-profiler skips the steps that pull raw query text without redaction. Skipped steps are logged in the run metadata and the run still finishes.
  • Less sensitive data in Snowflake extracts. The Snowflake profiler now hashes user names and drops free-text columns and pricing details from its extract.
  • More reliable SQL Server profiling. The MSSQL profiler now handles databases with mixed or case-sensitive collations, older SQL Server versions, and passwords that contain special characters.

Profilers

Enhancements

  • Step labels and a new --excluded option (#2679)
    Profiler pipeline steps can now carry a label. Labels come from a fixed set, and unknown values are rejected when the config loads. Running databricks labs lakebridge execute-database-profiler --excluded unredacted skips every step labelled unredacted. Those steps are recorded as SKIPPED in profiler_run_metadata, and the run still completes. The Snowflake query_samples and Oracle perf_sqltext steps are labelled unredacted.

Fixes

  • Fixed DuckDB DECIMAL overflow across all profilers (#2657)
    Source queries that return DECIMAL/NUMERIC/NUMBER values could fail when loaded into DuckDB, because DuckDB picks a narrow DECIMAL type from a sample of rows and then overflows on larger values. The affected Synapse, Oracle and Teradata queries now cast those values to a floating-point type in the query itself.

SQL Server

  • Hardened queries for collations and older versions (#2675)
    Queries that span several databases now work when those databases use different collations (adds COLLATE DATABASE_DEFAULT). sys_info extraction now falls back gracefully on older SQL Server versions that lack some sys.dm_os_sys_info columns. Metadata identifiers and aliases now use consistent casing, so the SQL Server and Synapse queries work in case-sensitive catalogs.
  • Fixed passwords with special characters (#2658)
    Passwords containing ODBC-special characters such as ? are now escaped properly when the connection string is built. Before, they caused a ConnectionStringParseError that blocked the connection test in configure-database-profiler.

Snowflake

  • Dropped potentially sensitive values from the extract (#2678)
    rate_sheet now keeps only the account tier. User names in query_history and query_samples are hashed. Free-text columns are removed from query_history and database_objects.
  • ~ is now expanded in private_key_path (#2674)
    A key path like ~/snowflake.pem is now resolved against your home directory, like every other path Lakebridge reads. Before, it failed with an unclear read/MalformedFraming error.

New Contributors

Full Changelog: v0.15.2...v0.15.3

Contributors

surajs-db
Assets 6
Loading