Skip to content

Releases: Se7enquick/eliza-dq

v0.2.1

Choose a tag to compare

@Se7enquick Se7enquick released this 09 Sep 19:19

Fixes

  • MySQL/ClickHouse CAST: CAST(AS VARCHAR) → CAST(AS CHAR) for MySQL, CAST(AS STRING) for BigQuery/Databricks, CAST(AS String) for ClickHouse. Affected checks: not_missing, min_length, max_length.
  • Regex backslash on MySQL/ClickHouse: \d, \w, \. patterns were silently stripped by string parser. Now doubled for dialects that consume backslashes.
  • reference check in SQL runner: result was silently dropped (missing parser). Now returns orphan_count correctly.
  • agg_row crash: defensive handling when aggregation query returns empty or unexpected result.

CI

Integration tests against 4 real engines on every push:

Engine Tests How
PostgreSQL 18 Service container
MySQL 16 Service container
DuckDB 17 In-process
Polars 19 DataFrame/parquet/CSV

All tests use the same 10-row dataset with exact fail count assertions.

workflow_dispatch dropdown to run one specific dialect when fixing a bug.

v0.2.0

Choose a tag to compare

@Se7enquick Se7enquick released this 09 Sep 18:37

New

  • [Feature] Lazy Polars import -- SQL path cold start 20ms instead of 176ms (Polars loads only when DataFrames used)
  • [Feature] Default sample queries select only checked columns instead of SELECT * -- cheaper on BQ/Athena with no configuration
  • [Feature] sample_id and sample_columns YAML options for full control over sample query cost
  • [Feature] Benchmark script supports 259M rows via --full

Fixes

  • [Fix] SQL errors report status=error, exit_code=2 (was silently passing)
  • [Fix] Regex backslash preservation in all 8 SQL dialects
  • [Fix] Connector dialect correctly routed to SQL generation
  • [Fix] HAVING COUNT(*) > 1 for Postgres/Redshift/Snowflake unique check
  • [Fix] COUNT(CASE...THEN 1 END) aggregation (26% faster than SUM)

Verified benchmarks

  • Athena: 2x faster than Soda Core (179M-492M rows, parallel samples + 20ms cold start)
  • BigQuery: similar speed to Soda (slot scheduling dominates), up to 92% cheaper with sample_id on wide tables
  • DataFrames: 16x faster than Pandera on 126M rows (both check everything)
  • Streaming: 259M rows from disk in 1.1s, under 1 GB RAM

pip install eliza-dq --upgrade

v0.1.3

Choose a tag to compare

@Se7enquick Se7enquick released this 09 Sep 17:13
  • [Feature] sample_id and sample_columns for cheaper DWH sample queries (up to 92% cost reduction on wide BQ tables)
  • [Feature] Parallel sample collection via ThreadPoolExecutor
  • [Fix] SQL errors now report status=error, exit_code=2
  • [Fix] Regex backslash preservation in all SQL dialects
  • [Fix] Connector dialect correctly routed to SQL generation
  • [Fix] HAVING COUNT(*) > 1 for Postgres/Redshift/Snowflake unique check
  • [Fix] COUNT(CASE...THEN 1 END) optimized aggregation (26% faster)
  • [Docs] Verified benchmarks on 10 fresh tables (BQ + Athena, cache disabled)

pip install eliza-dq --upgrade