v1.1.0

Pointblank v1.1.0 adds table comparison and makes validation results more correct across backends. The new pb.compare() function, the tbl_reconcile() validation step, and the pb compare CLI command show exactly how two tables differ: row by row, column by column, and in aggregate. tbl_match() now runs on the same engine, with full step reports. This release also fixes how missing values, Ibis tables, and expression errors are handled in many validation methods. Steps that can't be evaluated now report an evaluation error instead of a misleading failure.

New Features

• Table comparison with pb.compare() — Aligns two tables by key, by position, or as multisets (keys="*"). Each row is classified as match, changed, missing, extra, or duplicate key, and schema differences are reported too. You can set numeric and datetime tolerances, normalize strings, map renamed columns with column_map=, and compare a subset of columns. The result is a Comparison object with counts, row samples, a per-column summary, JSON export, and a Great Tables report in all 40 reporting languages. Inputs are scanned lazily, and partitions= limits memory use on very large tables. (#443)
• tbl_reconcile() validation step — Reconciles aggregates (sum, count, mean, min, max) between two tables, either by key group or as grand totals. (#443)
• pb compare CLI command — Compares two tables from the command line, with HTML or JSON output, --engine duckdb, and exit codes for CI use (0 match, 1 differences, 2 error). (#443)
• A richer tbl_match() — Now uses the comparison engine and accepts the same settings. It gains a full step report, failing-row extracts from both tables, sundering, a compact settings badge, and to_code()/to_yaml() support. Note: each aligned row now counts as one test unit (before, the whole step was one unit), so fractional thresholds now refer to the share of failing rows. Pass/fail verdicts are unchanged. To fail on any difference as before, use thresholds=1. (#443)

Enhancements

• Anthropic is no longer a required dependency. It's now installed only with the pointblank[generate] extra, so core installs are lighter. Thanks @timkpaine! (#437)
• col_vals_increasing() and col_vals_decreasing() gain an order_by= parameter, so they now work on database tables, Ibis, and PySpark. Each value is now compared with the last non-missing value before it. (#451)
• col_vals_not_in_set() gains an na_pass= parameter. Both set-membership methods now document how they handle missing values and NaN. (#450)
• to_code() and to_yaml() now include col_vals_in_table() steps (with a placeholder for the reference table). (#461)
• col_vals_in_table(ref_table=) now accepts file paths and connection strings. (#463)
• col_vals_in_table() now matches join key types across backends and handles numeric dtype mismatches in composite keys. (#431)
• A pre= function with a dfn parameter now always receives a Narwhals DataFrame, whether it's a lambda, a named function, or another callable. This now also works with col() selectors and segments=. (#458)

Bug Fixes

Evaluation errors instead of misleading failures

• Invalid expressions in col_vals_expr() and conjointly() are now reported as evaluation errors. (#448)
• Comparison steps on columns with incompatible types are now reported as evaluation errors. (#459)
• prompt() steps now record an evaluation error when an AI provider package is missing or every model request fails (e.g., an invalid API key). Before, every row was reported as failing. (#456)
• Step reports now explain evaluation errors, and failing rows are now extracted for Ibis tables. (#452)
• Tables are now converted to a common backend without changing their dtypes, and conversion failures are reported. (#453)

Missing values

• Missing values now fail col_vals_not_in_set() on every backend. Thanks @raashish1601! (#432)
• col_vals_eq() and col_vals_ne() now count missing values consistently when comparing against another column. Thanks @raashish1601! (#433, #449)
• Rows where a col_vals_expr() expression evaluates to null now count as failing. Thanks @raashish1601! (#435)

Backend support

• col_vals_increasing() and col_vals_decreasing() now support Polars LazyFrames. Thanks @raashish1601! (#434)
• tbl_match() now materializes Ibis memtables and other Ibis backends. Thanks @raashish1601! (#436)
• conjointly() expressions now evaluate on Ibis tables. Thanks @raashish1601! (#438)
• pb.load_dataset() no longer raises a permission error for DuckDB datasets on Windows. Thanks @matt-humphrey! (#245)

Reports

• Step reports can now be built for passing steps that have no extracts. Thanks @raashish1601! (#440)
• get_dataframe_report() now supports steps with several columns and serializes non-string values for DuckDB. Thanks @cristianchiriac! (#444, #445)
• DuckDB report values are now stored as JSON text, full composite keys are reported, and step_evaluated is filled in. (#454)
• col_summary_tbl() no longer errors on boolean columns, including ones with missing values. (#466)

Data generation

• generate_dataset() now respects unique=True for coherent presets. (#447)

CDISC conformance

• The bundled controlled terminology is now generated from the official NCI EVS file for CDISC CT 2024-09-27. Before, only 3 of 19 bundled codelists matched the official ones, so official values (e.g., NY = U, SEX = INTERSEX) were rejected. COUNTRY now covers all ISO 3166-1 alpha-3 codes, and RACE accepts MULTIPLE. (#460)
• The AEACN, FREQ, and EPOCH codelists have been corrected, and ten rules that contradicted SDTMIG 3.4 have been removed (along with the AEREL and AETOXGR codelist rules). (#455, #457, #460)
• SDTMIG 3.4 presence and not-null rules now follow the IG's core designations. (#464)
• Define-XML extensions of extensible CDISC codelists are now accepted in conformance checks. (#462)

Documentation

• The lambda pre= example uses a lambda function again. Thanks @dpprdan! (#330)