Skip to content

v0.26.0

Latest

Choose a tag to compare

@rich-iannone rich-iannone released this 27 Jul 23:25
· 32 commits to main since this release
af6dc15

Pointblank v0.26.0 is a major release focused on clinical data standards, data quality measurement, and AI-assisted validation editing. This release introduces comprehensive CDISC conformance validation with a built-in rule engine and bundled controlled terminology, a metadata import/export system supporting SDTM, ADaM, Frictionless, and CSVW formats, and structured missingness handling through a new MissingSpec type. Validation steps are now automatically tagged with data quality dimensions and rolled up into health scores, and the new EditValidation class lets you modify existing validation plans through interaction with an LLM.

New Features

  • Data quality dimensions and health scoring — Every validation step is now tagged with a data quality dimension (Completeness, Validity, Uniqueness, Consistency, Timeliness, or Volume) inferred automatically from the assertion type. Dimensions are overridable per step via dimension= or remappable globally via pb.config(dimension_map=...). New methods get_dimension_scores(), get_health_score(), get_scorecard(), and get_validation_summary() expose test-unit-weighted scores, with assert_dimension_scores() available to fail a run when a dimension falls below a threshold. The validation report shows color-coded dimension badges on each step number with a health-score summary in the footer. (#409)

  • CDISC submission conformance — A new conformance validation framework provides a rule engine, bundled SDTMIG 3.4 rules with SDTM controlled terminology, and public APIs for validating study submission packages. Includes ConformanceReport with JSON and Excel export, findings_df() and get_findings_table() for detailed inspection, and integration with the CDISC CORE engine. (#413)

  • Metadata standards — New metadata import, export, and validation workflows for clinical data standards. The import_metadata() function reads Frictionless Table Schema, CSVW, CDISC Define-XML, CDISC Controlled Terminology, and SPSS/SAS/Stata files into a unified MetadataImport object. Built-in SDTM and ADaM templates enable domain-level structure validation with validate_sdtm() and validate_adam(). Metadata can be exported back to Frictionless format and converted to Pointblank schemas. (#403)

  • Schema enforcement API — A contract import/export system with a modular adapter architecture. import_contract() and export_contract() translate between external schema formats (JSON Schema, Frictionless Table Schema) and Pointblank validations or contracts. Custom adapters can be registered via the @register_adapter decorator. (#402)

  • Structured missingness — A new MissingSpec type maps sentinel values to reasons, and a missing= parameter on col_vals_*() methods excludes coded values from checks. Four dedicated validation steps are added: col_pct_missing(), col_missing_coded(), col_missing_only_coded(), and col_missing_consistent(). (#404)

  • AI validation editor — The new EditValidation class lets you modify an existing validation plan from a natural-language instruction, with diff(), changed_steps(), preview(), and accept() methods for reviewing changes. Plan serialization via Validate.to_code() and Validate.to_yaml() is fully round-trippable. Also adds Validate.suggest_improvements(), Validate.from_prompt(), Validate.from_plans(), a syntax-lint guardrail with auto-reprompt, and a pb edit CLI command. (#408)

  • float_field() precision= parameter — Generated float values can now be rounded to a specified number of decimal places, useful for prices, scores, and other fields where full floating-point precision is not desired. (#412)

Bug Fixes

  • Per-row validation checks on Polars LazyFrames are no longer 8–11× slower than v0.24.0. The result-tallying code now computes row, pass, fail, and null counts in a single aggregation instead of issuing three to four separate .collect() calls that re-executed the entire lazy plan. (#407)

  • Segmented validation on Polars LazyFrames no longer raises an UnboundLocalError for seg_tuples(). Segment expressions now work consistently for both eager and lazy DataFrames. (#414)

Documentation

  • CDISC validation user guide examples are now fully executable against sample metadata documents bundled with the package via the new load_metadata_example() function. (#406)