v0.32.0

[cosmicBboy]

⭐️ Highlights: Add support for narwhals backend

In this release, we're adding support for a narwhals backend that supports the polars, ibis and pyspark-sql schema APIs. To enable it, install the narwhals extra with the support framework of your choice:

pip install 'pandera[narwhals,polars]'   # Polars
pip install 'pandera[narwhals,ibis]'     # Ibis
pip install 'pandera[narwhals,pyspark]'  # PySpark SQL

Then enable with an environment variable:

export PANDERA_USE_NARWHALS_BACKEND=True

Or configure programmatically:

import pandera.polars as pa 
# import pandera.pyspark as pa  # 👈 for pyspark schemas
# import pandera.ibis as pa  # 👈 for ibis schemas

# CONFIG.use_narwhals_backend is read here — not at import time above
pa.config.set_config(use_narwhals_backend=True)

schema = pa.DataFrameSchema({"name": pa.Column(str)})  # narwhals backends registered
schema.validate(df)

See the docs for more information.

What's Changed

• fix: Ibis validation fails checks if column all nulls by @cosmicBboy in fix: Ibis validation fails checks if column all nulls #2296
• fix: StringDtype validation fails for empty DataFrames by @cosmicBboy in fix: StringDtype validation fails for empty DataFrames #2295
• Clean up duplicate code and extra files from fix: Ibis validation fails checks if column all nulls #2296 by @deepyaman in Clean up duplicate code and extra files from #2296 #2298
• Pyspark: Added unique_values_eq builtin check by @aditypan in Pyspark: Added unique_values_eq builtin check  #2300
• feat(narwhals): implement the new unifying backend by @deepyaman in feat(narwhals): implement the new unifying backend #2223
• refactor(narwhals): make backend opt-in via config by @deepyaman in refactor(narwhals): make backend opt-in via config #2314
• refactor(narwhals): add per-backend test structure by @deepyaman in refactor(narwhals): add per-backend test structure #2315
• [Narwhals] add docs, some code cleanup by @cosmicBboy in [Narwhals] add docs, some code cleanup #2312
• fix(polars): handle PydanticModel dtype in DataFrameModel.empty() by @Dev-iL in fix(polars): handle PydanticModel dtype in DataFrameModel.empty() #2313
• fix: ensure index fields have persistent metadata by @WPDOrdina in fix: ensure index fields have persistent metadata #2217
• refactor(polars): preserve input frame kind in coerce_dtype by @Dev-iL in refactor(polars): preserve input frame kind in coerce_dtype #2317
• chore: delete extraneous isort hook for pre-commit by @deepyaman in chore: delete extraneous isort hook for pre-commit #2320
• test(ibis): enable decimal datatype test after fix by @deepyaman in test(ibis): enable decimal datatype test after fix #2321
• fix(tests): drop deprecated axis=None from DataFrame.sum call (Fix FutureWarning in test_strategies.py: DataFrame.sum(axis=None) is deprecated #2309) by @terryzm1 in fix(tests): drop deprecated axis=None from DataFrame.sum call (#2309) #2311
• docs: clarify required extra for data synthesis strategies by @RedZapdos123 in docs: clarify required extra for data synthesis strategies #2304
• chore: add Time as a cross-engine semantic dtype by @deepyaman in chore: add Time as a cross-engine semantic dtype #2323
• refactor: consolidate backend-specific test suites by @deepyaman in refactor: consolidate backend-specific test suites #2318
• Bugfix/2319 by @Concrete-Slab in Bugfix/2319 #2324
• feat(strategies): aggregate check constraints to skip filter chains by @cosmicBboy in feat(strategies): aggregate check constraints to skip filter chains #2322
• Improvement/ enable raising schema error in custom check by @Jarek-Rolski in Improvement/ enable raising schema error in custom check #2340
• fix(polars): stringify nested dtypes in lazy failure-case formatter by @lukew-cogapp in fix(polars): stringify nested dtypes in lazy failure-case formatter #2341
• docs: clarify optional missing columns by @remsky in docs: clarify optional missing columns #2338
• fix: handle SchemaErrors in check_types Union fallback (check_types decorator does not handle SchemaErrors (plural) #2325) by @SAY-5 in fix: handle SchemaErrors in check_types Union fallback (#2325) #2326
• Fix mypy type hinting for Polars DataFrameModel field annotations by @RedZapdos123 in Fix mypy type hinting for Polars DataFrameModel field annotations #2303
• Fixed issue where Pandera cannot handle metadata through Annotated types by @NGHades in Fixed issue where Pandera cannot handle metadata through Annotated types #2111
• fix narwhals incompatibility by @cosmicBboy in fix narwhals incompatibility #2342
• refactor: pydantic/polars cleanup and benchmark coverage by @Dev-iL in refactor: pydantic/polars cleanup and benchmark coverage #2357
• Clarify ignore_na null handling in checks docs by @vanta722 in Clarify ignore_na null handling in checks docs #2355
• fix(polars): avoid coercing missing required columns by @remsky in fix(polars): avoid coercing missing required columns #2346
• feat(pyspark): register Narwhals backend for Spark by @deepyaman in feat(pyspark): register Narwhals backend for Spark #2339
• Address comments from full Narwhals backend review by @deepyaman in Address comments from full Narwhals backend review #2360
• docs(pydantic): clarify v2 string coercion semantics by @RedZapdos123 in docs(pydantic): clarify v2 string coercion semantics #2348
• update narwhals docs by @cosmicBboy in update narwhals docs #2365
• update config, add set_config by @cosmicBboy in update config, add set_config #2366
• Harden CI dependency installation by @RedZapdos123 in Harden CI dependency installation #2369
• fix(extensions): reject extra positional args in register_check_method (Registered checks with no statistics accept positional arguments #480) by @jbbqqf in fix(extensions): reject extra positional args in register_check_method (#480) #2347
• fix: handle mixed-timezone pandas datetime coercion by @RedZapdos123 in fix: handle mixed-timezone pandas datetime coercion #2350
• fix(pandas): keep ArrowString resolution stable by @RedZapdos123 in fix(pandas): keep ArrowString resolution stable #2353
• feat: allow rename_columns to work as long as final columns are unique by @AchillesTurtle in feat: allow rename_columns to work as long as final columns are unique #2373
• fix(polars): don't error on optional regex column with no match by @gaoflow in fix(polars): don't error on optional regex column with no match #2374
• fix: reload built-in check dispatcher by its own name so a custom name= works (Checks with kwarg name specified raise a KeyError on exceution. #2042) by @vineethsaivs in fix: reload built-in check dispatcher by its own name so a custom name= works (#2042) #2371
• fix(polars): report concrete column name in regex coercion errors (Unclear which regex-matched column error occurs in with Polars with data coercion #2363) by @gaoflow in fix(polars): report concrete column name in regex coercion errors (#2363) #2376
• feat: enable runtime narwhals backend toggling via set_config by @cosmicBboy in feat: enable runtime narwhals backend toggling via set_config #2375
• fix(pandas): preserve drop_invalid_rows in schema IO by @RedZapdos123 in fix(pandas): preserve drop_invalid_rows in schema IO #2354
• docs: note regex engine differences between pandas and polars backends (Add caveat note on differences in regex pattern rules for polars versus pandas #2349) by @cycsmail in docs: note regex engine differences between pandas and polars backends (#2349) #2378

New Contributors

• @WPDOrdina made their first contribution in fix: ensure index fields have persistent metadata #2217
• @Concrete-Slab made their first contribution in Bugfix/2319 #2324
• @lukew-cogapp made their first contribution in fix(polars): stringify nested dtypes in lazy failure-case formatter #2341
• @remsky made their first contribution in docs: clarify optional missing columns #2338
• @SAY-5 made their first contribution in fix: handle SchemaErrors in check_types Union fallback (#2325) #2326
• @NGHades made their first contribution in Fixed issue where Pandera cannot handle metadata through Annotated types #2111
• @vanta722 made their first contribution in Clarify ignore_na null handling in checks docs #2355
• @jbbqqf made their first contribution in fix(extensions): reject extra positional args in register_check_method (#480) #2347
• @AchillesTurtle made their first contribution in feat: allow rename_columns to work as long as final columns are unique #2373
• @gaoflow made their first contribution in fix(polars): don't error on optional regex column with no match #2374
• @vineethsaivs made their first contribution in fix: reload built-in check dispatcher by its own name so a custom name= works (#2042) #2371
• @cycsmail made their first contribution in docs: note regex engine differences between pandas and polars backends (#2349) #2378

Full Changelog: v0.31.1...v0.32.0

---

This discussion was created from the release v0.32.0.

[Dev-iL]

This one is also included (as part of one of my PRs):

• Fix PydanticModel polars coercion dropping dtype info and mis-inferring all-null columns as Null #2356