Skip to content

Conversation

@borchero
Copy link
Member

@borchero borchero commented Oct 20, 2025

Motivation

Fixes #93.

Changes

  • Introduce a custom polars plugin to allow for lazily raising on validation errors
  • Add an eager parameter to {Schema,Collection}.{validate,filter} to allow for lazy validation
    • The default remains True as validation should, by default, serve as a barrier
  • Fix the PyPI build to actually perform a release build (lol)

NOTE: The eager validation/filter performance is equivalent (and tends to be slightly faster than before). There are, however, significant performance penalties for lazy validation/filtering in collections as well as a notable performance drop for lazy filtering of schemas. This is likely largely due to pola-rs/polars#24129.

@github-actions github-actions bot added the enhancement New feature or request label Oct 20, 2025
@codecov
Copy link

codecov bot commented Oct 20, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (12ec352) to head (b42b747).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #179   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           50        53    +3     
  Lines         2915      2996   +81     
=========================================
+ Hits          2915      2996   +81     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@borchero borchero self-assigned this Oct 20, 2025
@borchero borchero marked this pull request as ready for review October 20, 2025 21:26
Copy link
Collaborator

@AndreasAlbertQC AndreasAlbertQC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @borchero ! I only really looked at the python code so far :) One request: Could take this opportunity to add a validation.md under docs/sites/features ? Particularly with eager / lazy differences, I think users will appreciate docs for this now. Also: We need to review the existing docs, bc this surely changes stuff, e.g. in quickstart

@borchero
Copy link
Member Author

Could take this opportunity to add a validation.md under docs/sites/features ?

This is still missing, I think this can be reviewed regardless 😄

Copy link
Collaborator

@AndreasAlbertQC AndreasAlbertQC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @borchero ! LGTM modulo docs changes

Copy link
Member

@delsner delsner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work @borchero 👏 nothing blocking from my side, looks great.

@delsner
Copy link
Member

delsner commented Oct 23, 2025

For some reason I'm getting this runtime error when executing the tests locally (osx-arm64):

E pyo3_runtime.PanicException: called Result::unwrap() on an Err value: DlSym { desc: "dlsym(0x2839e8910, _polars_plugin_get_version): symbol not found" }

@borchero
Copy link
Member Author

For some reason I'm getting this runtime error when executing the tests locally (osx-arm64):

Since CI doesn't fail it does not seem to be an issue with the implementation. Let's debug this offline.

@borchero borchero enabled auto-merge (squash) October 24, 2025 11:14
@borchero borchero merged commit d010ce6 into main Oct 24, 2025
30 checks passed
@borchero borchero deleted the lazy-validation branch October 24, 2025 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support fully lazy validation

3 participants