You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In above schema, column c has wrong check. But it will still flow through the entire process and may eventually fail in data side of validation but there may be other situations where check may not fail and fall through the cracks.
Such type of checks shouldn't be allowed for a given data type.
Describe the solution you'd like
There are 2 ways to solve this:
write a decorator to match the checks with respective allowed data types only. We are building it in forked branch here for pyspark.sql - https://github.com/NeerajMalhotra-QB/pandera.
Another (much better) solution will be to enhance register_checks and register_dtype to validate if a check should be allowed for a given type of the field.
We haven't adopted 2nd option yet, as it will require changes in common area (used by other frameworks) but will look into it in future release unless someone wants to take a stab on it first.
Is your feature request related to a problem? Please describe.
Imagine you have this schema:
In above schema, column
c
has wrong check. But it will still flow through the entire process and may eventually fail in data side of validation but there may be other situations where check may not fail and fall through the cracks.Such type of checks shouldn't be allowed for a given data type.
Describe the solution you'd like
There are 2 ways to solve this:
write a decorator to match the checks with respective allowed data types only. We are building it in forked branch here for
pyspark.sql
-https://github.com/NeerajMalhotra-QB/pandera
.Another (much better) solution will be to enhance
register_checks
andregister_dtype
to validate if a check should be allowed for a given type of the field.We haven't adopted 2nd option yet, as it will require changes in common area (used by other frameworks) but will look into it in future release unless someone wants to take a stab on it first.
cc: @cosmicBboy
The text was updated successfully, but these errors were encountered: