Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow checks based on data types #1169

Open
NeerajMalhotra-QB opened this issue Apr 27, 2023 · 1 comment
Open

Allow checks based on data types #1169

NeerajMalhotra-QB opened this issue Apr 27, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@NeerajMalhotra-QB
Copy link
Collaborator

NeerajMalhotra-QB commented Apr 27, 2023

Is your feature request related to a problem? Please describe.
Imagine you have this schema:

schema = pa.DataFrameSchema({
    "a": pa.Column(int, checks=pa.Check.le(10)),
    "b": pa.Column(float, checks=pa.Check.lt(-1.2)),
    "c": pa.Column(str, checks=pa.Check.le(20)),
})

In above schema, column c has wrong check. But it will still flow through the entire process and may eventually fail in data side of validation but there may be other situations where check may not fail and fall through the cracks.

Such type of checks shouldn't be allowed for a given data type.

Describe the solution you'd like
There are 2 ways to solve this:

  1. write a decorator to match the checks with respective allowed data types only. We are building it in forked branch here for pyspark.sql - https://github.com/NeerajMalhotra-QB/pandera.

  2. Another (much better) solution will be to enhance register_checks and register_dtype to validate if a check should be allowed for a given type of the field.

We haven't adopted 2nd option yet, as it will require changes in common area (used by other frameworks) but will look into it in future release unless someone wants to take a stab on it first.

cc: @cosmicBboy

@NeerajMalhotra-QB NeerajMalhotra-QB added the enhancement New feature or request label Apr 27, 2023
@NeerajMalhotra-QB
Copy link
Collaborator Author

This is implemented for native pyspark.sql in #1213.

We can leverage it and extend to other frameworks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant