-
-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Raise SchemaInitError when for unique field is specified in pyspark field and a test showing the issue as well as config when it works. #1592
Conversation
How is this the case? Specifying (Also see here to signing your commits for the DCO check) |
Okay so there are two issues here:
import pandera.pyspark as pa
class Model(pa.DataFrameModel):
class Config:
unique = ["col1", "col2"] # col1 and col2 should be jointly unique
class Model(pa.DataFrameModel):
col: int = pa.Field(unique=True) # values in col need to be unique For 1 the For 2, a |
Thanks @cosmicBboy, |
…showing the issue as well as config when it works. Signed-off-by: Marc Ferradou <mferradou@RWG7599T39.grubhub.local> Signed-off-by: zippeurfou <zippeurfou@gmail.com>
6e25dfb
to
349f65d
Compare
@cosmicBboy updated the PR according to my understanding. |
I am not sure why the linter didn't execute here. |
@@ -177,6 +178,10 @@ def Field( | |||
else: | |||
check_ = check_constructor(arg_value, **check_kwargs) | |||
checks.append(check_) | |||
if unique is True: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: this can just be if unique:
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1592 +/- ##
===========================================
- Coverage 94.29% 83.16% -11.14%
===========================================
Files 91 114 +23
Lines 7024 8504 +1480
===========================================
+ Hits 6623 7072 +449
- Misses 401 1432 +1031 ☔ View full report in Codecov by Sentry. |
Thanks @zippeurfou and congrats on your first contribution to pandera! 🚀 |
Following #1344 I am adding a bit of edited documentation.
![Screenshot 2024-04-19 at 5 39 04 PM](https://private-user-images.githubusercontent.com/4520383/324127317-03e8027e-8092-4107-ac8f-4dbc8dc7cb3e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYwMzg5MzcsIm5iZiI6MTcyNjAzODYzNywicGF0aCI6Ii80NTIwMzgzLzMyNDEyNzMxNy0wM2U4MDI3ZS04MDkyLTQxMDctYWM4Zi00ZGJjOGRjN2NiM2UucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDkxMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA5MTFUMDcxMDM3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OGZlZjg1OThmNjYyMzc2ODVlMGFhNWE5OGFkNGVkMDdlNzA2YjE1NWY4NmUxYWM0NmFkMTdkNmEwODVjODYzMSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.Wk3UijVEmB6MMlqzEXdJZVi9_RIiFofaq0FziCSN-Io)
I wasn't able to raise
SchemaInitError
as @cosmicBboy suggested as it turns out that if you usepa.Field(unique=True)
it is seen as None which most likely is the issue.In the follow up screenshot you can see the behavior when I did add the code where @cosmicBboy suggested.
As the ghost text show it is None when I put a breakpoint there so I did not add it.