Raise SchemaInitError when for unique field is specified in pyspark field and a test showing the issue as well as config when it works.

[zippeurfou]

Following #1344 I am adding a bit of edited documentation.
I wasn't able to raise SchemaInitError as @cosmicBboy suggested as it turns out that if you use pa.Field(unique=True) it is seen as None which most likely is the issue.
In the follow up screenshot you can see the behavior when I did add the code where @cosmicBboy suggested.

As the ghost text show it is None when I put a breakpoint there so I did not add it.

[cosmicBboy]

if you use pa.Field(unique=True) it is seen as None which most likely is the issue.

How is this the case? Specifying pa.Field(unique=True) should translate to unique is True when the DataFrameModel is translated into a DataFrameSchema

(Also see here to signing your commits for the DCO check)

[cosmicBboy]

Okay so there are two issues here:

1. specifying unique at the dataframe-level

import pandera.pyspark as pa

class Model(pa.DataFrameModel):
    class Config:
        unique = ["col1", "col2"]  # col1 and col2 should be jointly unique

3. specifying pa.Field(unique=True) at the column level.

class Model(pa.DataFrameModel):
    col: int = pa.Field(unique=True)  # values in col need to be unique

For 1 the SchemaInitError here still makes sense: https://github.com/unionai-oss/pandera/blob/main/pandera/api/pyspark/container.py#L148

For 2, a SchemaInitError here needs to be added: https://github.com/unionai-oss/pandera/blob/main/pandera/api/pyspark/model_components.py#L184. This is because the underlying Column definition doesn't even support the unique argument: https://github.com/unionai-oss/pandera/blob/main/pandera/api/pyspark/components.py#L15-L27

[zippeurfou]

Thanks @cosmicBboy,
Let me rephrase things a bit:
For 1 with config it does work as you expressed. I added a unit test with only one column but I can add a second one with 2 columns.
For 2, I can add it where you mentioned. I hadn't had the time to look too much at how internal works so I appreciate the direction.
My guess is given 1 was implemented and works as expected, 2 should not be impossible to implement but right now I don't have the bandwidth to do it sadly as I would need extra time to understand the internal of the library.
For the DCO I will try to do it as well when I have time. I appreciate the direction.

[zippeurfou]

@cosmicBboy updated the PR according to my understanding.

[zippeurfou]

I am not sure why the linter didn't execute here.

[cosmicBboy]

nit: this can just be if unique:

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.16%. Comparing base (4df61da) to head (4fffae2).
Report is 75 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1592       +/-   ##
===========================================
- Coverage   94.29%   83.16%   -11.14%     
===========================================
  Files          91      114       +23     
  Lines        7024     8504     +1480     
===========================================
+ Hits         6623     7072      +449     
- Misses        401     1432     +1031     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cosmicBboy]

Thanks @zippeurfou and congrats on your first contribution to pandera! 🚀