fix multimethod bug in pyspark

[cosmicBboy]

No description provided.

[cosmicBboy]

hey @mfkapton hope you don't mind, made a PR from your fork, I'll test your changes out further

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.08 🎉

Comparison is base (62bc484) 93.74% compared to head (78efc9c) 93.82%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1260      +/-   ##
==========================================
+ Coverage   93.74%   93.82%   +0.08%     
==========================================
  Files          90       90              
  Lines        6711     6704       -7     
==========================================
- Hits         6291     6290       -1     
+ Misses        420      414       -6     

Impacted Files	Coverage Δ
pandera/api/pyspark/types.py	100.00% <100.00%> (ø)
pandera/backends/pyspark/checks.py	100.00% <100.00%> (ø)

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[mfkaptan]

@cosmicBboy sorry for the late reply. Here's a way to replicate the bug:

import pyspark.sql.types as T
from pyspark.sql import DataFrame, SparkSession

import pandera.extensions as extensions
from pandera.pyspark import DataFrameModel


@extensions.register_check_method(
    supported_types=DataFrame,
)
def fake_check(df: DataFrame):
    return True

class ExampleDFModel(DataFrameModel):
    name: T.StringType()
    age: T.LongType()
    
    class Config:
        fake_check = ()

example_data_cols = ("name", "age")
example_data = [("foo", 42), ("bar", 24)]

# Get the pyspark session
session = SparkSession.builder.config().getOrCreate()
df = session.createDataFrame(example_data, example_data_cols)

out = ExampleDFModel(df, lazy=False)

raises an Exception:

SchemaError: Error while executing check function: DispatchError("No matching functions found")
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/pandera/backends/pyspark/container.py", line 197, in run_checks
    self.run_check(check_obj, schema, check, check_index)
  File "/home/airflow/.local/lib/python3.9/site-packages/pandera/backends/pyspark/base.py", line 85, in run_check
    check_result = check(check_obj, *args)
  File "/home/airflow/.local/lib/python3.9/site-packages/pandera/api/checks.py", line 229, in __call__
    return backend(check_obj, column)
  File "/home/airflow/.local/lib/python3.9/site-packages/pandera/backends/pyspark/checks.py", line 107, in __call__
    check_obj = self.preprocess(check_obj, key)
  File "/home/airflow/.local/lib/python3.9/site-packages/multimethod/__init__.py", line 408, in __call__
    raise DispatchError("No matching functions found")
multimethod.DispatchError: No matching functions found

Here's the output of pip freeze (I'm on python3.9):

req.txt

[mfkaptan]

@cosmicBboy I've added a test that could help you understand the issue. I've changed the order of the commits so now the test is the first commit. You can check out the commit with the test that fails without the fix:

git co 8f854961f47ee4cc8e03625d6fa179cabff18805
pytest tests/pyspark/test_pyspark_model.py::test_registered_dataframemodel_checks

[cosmicBboy]

hi @mfkaptan can you follow the instructions here to run the linters for this project?

[cosmicBboy]

looks like unit tests are failing

[mfkaptan]

looks like unit tests are failing

could you maybe rerun the tests? I feel like this looks irrelevant to my changes 🤔

[mfkaptan]

@cosmicBboy the tests are passing again ✅