-
-
Notifications
You must be signed in to change notification settings - Fork 299
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Design Data Types Library That Supports Both PySpark & Pandas #1360
Comments
Any comments on this? |
Long story short, the primitives are there, it'll just be some work before we can realize the vision of "one DataFrameModel to rule them all" 💍. I see this question can be broken down into two sub-problems:
Agreed, but you might be surprised how challenging this is to get right 🙃. I'd say (2) is a little easier to tackle right now. Basically we'd need to add the library-agnostic data types as supported equivalent types in the As for (1), that's going to take more work, but basically we'd need to create a generic @NeerajMalhotra-QB @jaskaransinghsidana FYI. I guess to kick this effort off, would you mind sharing some example code of what you're doing today @lior5654 ? |
Thanks for the detailed answer! I really appreciate it. I'll share a minimal PoC soon, but another question arises - do you recommend other libraries to do this these days? Are you aware of any libraries that currently support such concepts? @cosmicBboy |
@lior5654 as far as I know I don't know of efforts to create a "unified dataframe model for schema validation"... The "one DataFrameModel to rule them all" really is the goal here, but for the longest time pandera only supported Now with the efforts to support polars and ibis I think we're in a good position to generalize the API so we can have a generic |
Thanks again for the reply. before I send code samples, I have another question. |
But I think my intentions are pretty clear, here's a code sample: import pyspark.pandas as ps
import pandas as pd
import pandera as pa
# should GENERIC, for ALL pandas compliant APIS
# in the future - also for non pandas compliat APIS (only when applicable)
# currently - this is for pandas itself only
from pandera.typing import DataFrame, Series
class Schema(pa.DataFrameModel):
state: Series[str]
city: Series[str]
price: Series[int] = pa.Field(in_range={"min_value": 5, "max_value": 20})
# validate a pd.DataFrame (this will work)
# validate a ps.DataFrame (this has it's own Series type so you need to define the same class with a different Series
# ... I guess for a start, supporting all pandas-compliant APIs with a single class should be easy, right? |
Oh I just tested, and it seems like the pandera pandas DataFrame model works seamlessly with pyspark.pandas API, great to know! |
Yep! The pyspark.pandas integration has been around for longer |
Yes, this is possible today with the backend extensions plugin. This is currently done for dask, modin, and pyspark.pandas. The challenge is making |
Is there any update now that polars (#1064) is also supported, @cosmicBboy? Poking around the API and pandera's source code, it seems like I'm already very close to being able to define a functional shared Here is a POC that seems to work: Albeit I can't validate directly with the shared model. I guess it should be straight forward to enable validation with the shared model by having import pandas as pd
import polars as pl
# Pandera pandas
import pandera as pd_pa
# Pandera polars
import pandera.polars as pl_pa
# Pandera library-agnostic
from pandera.api.dataframe.model import DataFrameModel
from pandera.api.dataframe.model_components import Field
from pandera.dtypes import String, Int64
from pandera.errors import SchemaErrors
class SharedModel(DataFrameModel):
a_str: String = Field(isin=["ok"])
an_int: Int64 = Field(gt=0)
class PandasModel(SharedModel, pd_pa.DataFrameModel):
...
class PolarsModel(SharedModel, pl_pa.DataFrameModel):
...
def main():
# define a pandas and a polars dataframe with the same content
pa_df = pd.DataFrame({
"a_str": [
"ok",
"not ok", # should raise
],
"an_int": [
-1, # should raise
1,
]
})
pl_df = pl.from_pandas(pa_df)
print("pandas dataframe / model works")
try:
PandasModel.validate(pa_df, lazy=True)
assert False
except SchemaErrors as e:
print(e)
print("polars dataframe / model works")
try:
PolarsModel.validate(pl_df, lazy=True)
assert False
except SchemaErrors as e:
print(e)
print("pandas / polars dataframe with shared model doesn't work: raises NotImplementedError")
for df in (pa_df, pl_df):
try:
SharedModel.validate(df, lazy=True)
assert False
except NotImplementedError:
...
if __name__ == "__main__":
main() Which outputs:
|
Design Data Types Library That Supports Both PySpark & Pandas
Hi, I have multiple data types I commonly work with, sometimes in pandas and sometimes in pyspark.
I don't want to create 2 pandera DataFrameModels for each type, that seems like a really bad practice.
What's the best way to currently do this?
Is there also a way to write code that will work both on pyspark & pandas?
The text was updated successfully, but these errors were encountered: