How to match all other columns not explicitly defined in Schema Model for checks?

[SebBanDev]

I need to define a Schema Model where I expect some columns to have defined names and defined checks and all other columns should have different checks.

So far, I only made it work with some regex hack:

import pandas as pd
import pandera as pa
from pandera import check_types
from pandera.typing import DataFrame, Series


class Model(pa.SchemaModel):
    """ Example Model """

    some_col: Series[int] = pa.Field(coerce=True)
    another_col: Series[int] = pa.Field(coerce=True)
    _other_col_regex = rf"(?!{'|'.join(__annotations__.keys())}\b)\b.*"
    all_other_cols: Series[float] = pa.Field(gt=0, coerce=True, alias=_other_col_regex, regex=True)

    class Config:
        strict = True



@check_types
def dummy_func(data:  DataFrame[Model]):
    return data


data = pd.DataFrame({
    'some_col': [-1, -1, -1],
    'another_col': [-1, -1, -1],
    'this_1_is_a_$uper_#weird_col': [1, 1, 1]
})

dummy_func(data)

What is the most convenient way to apply some checks to "all other column" in the data frame but the explicitly defined ones?

[SebBanDev]

Is there an easier way to exclude other defined columns from a regex column?

If I change the _other_col_regex in my example to match all strings:
_other_col_regex = r".*"

It also matches the 'some_col' and 'another_col' columns and raises an error for them:

    raise errors.SchemaError(
pandera.errors.SchemaError: error in check_types decorator of function 'dummy_func': <Schema Column(name=some_col, type=DataType(float64))> failed element-wise validator 0:
<Check greater_than: greater_than(0)>
failure cases:
   index  failure_case
0      0          -1.0
1      1          -1.0
2      2          -1.0

Maybe I am missing a way to use data frame checks to check all columns but "some_col", "another_col" or whatever other column is defined already in the model in other cases without hardcoding the col names in the regex or getting them from __annotations__

[SebBanDev]

I found a workaround with a custom data frame check that does the job, but it also seems like a hacky solution:

import pandas as pd
import pandera as pa
from pandera import check_types
from pandera.typing import DataFrame, Series


class Model(pa.SchemaModel):
    """ Example Model """

    some_col: Series[int] = pa.Field(coerce=True)
    another_col: Series[int] = pa.Field(coerce=True)

    _all_other_cols = pa.SeriesSchema(
        float,
        checks=[
            pa.Check.greater_than(0),
        ],
        coerce=True
    )

    class Config:
        strict = False

    @pa.dataframe_check
    def undefined_columns(cls, df: pd.DataFrame) -> bool:
        for col in df.columns:
            if col not in cls.__fields__.keys():
                cls._all_other_cols.validate(df[col])
        return True


@check_types
def dummy_func(data:  DataFrame[Model]):
    return data


data = pd.DataFrame({
    'some_col': [-1, -1, -1],
    'another_col': [-1, -1, -1],
    'this_1_is_a_$uper_#weird_col': [1, 1, 1]
})

dummy_func(data)

Maybe there is still a better way to handle "other columns"?

[cosmicBboy]

hi @SebbanSms, good question! currently I'd recommend your first approach; here's a slightly modified version that exclude private vars:

class Model(pa.SchemaModel):
    """ Example Model """

    some_col: Series[int] = pa.Field(coerce=True)
    another_col: Series[int] = pa.Field(coerce=True)
    all_other_cols: Series[float] = pa.Field(
        gt=0,
        coerce=True,
        alias=rf"^(?!{'|'.join(k for k in __annotations__ if not k.startswith('_'))}).*$",
        regex=True,
    )

    class Config:
        strict = True

In looking into other solutions I realized that the Any type annotation is not supported, so #594 will add support for a solution like this:

class Model(pa.SchemaModel):
    """ Example Model """

    some_col: Series[int] = pa.Field(coerce=True)
    another_col: Series[int] = pa.Field(coerce=True)
    undefined_col: Series[Any] = pa.Field(alias=".*", regex=True)

    class Config:
        strict = True

    @pa.check(".*")
    def undefined_column(cls, series: pd.Series) -> pd.Series:
        if series.name in cls.__fields__.keys():
            return True
        return series > 0

Depending on the demand for a use case like this we might consider adding first class support for a notion of "apply these checks to undefined columns" or "exclude these patterns from regex", but for now I think these two solutions should suffice.