Bugfix/882 coercing twice

[ng-henry]

This fixes the coercing twice bug described in #882 + shows all errors even if coercion fails.

@cosmicBboy About your message here:

In summary, if a column cannot be coerced to the intended type, in this case float, pandera won't apply any of the downstream Checks to that column (which is why the nullability check is not picked up in failure_cases.
The reasoning is that if the column is not even of the correct type, then it's reasonable to assume that the validation checks (which assume that type) wouldn't work on that column.

I think that even if some values cannot be coerced to the correct type, we can still do downstream checks. For example, unique and nullable checks can still be applied.

This PR starts to implement some of these features and also fixes the twice coercion bug. It applies downstream checks even if coercion failed. A better approach (TODO) is applying downstream checks only for cells that pass coercion. Thus, it's guaranteed that downstream checks receive the correct type.

If you give the go-ahead, then I can start implementing the TODO.

[cosmicBboy]

what's this change for?

[ng-henry]

Because we are not running coerce in schema_components anymore, we run validate_column in here.

This means wrong values will be passed to validate_column. This error gets caught here because the dtype is wrong.

It just adds another error to the error lists, which is why we have that additional error.

[cosmicBboy]

I think that even if some values cannot be coerced to the correct type, we can still do downstream checks. For example, unique and nullable checks can still be applied.

A better approach (TODO) is applying downstream checks only for cells that pass coercion. Thus, it's guaranteed that downstream checks receive the correct type.

Agreed that this will improve UX and efficiency! The main concern here is that it introduces additional complexity in the validation routine. Among other things, it would need to:

• partition coerced and uncoerced cells in the dataframe, handling cases where coerced cells across columns are potentially unaligned.
• pass only coerced values to downstream checks, handling both column- and dataframe-level checks
• ensure that the failure case reporting takes account of all of this complexity

I think this will be a lot easier to reason about and implement with the overhauled core API:
https://github.com/unionai-oss/pandera/tree/core-schema

The main idea is that the pandera.core subpackage is the schema specification, while the pandera.backends subpackage actually implements the validation logic.

For example, here's the DataFrameSchemaBackend, which implements a validate method and a few core parsers and checks as methods: https://github.com/unionai-oss/pandera/blob/core-schema/pandera/backends/pandas/container.py#L16.

If you can create another issue for the problem of applying checks to an uncoercible column, we can try fleshing out a solution there, working off of the core-schema branch.

[codecov]

Codecov Report

Merging #901 (2151741) into dev (56265ff) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              dev     #901      +/-   ##
==========================================
- Coverage   97.41%   97.38%   -0.03%     
==========================================
  Files          43       43              
  Lines        4171     4172       +1     
==========================================
  Hits         4063     4063              
- Misses        108      109       +1     

Impacted Files	Coverage Δ
pandera/schemas.py	99.24% <100.00%> (+<0.01%)	⬆️
pandera/schema_components.py	99.06% <0.00%> (-0.47%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[ng-henry]

@cosmicBboy Not sure why some of these tests are failing here. I'm not too familiar with Dask and why it prints "2 graph layers" instead of "4 tasks" even though the dataframe is completely the same.

Expected:
    Dask DataFrame Structure:
                    state    city  price
    npartitions=2
    0              object  object  int64
    3                 ...     ...    ...
    5                 ...     ...    ...
    Dask Name: validate, 4 tasks
Got:
    Dask DataFrame Structure:
                    state    city  price
    npartitions=2                       
    0              object  object  int64
    3                 ...     ...    ...
    5                 ...     ...    ...
    Dask Name: validate, 2 graph layers

[cosmicBboy]

yeah @ng-henry this isn't related to your changes. I just fixed this in #877, if you rebase on dev the errors should be handled