Raise ValueError when predict/predict_proba input types don't match fit input

[bchen1116]

fix #2855

Running

import pandas as pd
from evalml import AutoMLSearch
import logging
logging.basicConfig(level=logging.ERROR)

path = "/Users/bryan.chen/Downloads/string_nan_ex/1625078186889-mushroom_subset.csv"

df = pd.read_csv(path)
y_train = df['class']
X_train = df.drop('class', axis=1)

aml = AutoMLSearch(X_train, y_train, 'binary', verbose=False)
aml.search()


pipeline = aml.best_pipeline
holdout = pd.read_csv("/Users/bryan.chen/Downloads/string_nan_ex/mushroom_holdout.csv")
pipeline.predict(holdout)

now results in:

[codecov]

Codecov Report

Merging #3036 (76f733b) into main (d6682a4) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3036     +/-   ##
=======================================
+ Coverage   99.8%   99.8%   +0.1%     
=======================================
  Files        312     312             
  Lines      30340   30421     +81     
=======================================
+ Hits       30249   30330     +81     
  Misses        91      91             

Impacted Files	Coverage Δ
evalml/utils/__init__.py	100.0% <ø> (ø)
evalml/pipelines/component_graph.py	99.8% <100.0%> (+0.1%)	⬆️
...valml/tests/pipeline_tests/test_component_graph.py	99.9% <100.0%> (+0.1%)	⬆️
evalml/tests/pipeline_tests/test_pipelines.py	99.8% <100.0%> (+0.1%)	⬆️
evalml/tests/utils_tests/test_woodwork_utils.py	100.0% <100.0%> (ø)
evalml/utils/woodwork_utils.py	100.0% <100.0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d6682a4...76f733b. Read the comment docs.

[bchen1116]

Need to add this when we mock pipeline fit in order to not raise the valueError

[bchen1116]

After discussion with @freddyaboulton and @angela97lin, we decided to treat Integer and Doubles as the same logical types until @chukarsten's work with NullableInteger goes in. At that point, we can revisit this and see what the best way to accommodate that would be.

[freddyaboulton]

@bchen1116 I think we can get rid of schema_is_equal and let woodwork do the comparison for us if we use exclude. Something like this

        if not fit:
            if X.ww.select(exclude=['integer'], return_schema=True) != self._input_types:
                raise ValueError(
                    "Input X data types are different from the input types the pipeline was fitted on."
                )
        else:
            self._input_types = X.ww.select(exclude=['integer'], return_schema=True)

[bchen1116]

@freddyaboulton I had to implement this because the schema for logical types included the logical type objects (ie Categorical(), Integer()), and the equality would fail here when the objects are different instances.

[freddyaboulton]

Wild, thanks for explaining! Can we add a unit test for this scenario? The unit tests on this branch pass with using the schema equality method above. I think we should also file a woodwork issue? I feel like schema equality should work in this case.

[bchen1116]

Can add a test for this case! I brought it up to the woodwork team, but seems like this is something they want to keep

[freddyaboulton]

I looked into this yesterday and I think it's cause the DateTime format is None in X2 in the repro you shared.

I will try to come up with a minimal repro and share with ww team. This ends up working

from evalml.demos import load_fraud

X, y = load_fraud(1000)

X2 = X.ww.copy()

X.ww.schema == X2.ww.schema

[freddyaboulton]

Thank you @bchen1116 ! I am excited to get this out the door, hopefully it'll help debug some tricky problems that can happen between fit and predict.