Fix permutation importance when pipeline has target transformer

[freddyaboulton]

Pull Request Description

Fixes #2781

This can go in the release in two weeks. Just opening up for review.

---

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

[codecov]

Codecov Report

Merging #2782 (aaaa613) into main (ce3fc7a) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2782     +/-   ##
=======================================
+ Coverage   99.8%   99.8%   +0.1%     
=======================================
  Files        298     298             
  Lines      27595   27604      +9     
=======================================
+ Hits       27527   27536      +9     
  Misses        68      68             

Impacted Files	Coverage Δ
...alml/model_understanding/permutation_importance.py	100.0% <100.0%> (ø)
...understanding_tests/test_permutation_importance.py	100.0% <100.0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce3fc7a...aaaa613. Read the comment docs.

[angela97lin]

Looks good to me, thank you! I'm curious how you found this bug and determined which permutation importance value was correct 😂

[angela97lin]

If I'm following correctly, we only need to change the fast_scorer because in the slow_scorer we call score --> predict --> predict_in_sample which does inverse_transform, correct?

[freddyaboulton]

Yea - this is why the slow score is the correct one. In the "slow" method we basically treat the whole pipeline as an estimator. So we call pipeline.predict, which does pipeline.inverse_transform. In the fast method, we optimize by only doing the feature engineering once and get the predictions by doing estimator.predict. Since the estimator has no idea it's part of a pipeline, we need to remember to call inverse_transform.

[angela97lin]

Interesting way to use a classification dataset for regression 😂

[freddyaboulton]

I can't decide if I like parametrized tests or not. It's nice to be able to test a lot of cases but it's tough to add a case that doesn't line up 100% with the other cases. Maybe it means I didn't parametrize the test well enough originally 😂

[chukarsten]

Looks good to me!