[MNT] handle pandas future warnings

[solegalli]

No description provided.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.22%. Comparing base (aa60b23) to head (b9942bf).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #891   +/-   ##
=======================================
  Coverage   98.22%   98.22%           
=======================================
  Files         113      113           
  Lines        4842     4844    +2     
  Branches      768      769    +1     
=======================================
+ Hits         4756     4758    +2     
  Misses         55       55           
  Partials       31       31           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[solegalli]

Hey @ankitlade12 @glevv @david-cortes

I would appreciate your input on the changes in match_columns and match_categories. They are done to avoid pandas 4 future warning.

My understanding is that the current pandas implementation will automatically convert to nan values not present in categories. But pandas 4 will fail instead, so we need to explicitly convert the values to nan before the transformation.

Please correct me if I am wrong. I haven't used the categorical dtype in pandas, so not my forte.

[ankitlade12]

Hi @solegalli,

You are absolutely correct. Your understanding aligns with the direction Pandas is heading in terms of stricter categorical data handling.

Why this is necessary:

1. Stricter Type Enforcement: Pandas is moving away from "implicit" behavior where it silently introduces NaNs for unseen categories. In future versions (anticipating Pandas 4.0), operations that result in unobserved categories are expected to be much stricter.
2. Explicit is Better than Implicit: By explicitly converting unseen values to NaN before the transformation, we avoid the FutureWarning that warns about "assignment to a categorical column involving new categories."
3. Avoiding Reindexing Issues: As noted in the recent changes to match_columns.py, using reindex() with a fill_value that isn't already a category can trigger warnings. Explicitly adding missing columns and then setting values is a more robust pattern for the future.

Conclusion:

The current implementation in match_columns and match_categories correctly future-proofs the transformer. It ensures we don't rely on pd.Categorical's default behavior of setting "unseen" values to -1 (code for NaN) without explicit handling.

I'm happy with this approach as it makes the transformers much more resilient to the upcoming Pandas changes.