fix(preprocessing): remove constant NaN columns

[arthur-priorlabs]

Issue

Closes PRI-241 — all-NaN columns were not being removed as constant columns, affecting downstream preprocessing.

Motivation and Context

RemoveConstantFeaturesStep decides which columns to keep using an equality comparison (X[0:1, :] == X). For an all-NaN column every comparison is False (NaN != NaN), so the column was treated as non-constant and kept, even though it carries no information. These columns then propagate NaNs into later preprocessing steps.

This adds an explicit guard so a column that is entirely NaN is also treated as constant and dropped:

• NumPy path: ~np.all(np.isnan(X), axis=0) is AND-ed into the keep mask.
• Torch path: & ~X.isnan().all(dim=0).

At this point in the pipeline X is always a numeric float array (categoricals/strings are ordinally encoded and cast to float64 by clean_data upstream), so np.isnan / Tensor.isnan are safe.

---

Public API Changes

• No Public API changes
• Yes, Public API changes (Details below)

---

How Has This Been Tested?

• Added unit tests covering all-NaN column removal for both NumPy and torch inputs, and ran the step's full test suite locally.
• Evaluated the actual performance trade-off

---

Checklist

• The changes have been tested locally.
• Documentation has been updated (if the public API or usage changes). — N/A, no API change
• A changelog entry has been added (see changelog/README.md), or "no changelog needed" label requested.
• The code follows the project's style guidelines.
• I have considered the impact of these changes on the public API.

[gemini-code-assist]

Code Review

This pull request updates the RemoveConstantFeaturesStep to correctly identify and drop columns that consist entirely of NaN values for both PyTorch and NumPy inputs, and adds corresponding unit tests. The review feedback points out a potential runtime error when applying NaN checks on integer-type inputs, suggesting that we should verify if the input has a floating-point data type before checking for NaNs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[arthur-priorlabs]

One thing to consider is that #1055 will pass infs as nans through the preprocessing pipeline - meaning that a column with only nan, inf and -inf will get parsed as all-nan and dropped instead of passed through.
Don't think it's hyper relevant because it's all missing data at the end of the day.

[adrian-prior]

Generally, lgtm! I am still wondering about what you have mentioned:

Do inf/-inf not (at least theoretically) carry more signal than just all NaNs? E.g., I might have sensors that record inf/-inf if the values fall outside the range I can measure. If this is the case, we would drop columns that could carry meaningful signal.

I am not super in the loop on the current inf/NaN, so apologies if this doesn't make sense.

[arthur-priorlabs]

Good question!

inf and -inf do theoretically carry more signal than all NaNs, but so do all-constant columns, yet we're dropping them too: what's the actual rationale behind RemoveConstantFeaturesStep?.

I get your point with sensors, but I'm not sure how often we'll see the same sensor only ever output inf and -inf in the same dataset in practice; sounds like a very faulty sensor, but I guess that is information too!

[arthur-priorlabs]

Added a guard against this in 6c0c8d7 (so in #1055)

[adrian-prior]

Sorry, for coming back late I was OOO on Friday.

Approving this to not block anything, but I am not sure that I am convinced yet!

I get your point with sensors, but I'm not sure how often we'll see the same sensor only ever output inf and -inf in the same dataset in practice; sounds like a very faulty sensor, but I guess that is information too!

If we don't think -inf/+inf should be treated differently from NaNs, then shouldn't we just overwrite them as NaNs in a previous step and check only for NaNs here? I feel like I might be missing some details regarding what the exact idea of this refactor is, but I feel like either (a) we treat Infs as a first class citizen or (b) just overwrite them with NaNs. I don't really see how the in-between makes sense.

[arthur-priorlabs]

Thanks!
I've fixed it in 6c0c8d7 anyway :)