FEA Add missing-value support for ExtaTreeClassifier and ExtaTreeRegressor

[adam2392]

Reference Issues/PRs

Towards: #27931

Follow-up to: #26391 and #23595

What does this implement/fix? Explain your changes.

• Adds missing-value support to RandomSplitter in _splitter.pyx
• Enables the "random" splitter kwarg for ExtraTreeClassifier and ExtraTreeRegressor
• Adds unit-tests for ExtraTreeClassifier and ExtraTreeRegressor
• Make unit-tests for DecisionTree* and ExtraTree* more numerically robust by increasing the tolerance for a GLOBAL_RANDOM_SEED check, and using cross-validation scores rather than a single score

Any other comments?

Compared to BestSplitter, there can be an expected cost to doing splits on missing-values, as we can either:

1. choose a random threshold and then randomly send missing-values to left/right, OR
2. send all missing-values to left/right and all non-missing values to right/left

The push of missing values down the tree can be done randomly (i.e. first option), OR the second option can actually be evaluated. There is a computational cost to doing so, but more importantly there is an interpretation tradeoff. The tradeoff imo comes from the assumption of the missing-values:

• if missing-completely-at-random (MCAR), then option 1 is ideal because one simply should ignore the missing values or impute them
• if missing-at-random (MAR), then option 2 is nice because sometimes the missing-ness of the data can be informative.

However, I think the difference at a tree level is not super important. E.g. in the ExtraTree forest, #28268 demonstrates that the ExtraTrees when combined as a forest are resilient and predictive for missing-values.

Benchmarks demonstrating no significant runtime performance degradation

There is some complexity involved in checking if there are missing values. However, this only occurs at the Python level as shown by the following benchmark. In terms of the Cython code, there is no regression.

Benchmarks with and without Python Check

Also ran this benchmark for ExtraTrees, which demonstrates that this check is negligible at the forest level, since it only occurs once. See #28268, which has the short code to enable it for ExtraTrees.

Benchmarks on ExtraTrees

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 20e9d9f. Link to the linter CI: here}

[OmarManzoor]

Minor comments otherwise this looks good now

[adam2392]

Perhaps @glemaitre and @thomasjpfan can take one last look then?

I think this has changed a bit since @glemaitre approved changes, and I believe I've addressed most of the core issues raised.

Thanks everyone for reviewing!

[OmarManzoor]

LGTM! Thanks a lot @adam2392.

[glemaitre]

Sorry for the delay, I was a bit under the water. LGTM and I see the auto-merge was activated.
Thanks @adam2392