test_sample_order_invariance in common metrics tests applied to threshold metrics

[amueller]

This sample invariance test is currently applied to the ranking tests, and that seems strange.
For ranking the scores should have a different shape and be integer (right now y_true and y_pred are binary so it's valid scores but it's not a great test).

Also, the same should be tested for multi-class.
Via #10481.

[amueller]

The same kinda applies to test_format_invariance_with_1d_vectors though that test might make less sense for multiclass.

[amueller]

Similarly test_thresholded_invariance_string_vs_numbers_labels uses binary outcomes instead of scores and doesn't use multiclass data.

[amueller]

@thomasjpfan you implemented this, right?

[thomasjpfan]

@thomasjpfan you implemented this, right?

Nope. Every test that uses METRIC_UNDEFINED_BINARY_MULTICLASS is mentioned in this issue. Looking into this.

[jnothman]

I'm confused by your notion of ranking. I'm pretty sure ranking as far as our metrics are concerned is about having the estimator assign a score to each candidate label, and for the ranking to then be implied, not for the rank to be provided necessarily as an integer.

Not sure exactly what you're looking for here. But yes, it can be hard to know if a particular metric is being fully covered by the common tests.