BUG: spatial: improve dtype stability for all distance metrics

[AtsushiSakai]

Reference issue

fix #9961

What does this implement/fix?

As reported in #9961, current distance metric functions might have different results with different dtypes (integer vs float).
So, I moved the type conversion codes from sqeuclidean to _validate_vector in order to apply it for every distance metric.
I added a test for it.

[tylerjereddy]

Do we want to use i.e., np.character subdtype check here in case of the use of another flexible dtype like Unicode?

[tylerjereddy]

So we always convert array-like inputs (even a list of "characters") as long as they are not a NumPy array proper?

For proper NumPy arrays, we do coerce to a floating point type when it makes sense, and that includes additional checking for things like characters that can't easily be converted to floats (which we don't check for with other array-like).

I don't know if it matters too much--perhaps attempting an asarray() temporary conversion for the array-like values just for this check might be overkill anyway.

[AtsushiSakai]

Thank you. Right, distance.hamming might receive a string list. I think only a NumPy array should be converted to a floating array. I fixed it and add some tests for it. PTAL.

[tylerjereddy]

here and above I suppose we may wish to use the recommended assert_allclose when adding new code per: https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_almost_equal.html

[tylerjereddy]

Doesn't this change the behavior for sqeuclidean() from what it was before? I'm not sure if we need to worry about it too much, but I'll just outline this below with an example.

This would mean two changes I think:

• we would no longer coerce subtypes of ndarray to floats because of the new is np.ndarray check
• we would no longer coerce array-like to float64 arrays, which it looks like we did before because of the short-circuit on hasattr(u, "dtype")

import numpy as np
from scipy.spatial.distance import sqeuclidean

np.random.seed(31415)

a = np.random.randint(100, size=100, dtype='uint8')
b = np.random.randint(100, size=100, dtype='uint8')

class C(np.ndarray):
    pass

c_arr_a = a.view(C)
c_arr_b = b.view(C)
print(sqeuclidean(c_arr_a, c_arr_b))
print(sqeuclidean(c_arr_a.astype(float), c_arr_b.astype(float)))
print(sqeuclidean(c_arr_a.tolist(), c_arr_b.tolist()))
print(type(sqeuclidean(c_arr_a.tolist(), c_arr_b.tolist())))

# issue9961 feature branch:
# 248
# 166136.0
# 166136
# <class 'numpy.int64'>
# master:
# 166136.0
# 166136.0
# 166136.0
# <class 'numpy.float64'>

[AtsushiSakai]

Thank you again. I missed the a.view(C) case and the type mismatch case. I extended these test cases.
The only solution I could come up with is to convert the input u to ndarray beforehand. Do you have any other better idea?

[peterbell10]

I don't think this is a good idea because it effects metrics expecting boolean input, where floating point input isn't necessary or helpful. Instead, how about creating a new function _validate_real_vector and call it in only the functions that require real inputs.

[AtsushiSakai]

The jensenshannon does not use _validate_vector now. I'm not sure this is intended or not.

scipy/scipy/spatial/distance.py

Lines 1369 to 1370 in eca0cd3

	p = np.asarray(p)
	q = np.asarray(q)

[peterbell10]

It can't use _validate_vector because it supports an axis argument and higher-dimensional input.

[AtsushiSakai]

I rebased to fix CI failures.