ENH reduce memory consumption in nan_euclidean_distances

[jnothman]

This goes towards fixing #15604

Basic benchmark:

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.metrics.pairwise import nan_euclidean_distances

calhousing = fetch_california_housing()

X = pd.DataFrame(calhousing.data, columns=calhousing.feature_names)
y = pd.Series(calhousing.target, name='house_value')

rng = np.random.RandomState(42)

density = 4  # one in 10 values will be NaN

mask = rng.randint(density, size=X.shape) == 0
X_na = X.copy()
X_na.values[mask] = np.nan

%time nan_euclidean_distances(X_na)

At master:

CPU times: user 33 s, sys: 9.42 s, total: 42.4 s
Wall time: 24 s

This branch:

CPU times: user 27.3 s, sys: 3.71 s, total: 31 s
Wall time: 14.8 s

Faster times could possibly be achieved with chunking.

[jnothman]

The test failure is because of this expected distance matrix:

        X = np.array([[missing_value, missing_value], [0, 1]])
        exp_dist = np.array([[np.nan, np.nan], [np.nan, 0]])

This sets the standard that if a sample is all-nan, its euclidean distance to itself is nan rather than 0. My code currently sets the diagonal to 0 in all cases. What do we consider to be the right behaviour?

[thomasjpfan]

Originally I had the diagonal at zero (when X is Y), but I was concerned with how the following were not equal:

X = np.array([[missing_value, missing_value], [0, 1]])

nan_euclidean_distances(X, X.copy())
nan_euclidean_distances(X, X)

[glemaitre]

What do we consider to be the right behaviour?

At least it is documented nan.

Since the distance is a bit specific to handle nan, I would not be surprised that it returns nan while I would be surprised to have some nan in other metrics due to division by zero or stuff like that.

@jnothman Do you have a use case or any thought on what it should be 0 instead of nan.

[glemaitre]

Apart from this LGTM

[jnothman]

I'm happy with @thomasjpfan's reasoning for now. In any case, better that an efficiency fix like this does not change behaviour

[glemaitre]

So LGTM. @thomasjpfan Do you want to give a look.

[thomasjpfan]

Thank you @jnothman !