Fix MemoryError exception when calling barycenter_weights with large inputs

[bmaisonn]

Reference Issues/PRs

Fixes #10493

What does this implement/fix? Explain your changes.

barycenter_weiths was expecting a (n_samples, n_neighbors, n_dim) array as a second argument. With large number of samples, dimensions and neighbors this provoked MemoryError exceptions.
See a call example here

scikit-learn/sklearn/manifold/_locally_linear.py

Line 105 in 6ddaaed

data = barycenter_weights(X, X[ind], reg=reg)

As noticed in the issue this array is only used to iterate over the first axis so instead of building this huge array we can simply pass the list of indices to the method, then iterate over the indices instead.
The code below is simply changed by building A for each indice.

scikit-learn/sklearn/manifold/_locally_linear.py

Line 54 in 6ddaaed

for i, A in enumerate(Z.transpose(0, 2, 1)):

Any other comments?

This method is private in the module so we can afford changing its signature.
No test was added for this specific use case because doing this computation for large arrays is very long.
In another PR i'll try to improve the time it takes to perform this calculation by leverage n_jobs parameter when it's available.

[bmaisonn]

Hello @ogrisel,
Thx for the review the tests are passing now.
Bertrand

[ogrisel]

LGTM, thanks @bmaisonn. Can you please just add a new entry to the change log under doc/whats_new/0.24.rst?

[ogrisel]

Also can you please re-render the plots of the examples that use LLE and report the comments of this PR to compare those with the LLE plots of master to make sure that we did not introduce a regression not caught by the existing tests?

[jnothman]

Adding a commit with [doc build] in the commit name will re-render the examples in the cloud, accessible using the ci/circleci: doc artifact link.

[bmaisonn]

@ogrisel
Indeed i can see few differences in the documentation (Dev is on the left)
https://113900-843222-gh.circle-artifacts.com/0/doc/_changed.html

[bmaisonn]

Actually when i run those examples on my local env on master they never look the same :)

[bmaisonn]

Hello @ogrisel @jnothman, Is this what you expected?

[ogrisel]

The quality of the viz still look good. The non-determinism must come from something else. LGTM.

[jnothman]

Thanks

[jnothman]

Change log entries should refer to the affected public API items. barycenter_weights is private. Could you try give some sense of how this change affects the user?

[jnothman]

please provide a brief description of what indices is for.

[bmaisonn]

I updated the documentation @jnothman, let me know if you see something else

[jnothman]

Lgtm! Thanks, @bmaisonn

[jnothman]

I wonder if this should be Efficiency since there is no faulty logic as such, rather we have reduced the space complexity.

[bmaisonn]

This can make sense, i don't know all the categories :)
I pushed a new version