Distances for agglomerativeclustering

[FrancescoCasalegno]

Reference Issues/PRs

Fixes #16701.
Closes #16903.

What does this implement/fix? Explain your changes.

It adds an optional argument to AgglomerativeClustering that enables the return_distance switch on all subsequent calls to set the distances_ attribute. This could make plotting a dendrogram from the resulting model much easier (as requested in #16701)

Any other comments?

This PR takes over where the stalled PR #16903 left.

ToDo

• Add test for parameter distances_ in sklearn/cluster/tests/test_hierarchical.py
• We should update FeatureAgglomeration and document the distances_ attribute (we can just use [MRG] DOC document distances_ attribute #17308).

[rth]

Thanks @FrancescoCasalegno , we would need to add a test for this parameter in sklearn/cluster/tests/test_hierarchical.py

Please add an entry to the change log at doc/whats_new/v0.24.rst. Like the other entries there, please reference this pull request with :pr: and credit yourself (and other contributors if applicable) with :user:.

[thomasjpfan]

Thank you for the PR @FrancescoCasalegno !

[ogrisel]

I believe the documentation of the distances_ attribute should be reworked to better describe what those distances mean.

[FrancescoCasalegno]

Bonjour @ogrisel , thank you for your detailed feedback! I hope I managed to address all your concerns.

I believe the documentation of the distances_ attribute should be reworked to better describe what those distances mean.

On this specific point, could you please clarify what you mean by "better describing"?
Imho reading the doc of the previous attributes it is quite clear we are talking about the nodes of the hierarchical tree representing the recursive agglomerative approach of this clustering model. What would you precisely like to further precise in the docstring?

Moreover, notice that the distances_ and the definition of its meaning are not introduced by this PR, so maybe this can be addressed in another Issue/PR?

[ogrisel]

LGTM. Thank you very much @FrancescoCasalegno and @EmilieDel.

Maybe in a follow-up PR, could you please try to extehd the example https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html to show how to plot the dendrogram of non-complete clustering where we would use the compute_distance parameter without setting the threshold to 0 (for instance with n_clusters=4)?

[rth]

Thanks @EmilieDel and @FrancescoCasalegno .

LGTM, aside for a minor comment about the what's new.

[ogrisel]

Merged! Thanks again to all.