[MRG] Fix n_components kwarg missing in SpectralClustering. See #13698 #13726

fdas3213 · 2019-04-25T19:27:07Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

The SpectralClustering class does not have an instance variable n_components; that's why n_components cannot be accessed by an instance of SpectralClustering class. My temporary fix is to add n_components as an instance variable of SpectralClustering class, and add it as an argument of spectral_clustering in fit.
If this is a reasonable fix of the problem, I will then add a test function def test_n_components in test_spectral.py. Otherwise please let me know what is the best way to fix this.

Any other comments?

fdas3213 · 2019-04-30T21:54:36Z

Assertion error happens at the last line assert param_value == init_param.default, init_param.name in check_parameters_default_constructible in sklearn/utils/estimator_checks.py because param_value = n_clusters if n_components is None, but init_param.default = None, and because n_clusters != None, this error occurs.

I tried to fix this by setting self.n_components = n_components in the constructor of SpectralClustering class and then change the value of self.n_components to self.n_cluster in fit function, however by doing this, I got another error Estimator SpectralClustering should not change or mutate the parameter n_components from None to 8 during fit, which basically says that I cannot change the value of instance variable outside the constructor.

These two errors might be the reason why n_components is not set an instance variable for SpectralClustering class.

So does anyone have any idea on how to handle this issue?

thomasjpfan · 2019-05-06T14:08:09Z

sklearn/cluster/spectral.py

        self.n_clusters = n_clusters
        self.eigen_solver = eigen_solver
+        self.n_components = n_clusters if n_components \


Set self.n_components = n_components and pass self.n_components directly to spectral_clustering without modifying any of the parameters in __init__. spectral_clustering already handles the case when n_components is defined:

scikit-learn/sklearn/cluster/spectral.py

Line 256 in 8d3b4ff

n_components = n_clusters if n_components is None else n_components

Nice idea! thanks for help

fdas3213 · 2019-05-07T18:33:22Z

@thomasjpfan When I run pytest locally, there is no DocTestFailure; however when running checks after submitting new commit, I got 2 DocTestFailure. Do you know why this happens and how should I fix this problem? Thank you

thomasjpfan · 2019-05-07T23:24:17Z

Merge with master

fdas3213 · 2019-05-08T17:16:45Z

Merge with master

I think the branch is already up to date with the master. Could you please be more specific on how should I do this? Thanks

thomasjpfan · 2019-05-08T17:21:03Z

git checkout LOCAL_BRANCH_NAME
git remote add upstream https://github.com/scikit-learn/scikit-learn.git
git fetch upstream
git merge upstream/master

We updated how we build docs in circleci, you need to merge with master to build the docs without error.

fdas3213 · 2019-05-08T17:23:16Z

git checkout LOCAL_BRANCH_NAME
git remote add upstream https://github.com/scikit-learn/scikit-learn.git
git fetch upstream
git merge upstream/master
We updated how we build docs in circleci, you need to merge with master to build the docs without error.

thanks so much for help

@thomasjpfan
Seems that DocTestFailure still exists after merging with master.
I fixed this error locally by changing
array([1, 1, 1, 0, 0, 0])
to
array([1, 1, 1, 0, 0, 0], dtype=int64) and everything looks good locally when tested with pytest.
But even after merging with master, this DocTestFailure still exists. Any help is appreciated!

fdas3213 · 2019-05-12T21:32:46Z

@thomasjpfan There is still that doc error after merging with master. Do you have any hint about this? Or is there anything that I did not do right? thank you

thomasjpfan · 2019-05-13T18:17:27Z

sklearn/cluster/spectral.py

@@ -383,12 +386,12 @@ class SpectralClustering(BaseEstimator, ClusterMixin):
    ...         assign_labels="discretize",
    ...         random_state=0).fit(X)
    >>> clustering.labels_
-    array([1, 1, 1, 0, 0, 0])
+    array([1, 1, 1, 0, 0, 0], dtype=int64)


Revert this change

fdas3213 · 2019-05-14T20:10:28Z

@thomasjpfan I think this PR is ready to be reviewed. Thank you

jnothman

Please add a brief test to check that n_components affects the result, and that it is equivalent to n_clusters by default.

jnothman · 2019-05-21T08:43:28Z

sklearn/cluster/spectral.py

@@ -307,6 +307,9 @@ class SpectralClustering(BaseEstimator, ClusterMixin):
        to be installed. It can be faster on very large, sparse problems,
        but may also lead to instabilities.

+    n_components : integer, optional, default is n_clusters


Suggested change

n_components : integer, optional, default is n_clusters

n_components : integer, optional, default=n_clusters

Sure. Should I add the test as def test_n_components in test_spectral.py?

Yes, please!

To test that n_components is equivalent to n_clusters by default, I need to initialize the value of n_components in __init__ as self.n_components = n_clusters if n_components is None else n_components.

However, this would lead to Assertion error that happens at the last line assert param_value == init_param.default, init_param.name in check_parameters_default_constructible in sklearn/utils/estimator_checks.py because param_value = n_clusters if n_components is None, but init_param.default = None, and because n_clusters != None, this error occurs.

As thomasjpfan suggested, I "Set self.n_components = n_components and pass self.n_components directly to spectral_clustering without modifying any of the parameters in __init__. " But in this way, I cannot test n_components is equivalent to n_clusters by default, since self.n_components = None and self.n_clusters = n_clusters.

Any suggestion on this? Thank you

jnothman · 2019-05-22T22:52:33Z

I just mean that you can test that the result of the model is the same as if you set n_components=n_clusters. At this point you should only be modifying test files, no?

fdas3213 · 2019-05-23T00:04:55Z

I just mean that you can test that the result of the model is the same as if you set n_components=n_clusters. At this point you should only be modifying test files, no?

Got it. Thanks for the clarification. Yes, only test files will be modified.

jnothman

Good test, thanks!!

fdas3213 · 2019-05-24T16:58:42Z

Good test, thanks!!

No problem, glad it gets approved. There were still 2 checks failing, do I have to do something?

jnothman · 2019-05-25T13:22:22Z

Those failures are happening on all pull requests. Will be fixed by us

NicolasHug

LGTM,

Please add an entry to the change log at doc/whats_new/v0.22.rst. Like the other entries there, please reference this pull request with :pr: and credit yourself (and other contributors if applicable) with :user:.

jnothman · 2019-05-26T22:09:00Z

doc/whats_new/v0.22.rst

+:mod:`sklearn.cluster`
+..................
+
+- |Fix| :class:`cluster.SpectralClustering` now accepts a ``n_components`` 


I'd call this an enhancement. Fix indicates that someone's code was previously not functioning as documented, ordinarily. Rather, you provide a way to access existing capabilities

Thanks for letting me know. Just made the change

(scikit-learn#13726)

fdas3213 added 5 commits Apr 25, 2019

add n_components as instance variable

4e1cb1c

test SpectralClustering object access n_components

5323ca0

called fit function to access n_components

727f2a4

fixed assertion error

e71b4c5

fix doctest failure

fb12eff

fdas3213 mentioned this pull request Apr 30, 2019

n_components kwarg missing in SpectralClustering #13698

Closed

thomasjpfan reviewed May 6, 2019

View changes

do not modify value of self.n_components

6468bdf

fdas3213 closed this May 8, 2019

fdas3213 reopened this May 8, 2019

Merge remote-tracking branch 'upstream/master'

8089b72

thomasjpfan reviewed May 13, 2019

View changes

fdas3213 added 2 commits May 13, 2019

Merge remote-tracking branch 'upstream/master'

867b7b2

fix doctest error

81834c5

fdas3213 changed the title ~~[WIP] Fix n_components kwarg missing in SpectralClustering. See #13698~~ [MRG] Fix n_components kwarg missing in SpectralClustering. See #13698 May 13, 2019

jnothman reviewed May 21, 2019

View changes

Merge remote-tracking branch 'upstream/master'

ade380f

fdas3213 added 3 commits May 23, 2019

Merge remote-tracking branch 'upstream/master'

9098b4b

add brief test

16f034b

changed n_components to 1

ee6dde2

fdas3213 added 2 commits May 23, 2019

fix assertion error

8a70366

Merge remote-tracking branch 'upstream/master'

179b35d

jnothman approved these changes May 23, 2019

View changes

NicolasHug approved these changes May 25, 2019

View changes

fdas3213 added 2 commits May 26, 2019

Merge remote-tracking branch 'upstream/master'

776fed9

add entry to change log

cbe3a16

jnothman reviewed May 26, 2019

View changes

fdas3213 and others added 2 commits May 27, 2019

change fix to enhancement

3b829a8

Update v0.22.rst

ce1c3cb

jnothman merged commit db48ebc into scikit-learn:master May 27, 2019
8 of 13 checks passed

koenvandevelde pushed a commit to koenvandevelde/scikit-learn that referenced this issue Jul 12, 2019

ENH add n_components kwarg to SpectralClustering. See scikit-learn#13698

7a9653d

(scikit-learn#13726)

scikit-learn / scikit-learn Public

[MRG] Fix n_components kwarg missing in SpectralClustering. See #13698 #13726

[MRG] Fix n_components kwarg missing in SpectralClustering. See #13698 #13726

fdas3213 commented Apr 25, 2019 •

edited by jnothman

fdas3213 commented Apr 30, 2019

thomasjpfan May 6, 2019

fdas3213 May 7, 2019

fdas3213 commented May 7, 2019

thomasjpfan commented May 7, 2019

fdas3213 commented May 8, 2019

thomasjpfan commented May 8, 2019

fdas3213 commented May 8, 2019 •

edited

fdas3213 commented May 12, 2019

thomasjpfan May 13, 2019

fdas3213 commented May 14, 2019

jnothman left a comment

jnothman May 21, 2019

fdas3213 May 21, 2019

jnothman May 22, 2019

fdas3213 May 22, 2019

jnothman commented May 22, 2019

fdas3213 commented May 23, 2019

jnothman left a comment

fdas3213 commented May 24, 2019

jnothman commented May 25, 2019

NicolasHug left a comment

jnothman May 26, 2019

fdas3213 May 27, 2019

	n_components : integer, optional, default is n_clusters
	n_components : integer, optional, default=n_clusters

scikit-learn / scikit-learn Public

[MRG] Fix n_components kwarg missing in SpectralClustering. See #13698 #13726

[MRG] Fix n_components kwarg missing in SpectralClustering. See #13698 #13726

Conversation

fdas3213 commented Apr 25, 2019 • edited by jnothman

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

fdas3213 commented Apr 30, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fdas3213 commented May 7, 2019

thomasjpfan commented May 7, 2019

fdas3213 commented May 8, 2019

thomasjpfan commented May 8, 2019

fdas3213 commented May 8, 2019 • edited

fdas3213 commented May 12, 2019

Choose a reason for hiding this comment

fdas3213 commented May 14, 2019

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnothman commented May 22, 2019

fdas3213 commented May 23, 2019

jnothman left a comment

fdas3213 commented May 24, 2019

jnothman commented May 25, 2019

NicolasHug left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fdas3213 commented Apr 25, 2019 •

edited by jnothman

fdas3213 commented May 8, 2019 •

edited