Skip to content
This repository has been archived by the owner on Dec 6, 2023. It is now read-only.

[MRG] estimated step size for SAG* objects #42

Merged
merged 10 commits into from
Dec 11, 2015

Conversation

fabianp
Copy link
Member

@fabianp fabianp commented Dec 3, 2015

This estimates the step size from the data instead of relying on the user having to specify a step size. The result is that it converges for many cases for which the default step size will not converge, such as multiplying the 20news dataset by 3.

Some remarks:

  • The modified Huber loss is not Lipschitz-smooth so the step size is not well defined (AFAIK the theory does not guarantee convergence in this case). The step size that we propose in this case is given taking a bound on the second derivative ignoring the discontinuities, which seems to work fine in practice.
  • The hard work is done by @zermelozf, I just did cosmetic changes.

def get_max_squared_sum(X):
n = X.shape[0]
return np.max(X.dot(X.T)[range(n), range(n)])

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use np.sum(X ** 2, axis=1).max() in the dense case and csr_row_norms(X).max() in the CSR case (from sklearn.utils.sparsefuncs).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that function is available in scikit-learn 0.14, it looks like it was added in 0.15

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used your formula for the dense case at least.

Copy link
Member

@mblondel mblondel Dec 3, 2015 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mblondel
Copy link
Member

mblondel commented Dec 3, 2015

Very nice! Thanks!

@fabianp
Copy link
Member Author

fabianp commented Dec 10, 2015

I've changed the backport of get_max_squared_sum to use csr_row_norms as suggested by @mblondel . I've also updated the README to reflect this (the import is anyway lazy so lightning should at least import in older versions of sklearn).

@fabianp fabianp changed the title estimated step size for SAG* objects [MRG] estimated step size for SAG* objects Dec 10, 2015
And bump required version to 0.15
@mblondel
Copy link
Member

Merging, thank you very much!
We can add this for SVRG too but this can be done in another PR.

mblondel added a commit that referenced this pull request Dec 11, 2015
[MRG] estimated step size for SAG* objects
@mblondel mblondel merged commit 4a7fa54 into scikit-learn-contrib:master Dec 11, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants