[DEV] Adding julia solvers: GD, SGD, SAGA

[ngazagna]

No description provided.

[agramfort]

@ngazagna can you show here the figures you obtain?

[tomMoral]

Thanks a lot @ngazagna for moving the PR! 😃 I will take a look ASAP.

It seems that the tests are not run in PRs, we also need to fix the CI.

[ngazagna]

Sorry, I just had time to open the PR without testing it. Please find hereafter the plots obtained after running

benchopt run ./benchmark_logreg_l2

[agramfort]

it seems there is a bug as the objective function should reach the same values.

…

[ngazagna]

There is indeed a problem with sklearn[lbfgs]. I think it is due to the lack of homogeneity in the tolerances used in sklearn solvers.
When passing solver='lbfgs to an sklearn LogisticRegression model, it runs this scipy optimize.minimize(method=’L-BFGS-B’) function, with gtol=tol which stops when

max{|proj g_i | i = 1, ..., n} <= gtol

where pg_i is the i-th component of the projected gradient.
Whereas, for instance with solver=newton-cg, then stops when

max{|g_i | i = 1, ..., n} <= tol

where g_i is the i-th component of the gradient.

EDIT: I guess this post is relevant

[agramfort]

it does not explain why it does not reach the same optimal value. tol are sampled independently for all solvers

…

[tomMoral]

could you push a ci trigger commit to make the test run on the PR?

git commit --allow-empty -m 'CI trigger'
git push

I guess alex is referring to the second plot where GD/SGd/SAGA seem to plateau to a different value.

[tomMoral]

@ngazagna I just did an update to fix the failing test so don't forget to use git pull before doing more modifications.

I will try to run the bench and see if I can see why this seems to not converge toward the same value.

[tomMoral]

Relies on benchopt/benchopt#78 for julia dependencies and pre_install_hook functionalities.

[tomMoral]

It seems there is no more technical bug :)

now, the failing test is because SGD does not find the optimal point (which is expected).
Maybe we should set some flag to detect stochastic solvers and skip this test?

[agramfort]

It seems there is no more technical bug :) now, the failing test is because SGD does not find the optimal point (which is expected). Maybe we should set some flag to detect stochastic solvers and skip this test?

don't try to be too smart. If you really want this I would add an attribute to solver class saying if a test can be skipped

[tomMoral]

that was my idea of a flag, something like `approximate_solver` that skip the optimality test. Le ven. 2 oct. 2020 à 08:52, Alexandre Gramfort <notifications@github.com> a écrit :

…

> > It seems there is no more technical bug :) > > now, the failing test is because SGD does not find the optimal point > (which is expected). > Maybe we should set some flag to detect stochastic solvers and skip this > test? > don't try to be too smart. If you really want this I would add an attribute to solver class saying if a test can be skipped — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZKZ6PFEHXYI5R3CXK6Q53SIV2CJANCNFSM4R6BRNNA> .

[ngazagna]

could you push a ci trigger commit to make the test run on the PR?

git commit --allow-empty -m 'CI trigger'
git push

I guess alex is referring to the second plot where GD/SGd/SAGA seem to plateau to a different value.

Sorry, I did not get that Alex was pointing an issue related to SGD/SAGA. I guess we will face this problem as soon as algorithms need parameter tuning and when time in --timeout t is too small.

The other (expensive) option is do an extensive grid search (instead of the theoretical step size used here), try and retry until SGD converges.

[agramfort]

forget about SGD. Julia-SAGA is not clear to converge either. Maybe not enough samples for this benchmark?

…