Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite tests #23

Merged
merged 8 commits into from Mar 30, 2023
Merged

Rewrite tests #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f9d3668
Add seed in landscape plot
mBarreau Mar 9, 2023
a7e1d42
Update README.md
mBarreau Mar 9, 2023
ae1fbf9
Remove deprecation warnings with the latest version of Flux
mBarreau Mar 15, 2023
32fbc75
Merge branch 'baggepinnen:master' into fix_deprecation
mBarreau Mar 15, 2023
8d887be
Refactor tests and add time benchmarking
mBarreau Mar 21, 2023
6ca04e7
Add refresh and time benchmark figure
mBarreau Mar 21, 2023
0e0f8b2
Tune LS-BFGS
mBarreau Mar 22, 2023
a40b8fb
Merge branch 'master' into new_tests
mBarreau Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion Project.toml
View file
Open in desktop
Expand Up @@ -26,7 +26,8 @@ julia = "1.6"

[extras]
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
StatsPlots = "f3b207a7-027a-5e70-b257-86293d7955fd"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test", "Plots"]
test = ["Test", "Plots", "StatsPlots"]
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Expand Up @@ -19,6 +19,9 @@ res = Optim.optimize(Optim.only_fg!(fg!), p0, Optim.Options(iterations=1000, sto
The utility provided by this package is the function `optfuns` which returns three functions and `p0`, a vectorized version of `pars`. BFGS typically has better convergence properties than, e.g., the ADAM optimizer. Here's a benchmark where BFGS in red beats ADAGrad with tuned step size in blue, and a [stochastic L-BFGS [1]](https://arxiv.org/abs/1802.04310) ([implemented](https://github.com/baggepinnen/FluxOptTools.jl/blob/master/src/SLBFGS.jl) in this repository) in green performs somewhere in between.
![losses](figs/losses.svg)

From a computational time perspective, S-LBFGS is about 2 times slower than ADAM (with additionnal memory complexity) while the traditional L-BFGS algorithm is around 3 times slower than ADAM (but similar memory burden as SL-BFGS).
![times](figs/times.svg)

The code for this benchmark is in the `runtests.jl`.

## Visualize loss landscape
Expand Down