Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LBFGS does not work with SVI #1519

Closed
fehiepsi opened this Issue Nov 7, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@fehiepsi
Copy link
Collaborator

fehiepsi commented Nov 7, 2018

LBFGS requires us define a closure to be called in its step method. Defining a closure to call it with loss_and_grads is easy but I can't figure out how to incorporate it with the way PyroOptim handles parameters (creating an optimizer for each param).

@fehiepsi

This comment has been minimized.

Copy link
Collaborator Author

fehiepsi commented Nov 13, 2018

@eb8680 will we support it? LBFGS is a very convenient optimizer for models with small dataset! I have used it to optimize a lot of models. Many of them can not be trained with SVI.

@eb8680

This comment has been minimized.

Copy link
Member

eb8680 commented Nov 13, 2018

@fehiepsi sorry, I only closed the issue because I thought it was resolved by the PR #1518.

pyro.optim.PyroOptim is a klugy wrapper for handling varying amounts or orders of parameters. I don't see how we could modify it to take a closure argument, but I also think the way you're using torch.optim in your linked example is fine - if anything, we should be encouraging users to use that idiom instead of svi.step, since it's more explicit.

@eb8680 eb8680 reopened this Nov 13, 2018

@eb8680 eb8680 added the discussion label Nov 13, 2018

@fehiepsi

This comment has been minimized.

Copy link
Collaborator Author

fehiepsi commented Nov 27, 2018

Please find a solution for this issue and discussions why such solution is not appropriate to merged in #1562.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.