Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow passing in non-optimizable parameters? #53

Closed
johnmyleswhite opened this issue Apr 11, 2014 · 11 comments
Closed

Allow passing in non-optimizable parameters? #53

johnmyleswhite opened this issue Apr 11, 2014 · 11 comments

Comments

@johnmyleswhite
Copy link
Contributor

Lately I've found myself constructing some pretty complicated closures to work around the current Optim interface.

As such, I'd like to propose a revised protocol for functions passed to Optim. Instead of working with a function f(x::Array) -> Float64, I'd like to work with functions of the form f(x::Array, extra::Any) where extra can be used to provide any additional information needed to evaluate f given x. This would make several things simpler:

  • Fitting statistical models that need to be conditioned on observed data
  • Implementing alternating minimization
  • Working with functions that need to update several buffers during evaluation

What would people think of making this kind of breaking change? It would make Optim much more general, but might also make the interface less intuitive for newcomers since you'd want to do something like f(x, nothing) when your function doesn't depend on additional information.

@StefanKarpinski
Copy link

This is exactly what closures are for, it's a shame they are still such a performance hit. I hate to see API changes like this that may become unnecessary in the relatively near future. @JeffBezanson – any thoughts on how soon it might be plausible to get faster closures?

@johnmyleswhite
Copy link
Contributor Author

I would love to have faster closures.

But I think the use case I'm currently dealing with would be problematic even with faster closures: in these applications, I randomly modify the dataset being closed over on every single iteration of my optimization algorithm, which necessitates a new function definition on every iteration if you use closures to track the current data set being optimized against. Given Julia's approach to JIT-ing functions, it's not clear to me that closures will ever be able to catch up with the interface I'm describing.

@johnmyleswhite
Copy link
Contributor Author

Since I basically need this functionality to push a project at work forward, I think I'm going to start an Optim2 package to work out how I'd like to see things.

Would love to hear more what to expect from closures in future releases.

@mlubin
Copy link
Contributor

mlubin commented Apr 12, 2014

I don't support changing the default interface to do this, it's horribly ugly, not something that I would want to put on a slide during a talk to sell Julia. What we might need is another level of abstraction over the optimization code so that both the current and the f(x,::Any) user interfaces can sit on top of it.

@StefanKarpinski
Copy link

It does strike me as not the right default interface.

@johnmyleswhite
Copy link
Contributor Author

Any suggestions?

@johnmyleswhite
Copy link
Contributor Author

If we could dispatch on the argument signature of functions, this could be solved by automatically wrapping f(x) with g(x, nothing) = f(x). But I don't know how to emulate that kind of dispatch at the moment.

@johnmyleswhite
Copy link
Contributor Author

Since I'm on the topic of things I'd like to be able to do, the other functionality I've been needing is the ability to pass in a function that gets executed after every step. That lets you do things like SGD.

@mlubin
Copy link
Contributor

mlubin commented Apr 12, 2014

I have a sense that it would be better to break apart the optimization a
bit, instead of trying to stuff everything into callbacks. I'm referring to
an interface where you can "drive" the optimization algorithm at a high
level, choosing when to iterate and providing the necessary information at
each step without needing to get into the details of how the iterations are
performed. A number of Fortran optimization codes have this interface, and
I think that something like this would provide the flexibility that you're
looking for.

@johnmyleswhite
Copy link
Contributor Author

That sounds reasonable. I should look at a few other libraries and see how they handle this kind of thing.

@johnmyleswhite
Copy link
Contributor Author

Breaking apart the functionality of optimize might be a good idea at some point, but I've started creating custom code for the two main cases that inspired this issue. Going to close now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants