-
Notifications
You must be signed in to change notification settings - Fork 140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possible second argument to input function #77
Comments
Isn't this handled by closures already? |
True but in iterative algorithms where the second argument changes, the closure has to be redefined in each step/iteration, and this is not always the best design for extraneous reasons. |
The closure "updates" together with the captured variables. There is no need to redefine it. |
That's good to know. At the same time it is good to allow for extra dummy input arguments (dummy from the autodiff perspective) as it would enable more diverse development. For example I have separated parameters (methods) from states (data), so my software engineering approach is conceptually orthogonal to closures. |
I still don't understand why closures doesn't solve this. If you have a function |
The issue of allowing additional arguments to function was discussed in #32. It was also concluded that closures do indeed are a solution. A potential worry is that creating many closure --- for instance in |
Closing as duplicate of #32. @scidom Feel free to reopen if you think this needs to be revisited, but I really think closures are the best way to do this. If it helps to approach it a different way, remember that ForwardDiff now supports differentiating callable types. |
Thanks @jrevels. I wanted to kindly ask if it would be possible to add this as a feature request. I understand that closures is a possible solution, but I want as a programmer and ForwardDiff user to have the alternative option of inputing a function with more than one arguments for two reasons, to avoid any small performance penalty (however small it may be, no penalty is better than small penalty), and most importantly because it may be helpful to provide an alternative usage case/programming paradigm. If of course the answer is that it is not possible, then it is ok, but wanted to make the effort and ask for this feature. |
I tried the following:
It seems that actually f is a tiny bit faster than g, i.e. closures are a tiny bit faster, is this possible? |
Now you are using a global variable. That is not the same as creating an closure inside another function that captures function local variables. |
Consider what it would be like if we DID have this feature - I think the package would actually be worse off. Here are some reasons why:
|
Try this
|
You will not get any useful data by benchmarking functions accessing global variables. |
Thanks for the feedback @KristofferC and for the snippet @gragusa. @jrevels I can see your point about complexity of development. My primary criterion during development tends to be performance. Along these lines, to quantify performance, shall we try to think of a simple benchmark, without using globals as @KristofferC pointed, just to see what is the penalty of closures if any? It is hard for all of us to know the pros and cons without measuring them... I will try to think of a reasonable example, not necessarily related to ForwardDiff. |
You can always enclose the above into a function. But why accessing global variables should be different for closures? |
@gragusa you mean sth like the example below?
This still uses globals, plus we redefine the closure in each iteration, so we may want to improve on these. |
@scidom I was thinking more along the lines of
This is the worst case scenario --- when closures need to be created at each iteration. Think about MCMC within Gibbs: at each iteration you have to create a closure enclosing the blocks of variables. |
Global variables can change type at any time and must therefore be boxed. Local variables can be reasoned about. This does not change when using closures. Regarding performance. We are already passing in a function as argument to ForwardDiff which has its own overhead so the closure overhead should be amortized. For best performance, create a functor (type which overloads Base.call) and pass that to ForwardDiff. |
Yes, this Gibbs scenario is exactly the situation I am dealing with 👍 |
@KristofferC Yes, I understand the boxing. The fact that "this does not change when using closures" is what I thought --- and so the benchmarking in this case is not misleading because both functions deal with global in the same way. |
@gragusa I tried
and there is massive gap in performance. Performance with Gibbs sampling is even worse than our toy benchmark because each Gibbs iteration requires redefining several closures - pretty much as many as the number of sampled parameters. @KristofferC can you provide a toy example of what you meant about overloading |
See https://github.com/JuliaLang/julia/blob/master/base/functors.jl This types could also hold states. When passing these as arguments julia can do inlining and all that jazz. |
Thanks, I also found the following in the documentation: http://docs.julialang.org/en/release-0.4/manual/methods/#call-overloading-and-function-like-objects |
Also what is interesting is not the cost of the closure vs calling a normal function but the cost of passing a closure vs a function + arguments to ForwardDiff. My gut says there will be very little difference. I'm in bed now so can't test it. |
Here's a concrete example of making a callable type and using it with ForwardDiff: julia> immutable Foo{T}
x::T
y::T
end
julia> call(f::Foo, a) = a^(f.x) + (a / f.y)
call (generic function with 1033 methods)
julia> f = Foo(3, 2)
Foo{Int64}(3,2)
julia> f(2.5)
16.875
julia> using ForwardDiff
julia> ForwardDiff.derivative(f, 2.5)
19.25 As @KristofferC said, you can use these types instead of closures. Any time where you would make a new closure to update the values, you can instead make a new instance of one of these types. If your type is mutable, you could just update the fields as you go instead of making new instances (though keeping it immutable could work better if you're storing AFAIK, the plan is for closures to start using this callable type strategy "under the hood" in v0.5 (see JuliaLang/julia#13412). |
That's amazing, I must say. I am feeling a bit puzzled though in two respects; i) in your example |
Actually, I found the answer to the second question, I think |
I am trying to understand how this would work in practice, if I would need to introduce a new log-target type and make it callable, too tired to think clearly now, tomorrow again... |
I wanted to add 2 more thoughts. Firstly, the functors are a great idea but (as any tool) they are not the proper means for every situation. More specifically, consider the case of having a functor The second remark relates to the comment made by @KristofferC; I wrote a simple example for which in one case we pass a closure
The only problem of course is development-related, i.e. having one version of user-defined function |
There is no reason why you couldn't make two separate instances of a functor hold references to the same vector. Regarding recreating the closure, why not do something like this: function f()
n = 1000000
a = rand(n)
x = rand(n)
function g(x, y)
x+y
end
@time for i in 1:n
cl(x) = g(x, a[i])
cl(x[i])
end
a_val = 0.0
cl2(x) = g(x, a_val)
@time for i in 1:n
a_val = a[i]
cl2(x[i])
end
@time for i in 1:n
g(x[i], a[i])
end
end Since the captured variable updates there is no need to recreate the closure unless you will change the type of captured variable in every iteration (seems unlikely). |
I agree @KristofferC, there is no need to redefine the closure. The only difficulty is that I want to be able to define method fields in a parameter type independently of the enclosed value, which means that I need to delay the (one and only) definition of the closure until I know the delayed value. About functors, you are actually right. For example the following works (therefore I will consider more seriously the possibility of using functors):
|
P.S. I find the functor solution quite ingenious the more I think about it. |
Note that in the (not so far away) future, the plan for julia is for closures to basically lower into their functor equivalent version making them just as performant. |
This is great. I find functors more handy than closures in Lora because they are more low level exactly, which makes dev easier. |
After more thinking and an attempt to implement the idea, it turns out functors won't work unfortunately for the application at hand. The problem is that the definition |
After having spent a good deal of time thinking over this problem and after having tried to tackle it in practice, I would like to share my thoughts and to make a case for adding the extra feature of a I am convinced by now that there is no performance penalty when using closures, so performance is not a concern anymore. The reason I think the Functors are indeed a great solution but the main issue, as explained above, is that the definition of Closures are great, but in a larger scale project of a "real-life" application they can become impractical. For instance, if we were to ask the user to pass just one closure to The bottom line is that there is no performance consideration, but there are important use cases where it becomes less natural and more programmatically impractical to request the user to define a closure. Along these lines, I want to kindly re-open issue #32 and consider adding the |
@scidom, could you explain what you mean by this? |
I will try to explain using a fictitious toy example:
This works fine, but the problem is that I may want to define more than one priors. Since multiple dispatch for
for another prior... |
I could code-generate many functors for different priors with |
What would the |
I'm not sure how the issue is any different in that case |
I was thinking that if we have a function say
or perhaps
then this may allow passing a function |
To explain my problem, if I ask the user to provide a closure |
The other solution is that I ask the user to provide |
The functor object can just store the user-provided function and the arguments. |
All the solver packages, and optimization packages seems to do just fine with only closures so I am sure there should be some way to solve this. Regarding extra arguments in ForwardDiff. My first thought is that allowing 1 extra argument is completely arbitrary. What if you want to pass multiple extra arguments? The only way you can do that in a type stable way is to write a tailor made type for your specific function. This means that you suddenly impose on the user to always write their functions as f(x, extra_parameters), instead of letting them write them however they want and just ask them to pass the closure to the library. They also have to deal with the extra boilerplate of defining a new type for each function.
Isn't that pretty much exactly what a closure is? |
@KristofferC, I agree, one would need to generalize the way @mlubin your solution works for me, it is really great, thank you:
I will then close #32 and will use the proposed solution (and yes, @KristofferC, I presume this functor definition is pretty much conceptually equivalent to a closure). |
One more thing, the reason this lower-level functor approach works better for me than just a closure is because I can then define the constructor (using pseudocode below)
so the user doesn't need to define the dummy variable |
I may be doing sth wrong (highly likely due to being late), but I get an error when I try to use ForwardDiff with the above solution:
Although
|
So with current version how can I use additional arguments to function being differentiated? |
Same as before. Pass a closure as your function to be differentiated where it closes over your parameters. |
With Call overloading? Okay I tried like this.. it's working.
|
Call overloading is not needed anymore for performance. It is easier and more flexible to create a closure. Read through this thread and you will find examples. I am on mobile so can't post a full code example. |
Oh.. you mean like this?
|
As I am trying to connect ForwardDiff with Lora, I came to realise that this can work in one of two possible cases, namely when the log-target function has a signature such as
logtarget(x::Vector)
. There is a second case, however, which is perhaps more common and interesting in real applications; in this latter case, the log-target takes a second argument, which may be a cell array (or dictionary, but let's say cell array for now) of auxiliary variables.The main question is whether we can allow ForwardDiff to work with functions with signature
logtarget(x::Vector, y::Vector)
. In principle, ForwardDiff will carry on working the same way by performing autodiff with respect to x only, i.e. by ignoring any subsequent input argument after the first one. y will only appear in the body of logtarget to pass additional values in the body of log-target, but as far as autodiff goes y will be a constant.All it takes to add this extra feature is a slight change of API allowing extra arguments that won't change the course of differentiation. Do you think we could possibly deal with this?
The text was updated successfully, but these errors were encountered: