Documentation: Move logging example outside gradient block #1240
Conversation
Placing the logging callback inside the gradient block means Zygote will try to differentiate it. That gradient information is unlikely to be useful and likely to cause errors (especially about mutating arrays).
oxinabox
left a comment
oxinabox
left a comment
There was a problem hiding this comment.
While there are other options like ignore() do ...
this one has the key advantage that the PR is currently open and ready to be merged.
Thanks!
|
We could also add custom loop using function my_custom_train!(loss, ps, data, opt)
ps = Params(ps)
for d in data
train_loss, back = Zygote.pullback(() -> loss(d...), ps)
gs = back(1f0)
update!(opt, ps, gs)
end
end |
|
This looks like an all-around nicer syntax, although it would need some more explanation (pullback is not mentioned anywhere else in the Flux documentation). If you want to switch to this syntax, I think I should close this PR and try again with this technique (after I read about it some myself). |
Instead of switching syntax, we can just add another example using |
It might no be clear what should be returned after adding code here, make the return explicit.
This example achieves the same effect with the pullback() method.
|
OK, another try! How does this look? |
|
Looks good! I think we can add a pointer to Zygote's pullback documentation https://fluxml.ai/Zygote.jl/dev/adjoints/#Pullbacks-1 and we are done |
|
The link is in there, at the first mention of pullback on line 162. Thanks for looking this over! |
right. Thanks for this PR! bors r+ |
|
Build succeeded: |
3 participants
Placing the logging callback inside the gradient block means Zygote will try to differentiate it. That gradient information is unlikely to be useful and likely to cause errors (especially about mutating arrays).
PR Checklist
@MikeInnesor@dhairyagandhi96(for API changes).