Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] New Callback event, before and after backward #3644

Merged
merged 5 commits into from
May 11, 2022

Conversation

muellerzr
Copy link
Contributor

This PR introduces a new Callback event: after_backward. This is designed so custom backward pass implementations can be used (such as accelerate).

after_backwards is the same as before_step, but I opted to keep both because there isn't a clean way to combine the two into one without introducing too many changes.

If a callback implements their own backward pass, they just need to attach a RaiseBackwardException in the implementation.

cc @jph00

@muellerzr muellerzr requested a review from jph00 as a code owner May 11, 2022 17:27
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@muellerzr
Copy link
Contributor Author

ci failed unrelated to the current notebook

Copy link
Member

@jph00 jph00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you need to re-run this after fixing your cuda env

nbs/13a_learner.ipynb Outdated Show resolved Hide resolved
nbs/13a_learner.ipynb Outdated Show resolved Hide resolved
@jph00 jph00 merged commit eed715b into fastai:master May 11, 2022
@warner-benjamin
Copy link
Collaborator

Could you update the Callback documentation to mention after_backward and what it is used for? Currently it reads:

- `before_backward`: called after the loss has been computed, but only in training mode (i.e. when the backward pass will be used)
- `before_step`: called after the backward pass, but before the update of the parameters. It can be used to do any change to the gradients before said update (gradient clipping for instance).

@muellerzr
Copy link
Contributor Author

muellerzr commented May 11, 2022

@warner-benjamin after_backward can be synonymous with before_step. But everyone should use before_step really, hence why it was left unmentioned.

@warner-benjamin
Copy link
Collaborator

That strikes me as an important thing to mention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants