Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/module no grad #274

Merged
merged 9 commits into from
Apr 7, 2023
Merged

Feat/module no grad #274

merged 9 commits into from
Apr 7, 2023

Conversation

nathanielsimard
Copy link
Member

@nathanielsimard nathanielsimard commented Apr 6, 2023

The ability to dynamically set the gradient requirements for modules has been added. Previously, it was assumed that every parameter required gradients, which resulted in resetting the graph after each operation at the module level. This was problematic because we might want to execute operations on modules while still tracking operations. To address this, the fork function has been added to modules, which indicates when we want to send a module to another device with an empty graph.

The detach function has been removed from Module because it can lead to mistakes. Instead of using to_device().detach().require_grad(), which would override previous set_require_grad(false), we can call fork, which preserves gradient settings set when creating a module.

The from_inner function has been removed from ADModule. A static method cannot keep track of dynamic gradient configurations. Instead, the function inner(self) has been changed to inner(&self). Additionally, inner may be renamed to valid since it highlights the purpose of the method and is familiar to PyTorch users.

@nathanielsimard nathanielsimard merged commit f04fe10 into main Apr 7, 2023
@nathanielsimard nathanielsimard deleted the feat/module-no-grad branch April 7, 2023 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants