Autograd Basics

Page Maintainers: @alband, @soulitzer

Scope

• Understand how backpropagation works in theory
• Understand how to derive backward formulas and how to add a backward formula to an operator
• Understand what a composite autograd operators is and when it is useful
• Know when to use gradcheck and custom autograd Functions
• (optional) Understand how the autograd graph gets built and executed

Introduction to backpropagation

Read through link.

Given an operator, how do I derive a backward formula for it?

• How to derive a simple formula: torch.sin link.
• How to derive a more advanced formula: torch.mm link.

Given a new operator, how do I write a new backward formula? (using derivatives.yaml)

Coming soon!

When should I write a new backward formula?

Coming soon!

How do I test an autograd formula?

Coming soon!

What are custom autograd functions?

Coming soon!

Try out the Autograd Onboarding Lab

https://github.com/pytorch/pytorch/wiki/Autograd-Onboarding-Lab