An Intellectual History of Automatic Differentiation traces the research surrounding a collection of techniques for computing derivatives without using either approximation or the manipulation of subscript-filled equations used to terrorize high school students. While its simplicity gives this method the mystery of "deep magic," it has its roots in work on differential equations in the late 19th century; inspired Alonzo Church's discovery of the untyped lambda calculus; influenced the development of functional programming, concurrency, and Unix in the 1970s; and has been recently rediscovered with applications to type theory, modelling stochastic processes, and training recurrent neural networks.
References:
- On Operators in Physical Mathematics, Oliver Heaviside, 1893
- The Calculi of Lambda Conversion, Alonzo Church, 1941
- A Simple Automatic Derivative Evaluation Program, Robert Edwin Wengert, 1964
- Computer Aided Manipulation of Symbols, Fred McBride 1971
- Coroutines and Networks of Parallel Processes, Gilles Kahn & David MacQueen, 1977
- Doug McIlroy:
- Squinting at Power Series, 1989
- Power Series, Power Serious, 1998
- The Music of Streams, 2001
- Jerzy Karczmarczuk:
- Generating Power of Lazy Semantics, 1997
- Functional Coding of Differential Forms, 1999
- Functional Differentiation of Computer Programs, 2000
- Adjoint Codes in Functional Framework, 2000
- Lazy time reversal, and automatic differentiation, 2002
- Barak Pearlmutter & Jeffrey Siskind:
- Perturbation Confusion and Referential Transparency: Correct Functional Implementation of Forward-Mode AD, 2005
- Lazy Multivariate Higher-Order Forward-Mode AD, 2007
- Nesting Forward-Mode AD in a Functional Framework, 2007
- Reverse-Mode AD in a Functional Framework: Lambda the Ultimate Backpropagator, 2008
- Efficient Implementation of a Higher-Order Language with Built-In AD, 2016
- Calculus in Coinductive Form, Pavlovic & Escardo 1998
- Structure and Interpretation of Classical Mechanics, Sussman & Wisdom, 1999
- The Differential Lambda-Calculus, Ehrhard & Regnier, 2001
- The Derivative of a Regular Type is its Type of One-Hole Contexts, Conor McBride, 2001
- Smoking Adjoints: Fast Monte Carlo Greeks, Giles & Glasserman, 2006
- Beautiful Differentiation, Conal Elliott, 2009
- Learning to Transduce with Unbounded Memory, Grefenstette et al (DeepMind) 2015