WIP: AD stuff

[willtebbutt]

Work in progress PR to extent DiffRules to provide useful AD-related functionality. The scope of the additions include:

• A collection of forward- and reverse-mode sensitivity implementations. These will comprise the existing scalar implementations and linear algebra optimisations for both modes.
• New syntax for specifying sensitivities.

Initial commit:

• Refactor @define_diffrule macro to improve readability using MacroTools.
• Define initial interface for adding forward sensitivity implementations.
• Construct forward rules from existing diff rules.

ToDo:

• reverse-mode rules
• wrt API. See Supporting Linear Algebraic Primitives #10. Requires a more structured representation of the rules than I currently provide.
• Add linear algebra sensitivities from DiffLinearAlgebra.jl

Unknowns:

• specialisations for binary forward rules for when a sensitivity is provided only wrt one of the arguments. This requires a bit more structure in the sensitivity definitions. Is this necessary though? Could we just assume that zeros will be constant-propagated and rely on downstream packages to handle this appropriately?

[willtebbutt]

PR is in reasonable shape now, so probably time for a review before polishing. Things to address:

• Do we want to bump the require to 1.0.0? Tests are failing on 0.6. Do we care?
• The coverage appears to have plummeted, but I have no idea why... also not sure if we care.

[jrevels]

Sweet, I'll try to review this this week. Also interested in thoughts from @MikeInnes and @dfdx, if they have any.

Do we want to bump the require to 1.0.0? Tests are failing on 0.6. Do we care?

Nah, we can bump to 1.0 whenever.

The coverage appears to have plummeted, but I have no idea why... also not sure if we care.

Would be good to know why but not essential at the moemnt

[willtebbutt]

Most recent PR begins porting stuff over from DiffLinearAlgebra. See src/reverse/generic.jl for examples that (might) actually work (they're not tested yet).

[ararslan]

What's the status of this PR? Is this change still relevant going forward in the current Julia AD landscape? If so, I can try to help get it cleaned up.

[willtebbutt]

This is PR is basically dead given ChainRules.jl as far as I'm concerned. I'm happy for this to be closed.