| date | title | id |
|---|---|---|
2020-10-09 05:59:52 -0700 |
Symbolic Differentiation |
2020-10-09t12-59-52z |
Symbolic Differentiation is a method for computing the derivative of a closed form expression by making use of standard hard coded differentiation rules. It is basically a computer-automated version of manual differentiation, i.e. what humans do when they differentiation.
While symbolic differentiation does not suffer from the truncation and rounding errors present in numerical differentiation, it is still not ideal. This is because symbolic differentiation will suffer from expression swell, where computed derivatives become increasingly long and inefficient (i.e. repeating calculations) because of the nature of rules such as the product rule.
This issue is particularly problematic in Deep Learning, where the Soft ReLU function, which suffers greatly from expression swell, is widely used.
Symbolic Differentiation also is limited by the fact that it can only work on closed-form expressions, meaning that it cannot be used in the presence of control flow mechanisms such as loops and conditionals.