Discrepancies Between Different Modes

[julmb]

While investigating #108, I came across some more discrepancies between differentiation modes.

>>> import Numeric.AD.Mode.Forward as F
>>> import Numeric.AD.Mode.Reverse as R
>>> F.diff cos 0
>>> R.diff cos 0
-0.0
0.0

>>> import Numeric.AD.Mode.Forward as F
>>> import Numeric.AD.Mode.Reverse as R
>>> let f x = log (1 - exp x)
>>> F.diff (F.diff f) 0
>>> R.diff (R.diff f) 0
-Infinity
NaN

These examples may be somewhat contrived, but given the nature of algebra, I feel like any discrepancy could probably be amplified to yield significantly different results given the right circumstances.

[julmb]

It seems like Numeric.AD.Mode.Forward and Numeric.AD.Mode.Forward.Double also yield different results sometimes:

>>> import Numeric.AD.Mode.Forward as F
>>> import Numeric.AD.Mode.Forward.Double as FD
>>> f [x, y] = x ** y
>>> xs = [0, 2]
>>> F.grad f xs
>>> FD.grad f xs
[0.0,NaN]
[0.0,0.0]

I have not looked into this too much, but I have a hunch there are some simplification rules in Forward.Double that rely on the identity forall a. 0 * a = 0, which unfortunately does not hold in the presence of IEEE floating point special values (angry).