Extend and fix AD tests

[devmotion]

This PR extends the AD tests. It shows that on the considered problem finite differencing and forward AD yield (roughly) the same gradients, Jacobians, and Hessians if the delay length is not parameterized and the solution is continuous at the initial time point. This is actually somewhat reasonable and expected since in both cases the forward sensitivity equations have jump discontinuities at the time points with C1-discontinuities in the original DDE system since the location of these C1-discontinuities depends on the parameters (see, e.g., https://epubs.siam.org/doi/abs/10.1137/100814949).

[ChrisRackauckas]

This explains a ton, thanks

[ChrisRackauckas]

@Vaibhavdixit02

[devmotion]

We should really add the forward + adjoint sensitivity implementation to DiffEqSensitivity. Actually in my master thesis I used SymPy and a ridiculous amount of metaprogramming (I feel terrible about it nowadays 😄) to implement the forward sensitivity equations somewhat efficiently (IIRC), it would just be so much nicer to have a general implementation.

[devmotion]

BTW I could start working on an implementation, I'm just not familiar with the package and its design yet, so I might need some general advice or guidance in the beginning (or at least some time for studying the code base).

[ChrisRackauckas]

Yes, open an issue? https://github.com/SciML/DiffEqSensitivity.jl I've been meaning to get to add an issue for it. We just did SDEs (https://frankschae.github.io/post/gsoc2020-high-weak-order-solvers-sde-adjoints/), so DDEs are next.

Is there a paper on adjoint methods for DDEs?

[devmotion]

I would assume that it's possible to make use of the existing implementations for ODEs?

Regarding the adjoints, some references were mentioned in one of the papers I studied. I'll look it up.

[ChrisRackauckas]

It probably just needs callbacks for the discontinuities.