Ito sense for SDE adjoints

[frankschae]

@ChrisRackauckas Could you check if the way to add the Ito sense goes in the right direction?

I thought an additional type for the transformed function with its own vips are the best way to do it. However, I couldn't resolve up to now all errors in the vjps from the SensitivityFunction. (ReverseDiff is the only one which I could make running through the code but also there the gradients look slightly off.. )

[frankschae]

All tests in sde_transformation_test.jl are passing. So the vips are doing the proper transformation on the drift term.

[frankschae]

Not sure if there is still one conceptual issue (maybe related to the SOSRI issue in SciML/DiffEqNoiseProcess.jl#62)

The reversion using EM for the 1D Ito SDE should be okay according to what I saw in: SciML/DiffEqNoiseProcess.jl#62

These tests are also all fine:

res_sde_u0a, res_sde_pa = adjoint_sensitivities(sol,EM(),dg!,Array(t)
  ,dt=dtscalar,adaptive=false,sensealg=BacksolveAdjoint(autojacvec=DiffEqSensitivity.ReverseDiffVJP()))

@info res_sde_u0a, res_sde_pa

res_sde_u0b, res_sde_pb = adjoint_sensitivities(sol,EM(),dg!,Array(t)
  ,dt=dtscalar,adaptive=false,sensealg=BacksolveAdjoint(autojacvec=false))

@info res_sde_u0b, res_sde_pb

@test isapprox(res_sde_u0a, res_sde_u0b, rtol=1e-9)
@test isapprox(res_sde_pa, res_sde_pb, rtol=1e-9)

res_sde_u0b, res_sde_pb = adjoint_sensitivities(sol,EM(),dg!,Array(t)
  ,dt=dtscalar,adaptive=false,sensealg=BacksolveAdjoint(autojacvec=false), corfunc_analytical=corfunc)

@info res_sde_u0b, res_sde_pb

@test isapprox(res_sde_u0a, res_sde_u0b, rtol=1e-9)
@test isapprox(res_sde_pa, res_sde_pb, rtol=1e-9)

That should imply that the correction factor is correctly implemented since ReverseDiff over Zygote, ForwardDiff over Zygote, and also ForwardDiff over analytical correction factor all coincide.

However the gradients do not yet agree with the analytical / ForwardDiff result.

[frankschae]

The plot below shows the evolution of the state (top left), derivative wrt initial state (top right), derivative wrt p[1] and p[2] (bottom left and right, respectively).