Use finite-differences where forward-mode not implementable

[andrjohns]

Description

As discussed over in #2839, the ODE methods are not fvar<T> compatible, and it would be a significant amount of work to do so. This means that any downstream methods depending on fvar<> or higher-order autodiff will fail to compile, a current example of this is the new $hessian() method in cmdstanr.

In these instances it would be great to have a method that "works", even if it's slow - better to have in a limited form than to not have at all (imo).

Boost Math has existing optimised routines for finite-differencing, including the use of the complex step approximation for any complex-compatible functions (allowing for estimating the derivative with a single function evaluation)

Current Version:

v4.4.0

[WardBrian]

Here's a list of functions exposed in the Stan language which would be candidates to use with stan/math/fwd/functor/finite_diff.hpp:

• integrate_1d (Framework for generic fvar<T> support through finite-differences #2929)
• map_rect
• reduce_sum and reduce_sum_static
• ode_bdf_tol, ode_rk45_tol, ode_adams_tol, ode_bdf, ode_rk45, ode_adams, ode_ckrk, ode_ckrk_tol, ode_adjoint_tol_ctl (I think doing these would automatically add support for the historical integrate_ode_* variants?)
• dae and dae_tol
• solve_newton, solve_newton_tol, solve_powell, and solve_powell_tol (Again, I am hoping that algebra_solver_* would come for free if we did these)