Robust Singular Jacobian Handling

[avik-pal]

Current approach is:

1. Proceed as usual
2. On first linear solve failure:
   1. Construct a QRPivoted Cache and cache it
   2. Solve using Pivoted QR but warn the user that there is a potential singular Jacobian detected
   3. If successful, proceed as usual
   4. Otherwise:
      1. For "current" jacobian: stop nonlinear solve
      2. For approximate methods: follow a policy similar to line search routines
      3. For non-current exact jacobian: force a reset
3. On later linear solve failure:
   1. Goto 2.2

TODOs

• Switching to Pivoted QR
• Add tests
• Use verbose to turn off printing
• Integrate it correctly into the algorithms -- currently assumes that switching is always successful.
• How do we deal with non-standard Arrays? Basically how do we determine that PivotedQR is defined for the matrix A
• Add missing docstrings to the documentation

@ChrisRackauckas any idea how to handle this last part?

[avik-pal]

With this PR all of the following work:

using NonlinearSolve, LinearSolve

function nlls!(du, u, p)
    du[1] = 2u[1] - 2
    du[2] = (u[1] - 4u[2])^2 + 0.1
    du[3] = 0
end

u0 = [0.0, 0.0]
prob = NonlinearLeastSquaresProblem(
    NonlinearFunction(nlls!, resid_prototype = zeros(3)), u0)

solve(prob)

solve(prob, GaussNewton(; linsolve = QRFactorization()))

solve(prob, GaussNewton(; linsolve = LUFactorization()))

I always display a warning rn, but I will configure the verbose to not show the warning if needed.

[codecov]

Codecov Report

Attention: Patch coverage is 73.73737% with 26 lines in your changes are missing coverage. Please review.

Project coverage is 86.34%. Comparing base (b389d0e) to head (914f556).

Files	Patch %	Lines
src/core/approximate_jacobian.jl	28.57%	10 Missing ⚠️
src/core/generalized_first_order.jl	64.28%	5 Missing ⚠️
src/descent/steepest.jl	16.66%	5 Missing ⚠️
src/internal/linear_solve.jl	86.66%	4 Missing ⚠️
src/descent/damped_newton.jl	85.71%	1 Missing ⚠️
src/descent/newton.jl	92.85%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #414      +/-   ##
==========================================
- Coverage   86.77%   86.34%   -0.44%     
==========================================
  Files          46       47       +1     
  Lines        2851     2914      +63     
==========================================
+ Hits         2474     2516      +42     
- Misses        377      398      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[avik-pal]

Needs SciML/LinearSolve.jl#494 before the tests pass

[avik-pal]

@ChrisRackauckas do you have a suggestion on how to deal with this

How do we deal with non-standard Arrays? Basically how do we determine that PivotedQR is defined for the matrix A

Apart for this, we are good to go here.

Okay maybe we can use ArrayInterface.isstructured and AnyGPUArray to safeguard against this