Cases of creation of `NonlinearSystem`s have been broken

[TorkelE]

This used to work, but doesn't anymore:

using ModelingToolkit, OrdinaryDiffEq, Test
using ModelingToolkit: t_nounits as t, D_nounits as D

# Defines the model.
@parameters k1 k2 V0
@variables X1(t) X2(t) Y1(t) Y2(t) V(t) = V0 W(t)
@parameters v = V w = W Γ[1:2] = missing [guess = [1.0, 1.0]]
eqs = [
    0 ~ k1 * (Γ[1] - X1) - k2 * X1,
    0 ~ k1 * (Γ[2] - Y1) - k2 * Y1,
    0 ~ 1 - V,
    0 ~ 1 - W,
]
observed = [X2 ~ Γ[1] - X1, Y2 ~ Γ[2] - Y1]
@mtkbuild osys = ODESystem(eqs, t, [X1, X2, Y1, Y2, V, W], [k1, k2, V0, v, w, Γ]; observed)


@mtkbuild nlsys = NonlinearSystem(eqs, [X1, X2, Y1, Y2, V, W], [k1, k2, V0, v, w, Γ]; observed)


# Create the`ODEProblem`s.
u0 = [X1 => 1.0, X2 => 2.0, Y1 => 3.0, Y2 => 4.0, V => 5.0, W => 6.0]
ps = [k1 => 0.1, k2 => 0.2, V0 => 3.0]
nlprob = NonlinearProblem(nlsys, u0, ps);

' This works
oprob = ODEProblem(osys, u0, 10.0, ps);

[AayushSabharwal]

Supporting NonlinearProblem(::ODESystem) is fairly trivial and I have it working locally. However, the MWE expects Gamma to be solved for and doesn't include equations to solve for it. NonlinearProblem only runs parameter initialization, and thus observed equations are not included in initialization.

[TorkelE]

Actually just the creation of the NonlinearProblem from the NonlinearSystem fails here:

nlprob = NonlinearProblem(nlsys, u0, ps);

ERROR: MethodError: Cannot `convert` an object of type Missing to an object of type Real
The function `convert` exists, but no method is defined for this combination of argument types.

[AayushSabharwal]

That's what I mentioned

NonlinearProblem only runs parameter initialization, and thus observed equations are not included in initialization.

It can't solve for Gamma, because there is no parameter equation solving for Gamma. I do see a way to do it by turning X1 into Initial(X1) but will have to draw up a proper plan for making this work.

[TorkelE]

I thought the idea was that the observed equations could be used to solve for Gamma, given that we knew the other parts? I.e. previously we would (and still can) encode default values for Gamma which are used to compute these from the initial conditions. Then, the observed are used to compute X1 and Y2. However, with the recent update the idea is that we no longer provide the default values?

This all works for ODEs, right, so there is some part of the update where we lost NonlinearSystem support?

[AayushSabharwal]

I thought the idea was that the observed equations could be used to solve for Gamma, given that we knew the other parts?

For time-dependent systems, yes. However, running initialization to solve for x doesn't make sense in a NonlinearSystem whose sole purpose is to solve for x. Thus, it runs only parameter initialization. Since it excludes variables, it consequently excludes observed equations from initialization. We didn't "lose" nonlinearsystem support, we fixed a pretty big bug which was causing solves to fail. Currently, we only support a limited form of parameter initialization in nonlinear systems. This will be expanded to handle the case in the issue.

[TorkelE]

Right, so practically, how should we handle this in Catalyst? I.e. what kind of information should we/should not provide? Do correct me if I am wrong, but at least the cases of this that we had in our tests suite in Catalyst used to work without problems.

[AayushSabharwal]

Yeah this would have worked for some time, but it was incredibly broken and thus we hotfixed it out. For time-independent systems, initialization only solves for parameters using parameter dependencies and initialization_eqs. The latter are assumed to be parameter-only equations. We don't validate this but really should.

[TorkelE]

Well, I am glad stuff got fixed at least!

Do you have any idea of when this might be available again?

[AayushSabharwal]

I can't promise a timeline, unfortunately. I'm aware of the issue, but am currently focused on fixing a plethora of CI issues across SciML.

[isaacsas]

So should we make a separate issue for the changes discussed in the last few comments here that mean we can't use conservation law elimination with NonlinearSystems? That is blocker for making a new Catalyst release.