Query in robertsons code in predict_neuralode function

[yewalenikhil65]

Hi @jiweiqi ,In robertson's code.. shouldn't one solve for _prob and not prob .. copying here code for the reference from
https://github.com/DENG-MIT/CRNN/blob/main/robertson/rober_crnn.jl

function predict_neuralode(u0, p; sample = datasize)
    global w_in, w_b, w_out = p2vec(p)
    _prob = remake(prob, tspan=[0, tsteps[sample]])                               # this line
    sol = solve(prob, alg, u0=u0, p=p, saveat=tsteps[1:sample],             # this line
                sensalg=sense, verbose=false, maxiters=maxiters)

``

[jiweiqi]

You are right. Although it's not a big deal here since p and u0 are correctly set in silver.

By the way, for the PBE problem I don't think you need Robertson code, I suspect it is mainly the difficulty in inference nonlinear model parameters which can be improved by better designing experiments.

Or alternatively, don't expect that the model parameters should be same as the ground truth since there is no ground truth...

[yewalenikhil65]

You are right. Although it's not a big deal here since p and u0 are correctly set in silver.

By the way, for the PBE problem I don't think you need Robertson code, I suspect it is mainly the difficulty in inference nonlinear model parameters which can be improved by better designing experiments.

Ohh no..i was trying this code again on another stiff system.. So happened to notice this robertson again.This stiff case I shall add today to same PBE repo in new folder.. Do take a look whenever you feel free..currently giving me headache becuase of ode not solved issue.. Tried increasing number of reactions to overparametrize CRNN but still no luck,

For PBE, for now I am currently satisfied with good fit even though weights haven't been learnt.

[jiweiqi]

Got it! For this Robertson code, you can pass the tspan argument into solver directly without remake the problem and it will be the same.

What exactly the error is?

[yewalenikhil65]

Got it! For this Robertson code, you can pass the tspan argument into solver directly without remake the problem and it will be the same.

What exactly the error is?

Its that Retcode returned by ODEProblem..Remade ODEProblem with updated params seem not to be solved.
println("ode solver failed")
It also arises sometimes in robertson code when its executed.. But its occasional in robertsons case.

[jiweiqi]

What's your ode solver?
It happens for stiff problem. Try a good initialization and clip gradient initially to avoid bad parameters

[yewalenikhil65]

What's your ode solver?
It happens for stiff problem. Try a good initialization and clip gradient initially to avoid bad parameters

AutoTsit5(Rosenbrock).. same as the one required for generating the data...
Will try now again with new initialisation

[jiweiqi]

The solver looks good to me, although you could try pure stiff solver, like Rosenbrock23() by removing AutoTsit5

[jiweiqi]

I have just updated the code a little bit a2 132c3e8 and you can have look. One thing to note is that Rosenbrock23(autodiff-false) was used due to historical reasons. and you can set autodiff=true now.

[yewalenikhil65]

I have just updated the code a little bit a2 132c3e8 and you can have look. One thing to note is that Rosenbrock23(autodiff-false) was used due to historical reasons. and you can set autodiff=true now.

Yes, I will definetly look at this..
Also if you get time, do take a look here https://github.com/yewalenikhil65/Application_of_CRNN/tree/main/CRNN_hockinMann
It doesnt work yet. Its highly stiff

[jiweiqi]

The hockin Mann problem looks interesting and I will also try it myself.
At a first glance, it is indeed stiff, so you might consider using the training strategies in Robertson's problem.
Pay attention to

function p2vec(p)
    slope = abs(p[end])
    w_b = @view(p[1:nr]) .* (10 * slope)


    w_in = reshape(@view(p[nr * (ns + 1) + 1:nr * (2 * ns + 1)]), ns, nr)


    w_out = reshape(@view(p[nr + 1:nr * (ns + 1)]), ns, nr)
    w_out = @. -w_in * (10 ^ w_out)


    w_in = clamp.(w_in, 0, 2.5)
    return w_in, w_b, w_out
end

[jiweiqi]

I feel (10 ^ w_out) is useful to handle the scale separation in the rate constants.

[yewalenikhil65]

The hockin Mann problem looks interesting and I will also try it myself.
At a first glance, it is indeed stiff, so you might consider using the training strategies in Robertson's problem.
Pay attention to

function p2vec(p)
    slope = abs(p[end])
    w_b = @view(p[1:nr]) .* (10 * slope)


    w_in = reshape(@view(p[nr * (ns + 1) + 1:nr * (2 * ns + 1)]), ns, nr)


    w_out = reshape(@view(p[nr + 1:nr * (ns + 1)]), ns, nr)
    w_out = @. -w_in * (10 ^ w_out)


    w_in = clamp.(w_in, 0, 2.5)
    return w_in, w_b, w_out
end

Yup.. I am tryring this..

[yewalenikhil65]

Also if you get time, do take a look here https://github.com/yewalenikhil65/Application_of_CRNN/tree/main/CRNN_hockinMann
It doesnt work yet. Its highly stiff

How would you advise to check the the training of species (all 34 of them) ? push! pushes all of them into list_plt and is good for less number of species but not for too many species

:update--resolved..: I better thought to mkdirin figsfor each species

[yewalenikhil65]

pred = clamp.(Array(solve(prob, alg, u0=u0, p=p, sensalg=sense)), -ub, ub)

Here we try to clamp whole prediction to the upper bound. But for multiscale problems I think it makes more sense to have ub as a vector of upper bounds from each species(multiscaled) data, and then clamping the prediction of each species individually.. What do you think?

[jiweiqi]

Sounds like a good idea. I haven't really played with ub a lot. In general, for the non-stiff problems, ub is not so necessary.

[yewalenikhil65]

I have just updated the code a little bit a2 132c3e8 and you can have look. One thing to note is that Rosenbrock23(autodiff-false) was used due to historical reasons. and you can set autodiff=true now.

Yes, I will definetly look at this..
Also if you get time, do take a look here https://github.com/yewalenikhil65/Application_of_CRNN/tree/main/CRNN_hockinMann
It doesnt work yet. Its highly stiff

Trying that trick of ub with this problem. Going good, but very slow as there are too many(about 1500) paramaters.

Thought of parallelizing..Did you meant to use @distributed or @threads for the for loop of choosing random experiment among n_exp ? Parameters p are updated for each random choosing of experiments in currently serial code, I am unable to see how they'll be updated if we write parallel for loops

[jiweiqi]

Yes, normally, we can not do that as we have to feed the optimizer one batch by one batch. I have an impression that some work is trying to do distributed training but I don't think it is widely accepted yet.

[yewalenikhil65]

I think it's better to open in discussion section thread for multithreading and distributed computing in the code