FVNS mean-zero-pressure Exodus restart issue

[smharper]

Bug Description

With an INSFV system that uses a mean-zero-pressure kernel, it looks like not all of the information makes it through the exodus restart process. In particular, it looks like scalar lambda variables are not being properly restarted. This means that if you converge a system and restart it, the initial residual is not converged.

Steps to Reproduce

This can be reproduced with the test located at modules/navier_stokes/test/tests/finite_volume/ins/lid-driven. Copy the lid-driven.i input to something new like lid-driven-restart with the following changes:

11,19c11,14
<   [gen]
<     type = GeneratedMeshGenerator
<     dim = 2
<     xmin = 0
<     xmax = .1
<     ymin = 0
<     ymax = .1
<     nx = 20
<     ny = 20
---
>   [fmg]
>     type = FileMeshGenerator
>     file = 'lid-driven_out.e'
>     use_for_exodus_restart = true
25a21
>     initial_from_file_var = u
28a25
>     initial_from_file_var = v
31a29
>     initial_from_file_var = pressure
35a34
>     initial_from_file_var = lambda
174a174,177
> []
> 
> [Debug]
>   show_var_residual_norms = true

Run the lid-driven.i then lid-driven-restart.i. If restart was working well, the initial residual would be low, but it is not. I believe this is because the lambda value is not being properly read from the exodus file during restart.

Impact

Working restarts will be important for initializing big transient simulations. We can use execute_on = initial multiapps or setting the lambda/initial_condition as a workaround.

[lindsayad]

Ok @smharper starting on this now

[lindsayad]

Hmm yea we are reading as if the solution vector structure is this:
u0, v0, p0, lm0 ...

When the actual structure is:
u0, v0, p0, u1, v1, p1...... lm0

Here is output of final solution from part1 (for a 2x2 mesh)

-0.0886693
0.077205
-0.05
-0.077205
-0.0886693
0.05
0.077205
0.0886693
-0.15
0.0886693
-0.077205
0.15
4.15046e-21

and then the restarted solution vector:

-0.0886693
0.077205
-0.05
4.15046e-21
-0.0886693
0.05
0.077205
0.0886693
-0.15
0.0886693
-0.077205
0.15
0.

[lindsayad]

I think this is going to be about a two line fix. 🎉

[liangyu-psi]

@smharper Hi Smharper. Nice to meet you. May I ask another question about lid-driven.i? Did you try to use rho=1 mu=0.0001 to calculate? Or exchange the velocity or diameter to make the Re over 3000. It seems hard to converge. When the Re exceed 3000, this example would not converge. Have you ever meet the issue? And how did you handle it?

[lindsayad]

For those encountering the above comment, this is being discussed more in detail starting at #15644 (comment)