Vector variables don't work with distributed mesh?

[Eilloo]

Hi all,

Are vector variables compatible with distributed meshes/Nemesis format?

The behaviour I'm seeing is the simulation solves fine, and I can open the Nemesis files in Paraview.
However, the values of my vector variables contain garbage (e+37), and Paraview crashes if I try and go to any non-zero timestep.
The other variables look fine, and if I solve using a replicated mesh and exodus output, everything looks normal.
To be clear, I have been able to inspect my variables using another tool to see that the e+37 values persist at non-zero timesteps.

It seems like there are 3 likely possibilities: Either vector variables are not compatible, or I am consistently setting up something wrong with the input files, or there is a bug here.

Below is a MWE to reproduce this:

filename = NemesisMWEOut
inlet_velocity = 1e-4   #Re = 100

[Mesh]
    [base]
        type = GeneratedMeshGenerator
        dim = 2
        xmin = 0
        ymin = 0
        xmax = 10
        ymax = 2
        nx = 50
        ny = 10
    []
    allow_renumbering = false
    parallel_type = DISTRIBUTED
[]

[Variables]
    [pressure]
        family = LAGRANGE
    []
    [velocity]
        family = LAGRANGE_VEC
    []
[]

[Materials]
    [fluid]
        type = ADGenericConstantMaterial
        prop_names = 'rho mu cp k'
        prop_values = '1000 1e-3 1 1'
    []
    [ins_mat]
        type = INSADStabilized3Eqn
        velocity = velocity
        pressure = pressure
        temperature = '288'
        alpha = 1.0
    []
[]

[Kernels]
    [mass]
        type = INSADMass
        variable = pressure
    []
    [momentum_advection]
        type = INSADMomentumAdvection
        variable = velocity
    []
    [momentum_pressure]
        type = INSADMomentumPressure
        variable = velocity
        pressure = pressure
    []
    [momentum_viscous]
        type = INSADMomentumViscous
        variable = velocity
    []
    [mass_pspg]
        type = INSADMassPSPG
        variable = pressure
    []
[]

[BCs]
    [inlet]
        type = VectorDirichletBC
        variable = velocity
        values = '${inlet_velocity} 0 0'
        boundary = left
    []
    [outlet]
        type = ADDirichletBC
        variable = pressure
        value = 0
        boundary = right
    []
    [no_slip]
        type = VectorDirichletBC
        variable = velocity
        values = '0 0 0'
        boundary = 'top bottom'
    []
[]

[Preconditioning]
    [precond]
        type = SMP
        full = true
    []
[]

[Executioner]
    type = Steady
    solve_type = NEWTON
    petsc_options_iname = '-pc_type -pc_factor_shift_type'
    petsc_options_value = 'lu       NONZERO'
[]

[Outputs]
    [nemesis]
        type = Nemesis
        file_base = ${filename}
        enable = true
    []
[]

[lindsayad]

Distributed mesh and nemesis output are two distinct features. We test the INSAD implementation with distributed mesh every day on CIVET so I'm very confident that vector variables work with distributed meshes. However, I wouldn't be all that surprised if there are some issues with vector variables and nemesis output. @roystgnr have you ever looked at vector FE types in Nemesis?

[Eilloo]

A further question: What happens when I set parallel_type = DISTRIBUTED in the mesh block, but also set the output file type to use exodus?
I get warnings that exodus only supports serial meshes, but it goes ahead and solves, giving me an exodus file where the solution looks fine.
Does specifying exodus in the output block overwrite the use of a distributed mesh? Or does it somehow stitch the separate nemesis files together?

The reason is to test whether the problem is at the 'distributed mesh' stage, or if it is indeed specific to Nemesis.

[GiudGiud]

If you do that, we force a serialization then output to exodus.
it does not stitch nemesis together

[lindsayad]

To be clear we only serialize for Exodus output on processor 0. So on processor 0 you will have a copy of the entire mesh but all other processes will still only have parts of the mesh

[GiudGiud]

I confirmed the problem and raised this issue
#24193

We will address it when we have time. In the mean time, please just use this auxkernel to output each component separately.
If you still name each non-vector component variable like v_x v_y v_z, Paraview will still recognize it s vector

https://mooseframework.inl.gov/source/auxkernels/VectorVariableComponentAux.html