Exploring how to reduce setup time for large-element-stencil simulations

[lindsayad]

Closes #3422

I don't think we're fully settled on this yet, but I'm opening it so I have code to reference in the discussion on #3422

[roystgnr]

That's kind of reassuring. We've got an incompatible declaration in a unit test, but nothing in MOOSE?

Even the one in miscellaneous_ex9 and the other one in a unit test seem to have been written using the typedef properly.

[moosebuild]

Job Coverage on de7fc43 wanted to post the following:

Coverage

	7bbcc3	#3431 de7fc4
	Total	Total	+/-	New
Rate	59.63%	59.63%	-	100.00%
Hits	48033	48033	-	4
Misses	32525	32525	-	0

Diff coverage report

Full coverage report

This comment will be updated on new commits.

[lindsayad]

@loganharbour @roystgnr should I expect to see some values in the +/- column?

[roystgnr]

I figured I'd run this through LIBMESH_BENCHMARK, see if it shows any noticeable improvement there ...

And instead I get a crash from the GMG runs in fem_system_ex1, if I try to use more than 1 processor.

[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Input ilocal has duplicate entries which is not allowed for this PetscSF

The failing run is

mpirun -np 2 ./example-opt solver_type=petscdiff coarsegridsize=6 coarserefinements=3 transient=false n_timesteps=1 --use_petsc_dm --node-major-dofs -snes_view -snes_monitor -snes_converged_reason -snes_rtol 1.0e-4 -ksp_type fgmres -ksp_rtol 1.0e-5 -pc_type fieldsplit -pc_fieldsplit_0_fields 0,1 -pc_fieldsplit_1_fields 2 -pc_fieldsplit_type schur -pc_fieldsplit_schur_fact_type full -pc_fieldsplit_schur_precondition a11 -fieldsplit_0_pc_type mg -fieldsplit_0_pc_mg_galerkin both -fieldsplit_0_pc_mg_type full -pc_mg_levels 3 -fieldsplit_0_pc_mg_levels 3 -fieldsplit_0_mg_levels_pc_type sor -fieldsplit_0_mg_levels_pc_sor_its 5 -fieldsplit_0_mg_levels_ksp_type richardson -fieldsplit_0_mg_levels_ksp_richardson_self_scale -fieldsplit_0_mg_levels_ksp_max_it 5 -fieldsplit_0_mg_coarse_pc_type lu -fieldsplit_0_mg_coarse_ksp_type preonly -fieldsplit_0_ksp_max_it 1 -fieldsplit_0_ksp_type gmres -fieldsplit_p_ksp_max_it 5 -fieldsplit_p_ksp_type gmres -fieldsplit_p_pc_type none -fieldsplit_p_ksp_rtol 1.0e-4 -fieldsplit_p_inner_pc_type lu -fieldsplit_p_inner_ksp_type preonly -fieldsplit_p_upper_pc_type lu -fieldsplit_p_upper_ksp_type preonly

and the failing configuration is

../configure --prefix=/home/stogrh/LIBRARIES/libmesh-normal --enable-everything --disable-deprecated --enable-paranoid-warnings --enable-werror --enable-petsc-required PETSC_DIR=/home/stogrh/LIBRARIES/petsc PETSC_ARCH=linux-cxx-petscmpich --with-vtk-lib=/home/stogrh/LIBRARIES/vtk-9.1.0/lib --with-vtk-include=/home/stogrh/LIBRARIES/vtk-9.1.0/include/vtk-9.1/ --no-create --no-recursion

Could you see if you can replicate? I rebased on master to get the PETSc 3.18 compatibility. My next step is going to be to try a LIBMESH_BENCHMARK=1 run with just master to see if something simply broke from the PETSc upgrade alone.

[roystgnr]

Well, damn it. Broken with PETSc 3.18 in master too. Time to resurrect 3.17, for now.

[roystgnr]

LIBMESH_BENCHMARK doesn't show any noticeable change from this PR; it's faster but within measurement error. Which I guess is to be expected; all the build_sparsity() there put together is done in under 5 seconds.

[roystgnr]

That glitch was due to a Firefox crash, but you know what, let's go with it. Turning a test case runtime from 177 seconds to 20 seconds or even from 7.6 seconds to 4.7 seconds is double-approved.