Need a capability to generate coordinated subapp split mesh file

[YaqiWang]

Reason

When we specify --use-split on command line, we possibly are expecting the sub-apps to use a split mesh file and a different file as well. This feature is not available currently as showed with the following:

cd moose/test/tests/multiapps/full_solve_multiapp
mpirun -np 2 ../../../moose_test-opt -i master.i --split-file a --split-mesh 2
mpirun -np 2 ../../../moose_test-opt -i master.i --use-split --split-file a

It appears that both the main-app and the sub-app are loading the same split mesh file. Another issue here is that we have to use --split-file option to specify the name of the split mesh file because the input file does not use FileMesh (note that FileMeshGenerator does not work).

Design

I suggest to abandon --split-file option first. We can use the file_base to output the split mesh always. In this way, we can somehow generate the split mesh file for all subapps because they all have different file_base.

Secondly, currently we require a number or numbers separated by commas specified for --split-mesh. We should make it accept default empty value, with which we just split the mesh with the number of processors in the current run.

Third we merge moose/framework/src/actions/SplitMeshAction.C into moose/framework/src/actions/SetupMeshCompleteAction.C and when the input contains multiapps and with --split-mesh, we do not stop after main-app's mesh is setup, and keep going until multiapp is added (sub-apps are constructed). In this way, we will have the split mesh for the main-app and all sub-apps.

Impact

A new capability that may require changes of our current design on user interface.

[loganharbour]

I've thought about this some.

I think the best path forward here is to not let any of the split mesh CLI args (--split-mesh, --use-split, --split-file) apply to the multi apps. If you wish to use split mesh with multiapps, you can use the multiapp specific CLI args (I think this is something like multiapp0:--split-mesh).

In the cases where you have only one or two multiapps that really do need split mesh, this is an easy setup. If you have a huge hierarchy for multiapps... I think that it is fair to assume that you probably do not need split mesh for a TON of applications, because they are likely smaller applications. If this is not the case, we can deal with this in the future by adding additional options to apply split mesh to multiple applications.

[friedmud]

In case someone is reading this later...

--split-file is necessary because split meshes typically go somewhere different from the output files - so using file_base doesn't make sense.

Passing the number of processors to --split-file is necessary because the entire point of mesh splitting is to be able to use less procs per-node to read in a large mesh and split for different numbers of procs. It makes no sense to have it split for the number of procs the splitting process is using.

We definitely want a separate action for splitting - but I agree that we need to take into account that multiapps may need to be split as well.

[YaqiWang]

Using file_base means outputting the split mesh into something like input_out.cpr/ for input.i. Note that the name of the folder can be changed later while the split mesh in it is still valid. I did an attempt last night quickly including fixing the tests. It can help remove lots of lines of codes and make the user interface easier. For example, if an input has only mesh generators but not using FileMesh, users are required to provide a file name and provide the name again when using the split.

About Logan's suggestion, possibly we can have a parameter like --split-subapp-mesh default to false. When it is true, we pass the split mesh args to MultiApp. Otherwise it is difficult to remember the input syntax and figure out the subapp name for a normal user. This makes using file_base more appealing because otherwise users are forced to provide --split-file for potential naming conflict. When this parameter is set to true, we need to keep going to the point MultiApps are added.