Automatically run tests from submodules

[tophmatthews]

Reason

For some of the shepard apps like direwolf, it's nice to check that all the modules are working. A while back @andrsd suggested something like ./run_tests --spec-file bison, which works fine. It would be nice to automatically check all submodules instead of individually doing each submodule? For example, direwolf has 8 total submodules, making it easy to forget which have been tested. This is much easier than building each submodule and running tests. Furthermore, it ensures that the current executable does everything expected of it (test-wise that is).

Design

Maybe add run_tests script option like ./run_tests --submodules that uses the current executable to run the tests in all the submodules.

Impact

Easy user checking to make sure all submodules are working correctly.

[tophmatthews]

I was thinking you could parse .gitmodules and just run tests in there, but really, it'll take some wizardry for moose modules to test the ones that are actually built.

[tophmatthews]

@permcody

[tophmatthews]

I really just want to actually test the executable I will be using against all "promised" capabilities. For example, direwolf has Bison as a submodule. Sure, it's assumed that direwolf tests okay will all bison tests, but how do I know? Well, running tests with --spec-file. But what about moose modules?

There it actually breaks down because I have to know all the modules in play, and go into each one specifically (eg tensor_mechanics). I can't just go into moose/modules with --spec-file, since not all modules are built. Sure, I can build the executable in moose/modules and then test, but I'm not really testing the direwolf executable at that point

[permcody]

Tag @socratesgorilla. This is an interesting and important ticket to take a look at when you have time. Let me know if you have time to work on this and I can discuss some implementation details with you.
Tag @milljm for awareness.

[permcody]

@tophmatthews - Still thinking about this. I'm not sure we will always guarantee that a top level app can run the exact same input files as it's dependencies. The reason for this is that App objects can change syntax which isn't inherited (or even designed to be inherited) by a top level app. However, being able to test submodule tests is important but we may go about this a completely different way. We have been discussing the need to choose which app to instantiate for a given input file (e.g. actually creating a BisonApp when running Direwolf for the purpose of running a pure Bison file).

[socratesgorilla]

Was looking at tickets to do and saw I was already tagged - did this get resolved or is it still up in the air?

[permcody]

Not finished for sure. We’ll need to talk about what we want to happen here