TestHarness should be able to start directory walking in another directory

[permcody]

Reason

Some of our applications have the need to check in very large mesh files or other data files for running examples or problems. It makes sense to split these out into a separate repository so that our source repositories don't grow too large. The problem with doing this now is the TestHarness will not search for tests in submodules and other solutions where you use relative paths to find the TestHarness from the destination directory are a bit clunky. There are several ways to test in submodules but we need better support in the TestHarness.

Design

I can think of at least three possible designs to make this work:

1. Add an option to the TestHarness (maybe -C

   ) that will change to the directory before starting the walk. In this case, we wouldn't check the directory to make sure we aren't crossing git repository boundaries.

2. Allow the TestHarness to search up from the current directory until it finds the designated application. We already do this for finding testroot. This might be a good feature anyway. Of course this would probably mean that we'd need to have a somewhat orphaned run_tests script sitting in these separate submodules. This isn't necessarily a bad idea because it would give users of these submodules a natural way to test them since the script would be right in the root.

3. All users to override the "don't test in submodules" behavior in the TestHarness. As a global option this isn't something we want. The TestHarness would find moose_test tests, module tests and other things that most likely won't run under other application binaries. However, if we had a allow testing in a specific submodule, that would make running all the tests together really handy (e.g. run both the integration tests and the heavier examples in a single invocation).

It might turn out that the best path forward is to implement a few of these designs. They are somewhat complementary to one and other. I think my vote would be to implement at least 1 and 3, but we might want 2 eventually.

Impact

New features that simplify testing of separate tests located in a different submodule.

[rwcarlsen]

Right now, the test harness will run in non-app directories if you use the generic moose/scripts/run_tests script. For example, if you cd into your/app/special-test-submodule-dir, and then execute $MOOSE_DIR/scripts/run_tests, everything should work just like you expect. when searching for tests, it just doesn't recurse into git repos/submodules. But if you are already in one, it works.

[snschune]

Is anyone working on this right now? If not, I can take a look but I'd need some guidance on where to start.

[permcody]

All of the walking logic is in the main TestHarness.py file. Take a look at "findAndRunTests". There use to be a pattern match piece that would filter at that level, but I think that got removed in favor of a uniform "--re" flag instead. If we want to start elsewhere it would likely only affect this method.

[snschune]

There is --spec-file which can take a directory as argument. I tested it and it does seem to work. This is all we need, isn't it?