Inconsistently failing Anasazi tests on GCC 4.9.3 MPI clean build

[jwillenbring]

CC: @trilinos/framework

@trilinos/anasazi

There are two tests that seem to sometimes pass and sometimes fail on the GCC 4.9.3 clean build:

http://testing.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=2934307

The tests are

Anasazi_Epetra_ModalSolversTester_MPI_4
Anasazi_Epetra_OrthoManagerGenTester_1_MPI_4

For a clean build, we can't run tests that don't pass consistently. For example, going forward we will make automated decisions when everything passes for certain builds. Either disabling for fixing the tests would be a reasonable solution. Feel free to split the ticket into two tickets if the two tests should be dealt with separately.

[hkthorn]

I have seen this test failure for some time, but have never been able to reproduce it on any machines that I have access to. Even when I run the testing multiple times, the failures don't appear. The output indicates a catastrophic numerical issue that is hard to debug without being able to reproduce it.

[bmpersc]

I am seeing similar issues with a GCC 4.8.4 build. I am seeing the following tests failing intermittently: Anasazi_Epetra_ModalSolversTester_MPI_4
Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4.

You can see the most recent Anasazi_Epetra_ModalSolversTester_MPI_4 test at:
https://testing.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=2964014

You can see the most recent Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 at:
https://testing.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=2969422

Note that we promoted the 4.8.4 build to clean yesterday because it was running clean on Sunday and Monday. Of course Murphy raised his head last night and the first day as clean we have one of these intermittent failures. By any chance are there randomly generated values in these tests? If not this is likely indicative of a memory usage error.

[bmpersc]

@hkthorn, would it make sense for us to disable these handful of tests for the clean builds while the real issue is investigated?

[william76]

From our framework meeting this morning, @bmpersc is going to add the how-to instructions for disabling a specific test to this issue so I can put that into the jenkins job and disable this test for the time being.

[bmpersc]

@william76 There is a way to disable individual tests in the cmake configure. The basic form is -D<test name>_DISABLE:BOOL=ON, so for this case to disable these two tests you would use:

-DAnasazi_Epetra_ModalSolversTester_MPI_4_DISABLE:BOOL=ON
-DAnasazi_Epetra_OrthoManagerGenTester_1_MPI_4_DISABLE:BOOL=ON

This will disable the tests only for the configure(s) that you add it to so other tests could still run the test including anyone looking into the issue.

[william76]

@bmpersc I checked into this and it looks like this 4.9.3 build on Clean is being driven by the parameterized build... Do you know if it's possible to pass these disable flags in as command-line arguments to CTest or do they have to be in the CMake file itself?

We may need to think about adding an additional parameter in Jenkins. Maybe something like CTEST_EXTRA_ARGUMENTS or something like that.

[bmpersc]

@william76 the options I gave need to be in the cmake configure. If we were to pass them to ctest with -D we'd still have to do something in the cmake files to take those options and add them to the cmake configure. As for having another parameter for this purpose, that is something we can discuss in another venue.

[william76]

Merged by PR #1484

Added the options to the ctest file for the parameterized build.

• "-DAnasazi_Epetra_ModalSolversTester_MPI_4_DISABLE:BOOL=ON"
• "-DAnasazi_Epetra_OrthoManagerGenTester_1_MPI_4_DISABLE:BOOL=ON"

[bmpersc]

I'm not 100% sure, but I believe this is related. It looks like the test Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 is also failing randomly on the 4.8.4 clean build. It seems far less likely to happen, but it has happened a handful of times in the last month.

[william76]

I'll disable the Anasazi_Epetra_OrthoManagerGenTester_0_MPI_4 in the parameterized build since it's also showing up unstable on the 4.9.3 build in the clean track.