SEACAS tests 'Not Run' on ATDM mutrino builds

[fryeguy52]

CC: @trilinos/seacas , @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe

Next Action Status

Tests disabled in PR in #3530 merged on 9/29/2018 and these tests went missing in this build on 9/30/2018 as shown here. Next: Fix the tests?

Description

several seacas tests are showing up as "not run" in the ATDM builds on mutrino. As shown here the tests are:

• SEACASAprepro_aprepro_test_exodus
• SEACASIoss_exodus32_to_exodus32
• SEACASIoss_exodus32_to_exodus32_pnetcdf
• SEACASIoss_exodus32_to_exodus64

are not run in the build:

• Trilinos-atdm-mutrino-intel-opt-openmp-HSW

the test output on CDash for all 4 of these is:

Unable to find required file: CMND_PATH-NOTFOUND

Steps to Reproduce

One should be able to reproduce this failure on the machine mutrino as described in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md
  More specifically, the commands given for the system mutrino are provided at:
• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#mutrino
  The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh intel-opt-openmp-HSW

$ cmake \
  -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
  -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_SEACAS=ON \
  $TRILINOS_DIR

$ make -j16

$ salloc -N 1 -p standard -J $JOB_NAME ctest -j16

[bartlettroscoe]

@gsjaardema, any idea what is happening here?

[gsjaardema]

@bartlettroscoe I don’t have any idea. I will see if I can reproduce.

[gsjaardema]

When I run this manually on mutrino, I get the following:

4/52 Testing: SEACASIoss_exodus32_to_exodus32
4/52 Test: SEACASIoss_exodus32_to_exodus32
Command: "/projects/netpub/atdm/cmake-3.11.4/bin/cmake" "-DTEST_CONFIG=" "-P" "/lscratch1/gdsjaar/trilinos/mutrino/packages/seacas/libraries/ioss/src/main/SEACASIoss_exodus32_to_exodus32.cmake"
Directory: /lscratch1/gdsjaar/trilinos/mutrino/packages/seacas/libraries/ioss/src/main
"SEACASIoss_exodus32_to_exodus32" start time: 
Output:
----------------------------------------------------------
<end of output>
Test time =   0.00 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression not found.Regex=[OVERALL FINAL RESULT: TEST PASSED .SEACASIoss_exodus32_to_exodus32.
]
"SEACASIoss_exodus32_to_exodus32" end time: Sep 27 14:49 MDT
"SEACASIoss_exodus32_to_exodus32" time elapsed: 00:00:00
----------------------------------------------------------

Unable to find required file: CMND_PATH-NOTFOUND

It looks like the test runs, but then at the end there is the CMND_PATH-NOTFOUND

If I manually run the Command listed above, it works correctly and the CMND_PATH-NOTFOUND is not printed. I'm not sure where the NOTFOUND is coming from; maybe we need to just disable the tests?

[bartlettroscoe]

maybe we need to just disable the tests?

@gsjaardema, sure, I can create a PR to disable these tests.

Note that these tests are also showing as "not run" in the KNL build as well as shown here. So I will disable these tests in that build too.

[bartlettroscoe]

Tests disabled in PR in #3530 merged on 9/29/2018 and these tests went missing in this build on 9/30/2018 as shown here.

Adding "Disabled Tests" label and leaving open as per policy shown here.

[prwolfe]

This issue cropped up on our UVM OFF build over the past several days blocking develop->master promotion. I dug a bit and found a few things.

The tests in question are all structured in the same basic way

if (${CMAKE_PROJECT_NAME}_ENABLE_SEACASExodiff)
   set(SEACAS_EXODIFF_BINARY "../../../../applications/exodiff/exodiff")
endif()

if (SEACAS_EXODIFF_BINARY)
TRIBITS_ADD_ADVANCED_TEST(exodus32_to_exodus32
   TEST_0 EXEC io_shell ARGS ${DECOMP_ARG} ${JOIN_ARG} ${CMAKE_CURRENT_SOURCE_DIR}/test/8-block.g 8-block32.g
     NOEXEPREFIX NOEXESUFFIX
     NUM_MPI_PROCS ${NPROCS}
   TEST_1 CMND ${SEACAS_EXODIFF_BINARY} ARGS -pedantic ${CMAKE_CURRENT_SOURCE_DIR}/test/8-block.g 8-block32.g
  COMM mpi serial
  XHOSTTYPE Windows
  )

The TEST_1 uses CMND even though the exodiff it points at is in the build tree. The macro does not search for EXEC as it assumes it will be built and does for CMND (using FIND_PROGRAM). At configure time that executable will not be there for new builds but may be for re-builds.

I don't understand why we get this on only partial sets, but I expect we need to change both to EXEC and find a way to get that to do what is needed (main difference is mpiexec I think) or just live with the fact that TEST_1 WILL use mpiexec.

[prwolfe]

Tagging framework members @jwillenbring and @ZUUL42

[gsjaardema]

I think this has been fixed. If not, please reopen.