Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change TriBITS/Trilinos TPLs to use find_package(CUDATookit) to fix builds with downstream customers using find_package(CUDATookit) (#10954) #11093

Merged
merged 3 commits into from
Oct 5, 2022
Merged

Change TriBITS/Trilinos TPLs to use find_package(CUDATookit) to fix builds with downstream customers using find_package(CUDATookit) (#10954) #11093

merged 3 commits into from
Oct 5, 2022

Conversation

bartlettroscoe
Copy link
Member

@bartlettroscoe bartlettroscoe commented Oct 3, 2022 •
edited
Loading

CC: @trilinos/teuchos

Description

The main thing this PR does is change the TriBITS and Trilinos CUDA-related TPLs from from using find_package(CUDA) (or just raw find operations) to using find_package(CUDAToolkit) to avoid imported target namespace clashes with downstream CMake projects that find_package(CUDAToolkit) (see #10954 and especially #10954 (comment)).

This PR also includes the updated snapshot of TriBITS 'master' which also includes the TriBITS PRs and commits directly on 'master'

Of those other changes, the biggest is the renaming of TriBITS packages and TPLs, see the details in:

Instructions for reviewers

  1. Please review the TriBITS changes in the TriBITS PRs listed above. Do not try to review the huge patch in the snapshot commit 082ee03!
  2. Note that the matching TriBITS change for the Trilinos commit 97961c7 is cmake/tribits/core/std_tpls/FindTPLCUDA.cmake.

Testing

I tested offline the Trilinos PR builds for 'cuda-11' on 'ascicgpu16' and the 'cuda-10' build on 'vortex. I also tested against the Spack manager build of Nalu-wind and verified that it fixes the build problem reported in #10954.

For Trilinos PR testing, the repo state was:

Trilinos PR builds repo state: (click to expand) 
$ gitdist-status 
-----------------------------------------------------------------------------------------------------------------------
| ID | Repo Dir                         | Branch                   | Tracking Branch                      | C | M | ? |
|----|----------------------------------|--------------------------|--------------------------------------|---|---|---|
|  0 | Trilinos (Base)                  | 10954-nalu-wind-fix-cuda | rab-github/10954-nalu-wind-fix-cuda  |   |   |   |
|  1 | packages/framework/GenConfig     | HEAD                     |                                      |   |   |   |
|  2 | packages/framework/srn-ini-files | tril-10355-cmake-upgrade | cgl-rabartl/tril-10355-cmake-upgrade |   |   |   |
|  3 | run_trilinos_pr_builds           | master                   | origin/master                        |   |   |   |
|  4 | TriBITS                          | tril-10954-cudatoolkit   | rab-github/tril-10954-cudatoolkit    |   |   |   |
-----------------------------------------------------------------------------------------------------------------------

$ gitdist-repo-versions 
*** Base Git Repo: Trilinos
97961c75d7a9c3c8330ccc184eb8cfd14c90e7a5 [Fri Sep 30 06:25:20 2022 -0600] <rabartl@sandia.gov>
Change to use find_package(CUDAToolkit) (#10954)
*** Git Repo: packages/framework/GenConfig
924a08af66f0a0573b5dd1128179731489339aec [Wed Jun 15 15:55:48 2022 +0000] <eharvey@sandia.gov>
Merge branch 'del-gen-symlinks' into 'master'
*** Git Repo: packages/framework/srn-ini-files
07bd9273dbeb5245d6a5bfbe2a31b84278886beb [Wed Sep 7 10:03:10 2022 -0600] <rabartl@sandia.gov>
Upgrade all builds to cmake 3.22+ (trilinos/Trilinos#10355)
*** Git Repo: run_trilinos_pr_builds
23e934fbbd2983461e07d177fb755b01bc621d4a [Sun Oct 2 06:54:17 2022 -0600] <rabartl@sandia.gov>
Remove grep of CDash URL from summary script
*** Git Repo: TriBITS
ca865e97a0e11963e2dcacefa95ba749ea8b9c45 [Fri Sep 30 08:38:21 2022 -0600] <rabartl@sandia.gov>

Trilinos PR rhel7 'cuda-11.4.2-uvm-off' build on 'ascicgpu16'

To test this I ran:

$ ssh ascicgpu16

$ cd /fgs/rabartl/Trilinos.base2/BUILDS/PR/cuda-11.4.2-uvm-off/

$ time ./run_trilinos_pr_builds.py \
  -b tril-10954-cudatoolkit-2 \
  -c Trilinos_TRIBITS_DIR:STRING=TriBITS/tribits \
  -p ALL_PACKAGES \
  cuda-11.4.2-uvm-off \
  &> run_trilinos_pr_builds.rhel7-cuda.all.out

which submitted to CDash here showing:

image

Log file run_trilinos_pr_builds.rhel7-cuda.all.out (click to expand) 
=================================
Running set of Trilinos PR builds
=================================

Fri Sep 30 08:40:09 MDT 2022

See results on CDash at:

    https://testing.sandia.gov/cdash/index.php?project=Trilinos&filtercount=3&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=Experimental&field2=buildname&compare2=65&value2=tril-10954-cudatoolkit-2_&field3=buildstarttime&compare3=84&value3=now

Final set of builds: cuda-11.4.2-uvm-off

Generating the build directory and scripts: cuda-11.4.2-uvm-off

  full-build-name: 'rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables'

***
*** Running configure, build, and test for PR build 'cuda-11.4.2-uvm-off'
***

Full build name: rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables

Fri Sep 30 08:40:09 MDT 2022

Running: source load-env.sh --ci-mode &> load-env.out

Running: ./do-configure &> configure.out


real    2m42.599s
user    0m58.671s
sys     0m46.925s

Running: make dashboard &> make.dashbaord.out


real    348m38.528s
user    2148m41.267s
sys     453m5.566s

Results of configure, build, and tests:

Configure PASSED!

Build PASSED!

100% tests passed, 0 tests failed out of 2951

Fri Sep 30 14:32:06 MDT 2022

See results on CDash at:

    https://testing.sandia.gov/cdash/index.php?project=Trilinos&filtercount=3&showfilters=1&filtercombine=and&field1=groupname&compare1=61&value1=Experimental&field2=buildname&compare2=65&value2=tril-10954-cudatoolkit-2_&field3=buildstarttime&compare3=84&value3=now

real    351m56.370s
user    2149m57.316s
sys     453m59.537s

Trilinos PR rhel7 'ats2' build on 'vortex'

With that same repo state on 'vortex' I first build on the login node with:

$ ssh vortex

$ cd /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/

$ env \
    CTEST_SITE=vortex \
    CTEST_DO_TEST=OFF \
  ./run_trilinos_pr_builds.py \
    -c Trilinos_TRIBITS_DIR:STRING=TriBITS/tribits \
    -b 10954-nalu-wind-fix-cuda -r -p ALL_PACKAGES -j 16 -t 4 ats2 \
    &> run_trilinos_pr_builds.ats2.all.build.out

Log file run_trilinos_pr_builds.ats2.all.build.out.txt

then I ran the tests on launch node with:

$ bsub -J pr_builds -W 6:00 -Is bash

$ env \
    CTEST_SITE=vortex \
    CTEST_DO_NEW_START=OFF \
    CTEST_DO_CONFIGURE=OFF \
    CTEST_DO_BUILD=OFF \
    CTEST_DO_TEST=ON \
  ./run_trilinos_pr_builds.py \
    -c Trilinos_TRIBITS_DIR:STRING=TriBITS/tribits \
    -b 10954-nalu-wind-fix-cuda -r -p ALL_PACKAGES -j 16 -t 4 ats2 \
    &> run_trilinos_pr_builds.ats2.all.test.out

with log file run_trilinos_pr_builds.ats2.all.test.out.txt

There were three failing tests:

The following tests FAILED:
	3548 - TrilinosInstallTests_doInstall (Timeout)
	3549 - TrilinosInstallTests_find_package_Trilinos (Not Run)
	3550 - TrilinosInstallTests_simpleBuildAgainstTrilinos (Not Run)

As I found before, those fail because you can't install from the compute node with this setup. But when I re-ran them (as described below)

Analysis of failing TrilinosInstallTests tests on 'ats2' build (click to expand)

The 3 failing tests were:

The following tests FAILED:
	3548 - TrilinosInstallTests_doInstall (Timeout)
	3549 - TrilinosInstallTests_find_package_Trilinos (Not Run)
	3550 - TrilinosInstallTests_simpleBuildAgainstTrilinos (Not Run)

As I found before, those fail because you can't install from the compute node with this setup.

To make those pass:

$ ssh vortex

$ cd /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/

$ . load-env.sh

$ cd packages/TrilinosInstallTests/

$ ctest -R TrilinosInstallTests_doInstall
...
1/1 Test #1: TrilinosInstallTests_doInstall ...   Passed   54.85 sec
...

$ ctest -R TrilinosInstallTests_find_package_Trilinos --fixture-exclude-any doInstall
...
1/1 Test #2: TrilinosInstallTests_find_package_Trilinos ...   Passed    1.42 sec
...

$ bsub -J pr_builds -W 6:00 -Is bash

$ cd ../../

$ . load-env.sh

$ cd packages/TrilinosInstallTests/

$ ctest -R TrilinosInstallTests_simpleBuildAgainstTrilinos --fixture-exclude-any doInstall
...
1/1 Test #3: TrilinosInstallTests_simpleBuildAgainstTrilinos ...***Failed  Required regular expression not found. Regex=[OVERALL FINAL RESULT: TEST PASSED .TrilinosInstallTests_simpleBuildAgainstTrilinos.
]  9.91 sec
...

Wow, the test TrilinosInstallTests_simpleBuildAgainstTrilinos failed! It showed. The LastTest.log file showed:

-- Found CUDAToolkit: /usr/tce/packages/cuda/cuda-10.1.243/nvidia/include (found version "10.1.243") 
CMake Warning at /vscratch1/progenv-20220706/usr/tce/packages/cmake/cmake-3.23.1/share/cmake/Modules/FindCUDAToolkit.cmake:919 (message):
  Could not find librt library, needed by CUDA::cudart_static
Call Stack (most recent call first):
  /vscratch1/progenv-20220706/usr/tce/packages/cmake/cmake-3.23.1/share/cmake/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/external_packages/CUDA/CUDAConfig.cmake:2 (find_dependency)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/KokkosCore/KokkosCoreConfig.cmake:156 (include)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/TeuchosCore/TeuchosCoreConfig.cmake:196 (include)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/Teuchos/TeuchosConfig.cmake:193 (include)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/Trilinos/TrilinosConfig.cmake:123 (include)
  CMakeLists.txt:16 (find_package)



Found Trilinos!  Here are the details:

Let's try running that test from the login node:

$ ssh vortex

$ cd /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/

$ . load-env.sh

$ cd packages/TrilinosInstallTests/

$ ctest -R TrilinosInstallTests_simpleBuildAgainstTrilinos --fixture-exclude-any doInstall
...
1/1 Test #3: TrilinosInstallTests_simpleBuildAgainstTrilinos ...***Failed  Required regular expression not found. Regex=[OVERALL FINAL RESULT: TEST PASSED .TrilinosInstallTests_simpleBuildAgainstTrilinos.
] 37.10 sec
...

This time, only the test fails.

Trying to run the test from the launch node

$ cd TrilinosInstallTests_simpleBuildAgainstTrilinos/

$ ctest
...
1/1 Test #1: MyAppTest ........................***Failed  Required regular expression not found. Regex=[vec.norm1[(][)] = 40
]  8.27 sec
...

The test fails due to a an insufficient CUDA version exception:

terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaGetDeviceCount(&m_cudaDevCount) error( cudaErrorInsufficientDriver): CUDA driver version is insufficient for CUDA runtime version /home/rabartl/Trilinos.base/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:275
[vortex59:90612] *** Process received signal ***
[vortex59:90612] Signal: Aborted (6)
[vortex59:90612] Signal code:  (-6)

The configuration of the project simpleBuildAgainstTrilinos on the login node showed:

-- Found CUDAToolkit: /usr/tce/packages/cuda/cuda-10.1.243/nvidia/include (found version "10.1.243") 
CMake Warning at /vscratch1/progenv-20220706/usr/tce/packages/cmake/cmake-3.23.1/share/cmake/Modules/FindCUDAToolkit.cmake:919 (message):
  Could not find librt library, needed by CUDA::cudart_static
Call Stack (most recent call first):
  /vscratch1/progenv-20220706/usr/tce/packages/cmake/cmake-3.23.1/share/cmake/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/external_packages/CUDA/CUDAConfig.cmake:2 (find_dependency)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/KokkosCore/KokkosCoreConfig.cmake:156 (include)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/TeuchosCore/TeuchosCoreConfig.cmake:196 (include)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/Teuchos/TeuchosConfig.cmake:193 (include)
  /vscratch1/rabartl/Trilinos.base/BUILDS/PR/pr_builds/cuda-10.1.243/install/lib/cmake/Trilinos/TrilinosConfig.cmake:123 (include)
  CMakeLists.txt:16 (find_package)

We did not see that warning when configuring Trilinos on the login node.

I wonder what is going on here and if this is going to impact Trilinos customers or not on this system.

I think we will just have to wait and see what happens. But if Trilinos can configure successfully without this warning, it must be possible for other CMake projects to successfully call find_package(CUDAToolkit) as well.

Therefore, I am going to assume this is not going to be a problem for application codes. And if it is, we will work through the issues.

Testing against nalu-wind with Spack Manager on 'ascicgpu16'

Leveraging the directories I used to preproduce the problem in #10954 (comment) that already has everything build (except nalu-wind), I used the updated Trilinos branch from this PR and got a passing build for nalu-wind and a partially passing set of unit tests (which was said to be okay in #10954 (comment)). Details are given below.

Details for testing against nalu-wind with spack manager (click to expand)

I have completed testing of these TriBITS and Trilinos changes on the cuda-11 build on 'ascicgpu16' and for the cuda-10 build on 'vortex' (see above). Now to test against Nalu-Wind.

The Trilinos repo is in a bit of a mess due to Spack patching and writing a bunch of non-ignored files:

$ ssh ascicgpu16

$ cd /fgs/rabartl/SpackManager.base/spack-manager/environments/bug/trilinos/

$ git status
On branch 10954-nalu-wind-fix-cuda
Your branch and 'rab-github/10954-nalu-wind-fix-cuda' have diverged,
and have 1 and 3 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .spack_patched
        spack-build-01-cmake-out.txt
        spack-build-02-build-out.txt
        spack-build-03-install-out.txt
        spack-build-env-mods.txt
        spack-build-env.txt
        spack-build-out.txt
        spack-build-wmhhsdf/
        spack-configure-args.txt

no changes added to commit (use "git add" and/or "git commit -a")

First, I add ignores for these files to the local Trilinos .git/info/excludes file.

To update safely I do:

$ ssh ascicgpu16

$ cd /fgs/rabartl/SpackManager.base/spack-manager/environments/bug/trilinos/

$ git stash
Saved working directory and index state WIP on 10954-nalu-wind-fix-cuda: 9388b0f6812 WIP: Change to use find_package(CUDAToolkit) (#10954)

$ git log --name-status -1 stash@{0}
commit 4e9e03f8f0ace4b9dcc816331189c35e17ee4cd1 (refs/stash)
Merge: 9388b0f6812 9a0bc64ee4b
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Mon Oct 3 10:33:42 2022 -0600

    WIP on 10954-nalu-wind-fix-cuda: 9388b0f6812 WIP: Change to use find_package(CUDAToolkit) (#10954)

$ git fetch

$ git fetch github

$ git reset --hard @{u}
HEAD is now at a2eded0bcba Merge branch 'tribits_github_snapshot' into 10954-nalu-wind-fix-cuda (#10954)

$ git log-short --dirstat=0 HEAD --not github/develop 
a2eded0bcba "Merge branch 'tribits_github_snapshot' into 10954-nalu-wind-fix-cuda (#10954)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Mon Oct 3 10:14:04 2022 -0600 (22 minutes ago)

082ee037e28 "Automatic snapshot commit from tribits at 23dc20b9"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Mon Oct 3 10:09:04 2022 -0600 (27 minutes ago)

   0.1% cmake/tribits/ci_support/
  45.0% cmake/tribits/core/package_arch/
   0.4% cmake/tribits/core/std_tpls/
   0.7% cmake/tribits/core/utils/
   0.9% cmake/tribits/ctest_driver/
   4.7% cmake/tribits/doc/build_ref/
  10.6% cmake/tribits/doc/guides/ReducedMockTrilinosOutput/
   0.0% cmake/tribits/doc/guides/maintainers_guide/
   0.1% cmake/tribits/doc/guides/users_guide/
  36.1% cmake/tribits/doc/guides/
   0.1% cmake/tribits/examples/TribitsExampleProject/packages/with_subpackages/b/tests/testlib/
   0.7% cmake/tribits/

97961c75d7a "Change to use find_package(CUDAToolkit) (#10954)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Thu Sep 29 11:46:47 2022 -0600 (3 days ago)

 100.0% cmake/TPLs/

$ git stash pop

$ git status
On branch 10954-nalu-wind-fix-cuda
Your branch is up to date with 'rab-github/10954-nalu-wind-fix-cuda'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel.hpp

no changes added to commit (use "git add" and/or "git commit -a")

And with that, the repo versions I am using are:

$ ssh ascicgpu16

$ cd /fgs/rabartl/SpackManager.base/

$ cat .gitdist
spack-manager
spack-manager/environments/bug/nalu-wind
spack-manager/environments/bug/trilinos

$ gitdist-status 
------------------------------------------------------------------------------------------------------------------------------
| ID | Repo Dir                                 | Branch                   | Tracking Branch                     | C | M | ? |
|----|------------------------------------------|--------------------------|-------------------------------------|---|---|---|
|  0 | spack-manager                            | main                     | origin/main                         |   |   |   |
|  1 | spack-manager/environments/bug/nalu-wind | master                   | origin/master                       |   |   |   |
|  2 | spack-manager/environments/bug/trilinos  | 10954-nalu-wind-fix-cuda | rab-github/10954-nalu-wind-fix-cuda |   | 1 |   |
------------------------------------------------------------------------------------------------------------------------------

$ gitdist-repo-versions 
*** Git Repo: spack-manager
504897b6c2f664a42ca0bc7e1e53b8194152db92 [Mon Sep 26 10:47:19 2022 -0600] <58484958+tasmith4@users.noreply.github.com>
in SNL builds, separate trilinos stable vs develop gold file directories (#360)
*** Git Repo: spack-manager/environments/bug/nalu-wind
b3ed3104eeb3683da01f5a0f74591563c0fc5a39 [Wed Sep 28 15:02:17 2022 -0600] <jon.rood@nrel.gov>
Turn on more variants in the CI. (#1046)
*** Git Repo: spack-manager/environments/bug/trilinos
a2eded0bcbaf88171eb22c6e56292b3c723aaae4 [Mon Oct 3 10:14:04 2022 -0600] <rabartl@sandia.gov>
Merge branch 'tribits_github_snapshot' into 10954-nalu-wind-fix-cuda (#10954)

Now to test the builds the updated installs of Trilinos and Nalu-Wind:

$ ssh ascicgpu16

$ cd /fgs/rabartl/SpackManager.base/

$ cat load-env.sh 
export SPACK_MANAGER=$(pwd)/spack-manager
source ${SPACK_MANAGER}/start.sh
quick-create-dev --name bug --spec nalu-wind@master+cuda cuda_arch=70 trilinos@develop

$ . load-env.sh
+ spack-start
+ spack manager create-dev-env --name bug --spec nalu-wind@master+cuda cuda_arch=70 trilinos@develop
==> Warning: included configuration files should be updated manually [files=include.yaml]
==> Spec nalu-wind@master+cuda cuda_arch=70 already configured for development
==> Spec trilinos@develop already configured for development
+ spack env activate --dir /fgs/rabartl/SpackManager.base/spack-manager/environments/bug --prompt

$ time spack install
...
==> Installing trilinos-develop-wmhhsdfv3on6ahzhepsrxn7373eakdlx
==> No binary for trilinos-develop-wmhhsdfv3on6ahzhepsrxn7373eakdlx found: installing from source
==> Already patched trilinos
==> trilinos: Executing phase: 'cmake'
==> trilinos: Executing phase: 'build'
==> trilinos: Executing phase: 'install'
==> trilinos: Successfully installed trilinos-develop-wmhhsdfv3on6ahzhepsrxn7373eakdlx
  Fetch: 0.00s.  Build: 51.02s.  Total: 51.02s.
[+] /fgs/rabartl/SpackManager.base/spack-manager/spack/opt/spack/linux-rhel7-x86_64/gcc-9.3.0/trilinos-develop-wmhhsdfv3on6ahzhepsrxn7373eakdlx
==> Installing nalu-wind-master-gs4m3a3codejvactaj765tp7fxaoyods
==> No binary for nalu-wind-master-gs4m3a3codejvactaj765tp7fxaoyods found: installing from source
==> No patches needed for nalu-wind
==> nalu-wind: Executing phase: 'cmake'
==> nalu-wind: Executing phase: 'build'
==> nalu-wind: Executing phase: 'install'
==> nalu-wind: Successfully installed nalu-wind-master-gs4m3a3codejvactaj765tp7fxaoyods
  Fetch: 0.00s.  Build: 1m 5.41s.  Total: 1m 5.41s.
[+] /fgs/rabartl/SpackManager.base/spack-manager/spack/opt/spack/linux-rhel7-x86_64/gcc-9.3.0/nalu-wind-master-gs4m3a3codejvactaj765tp7fxaoyods
==> Warning: Module file /fgs/rabartl/SpackManager.base/spack-manager/spack/share/spack/modules/linux-rhel7-x86_64/nalu-wind-master-gcc-9.3.0-gs4m3a3 exists and will not be overwritten
==> Warning: Module file /fgs/rabartl/SpackManager.base/spack-manager/spack/share/spack/modules/linux-rhel7-x86_64/trilinos-develop-gcc-9.3.0-wmhhsdf exists and will not be overwritten

real    2m58.113s
user    2m10.527s
sys     0m53.395s

Now to test it:

$ build-env-dive nalu-wind

$ ctest -R unit -VV

...

8: Test command: /usr/bin/sh "-c" "/projects/wind/system-spack/opt/spack/linux-rhel7-x86_64/gcc-9.3.0/mpich-3.4.2-4h2muy6jlgfdahehxmxa4ybndfdb6gx2/bin/mpiexec -n 1  /fgs/rabartl/SpackManager.base/spack-manager/environments/bug/nalu-wind/spack-build-gs4m3a3/unittestX  --gtest_filter=BasicKokkos.discover_execution_space:*.NGP*"
8: Test timeout computed to be: 20000
8:    Nalu-Wind Version: v1.2.0
8:    Nalu-Wind GIT Commit SHA: b3ed3104eeb3683da01f5a0f74591563c0fc5a39
8:    Trilinos Version: 13.5-ga2eded0bcba
8: 
8: Note: Google Test filter = BasicKokkos.discover_execution_space:*.NGP*
8: [==========] Running 144 tests from 34 test suites.
8: [----------] Global test environment set-up.
8: [----------] 1 test from BasicKokkos
8: [ RUN      ] BasicKokkos.discover_execution_space
8: 
8: Kokkos::Cuda is available.
8: Default execution space info: Device Execution Space:
8:   KOKKOS_ENABLE_CUDA: yes
8: Cuda Atomics:
8:   KOKKOS_ENABLE_CUDA_ATOMICS: no
8: Cuda Options:
8:   KOKKOS_ENABLE_CUDA_LAMBDA: yes
8:   KOKKOS_ENABLE_CUDA_LDG_INTRINSIC: yes
8:   KOKKOS_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE: yes
8:   KOKKOS_ENABLE_CUDA_UVM: yes
8:   KOKKOS_ENABLE_CXX11_DISPATCH_LAMBDA: yes
8: 
8: Cuda Runtime Configuration:
8: macro  KOKKOS_ENABLE_CUDA      : defined
8: macro  CUDA_VERSION          = 11020 = version 11.2
8: Kokkos::Cuda[ 0 ] Tesla V100-PCIE-16GB capability 7.0, Total Global Memory: 15.78 G, Shared Memory per Block: 48 K : Selected
8: Kokkos::Cuda[ 1 ] Tesla V100-PCIE-16GB capability 7.0, Total Global Memory: 15.78 G, Shared Memory per Block: 48 K
8: 
8: [       OK ] BasicKokkos.discover_execution_space (0 ms)
8: [----------] 1 test from BasicKokkos (0 ms total)
8: 
8: [----------] 9 tests from Hex8MeshWithNSOFields
8: [ RUN      ] Hex8MeshWithNSOFields.NGPElemDataRequests
8: [       OK ] Hex8MeshWithNSOFields.NGPElemDataRequests (4 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPMeshField
8: [       OK ] Hex8MeshWithNSOFields.NGPMeshField (5 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPScratchViews
8: [       OK ] Hex8MeshWithNSOFields.NGPScratchViews (8 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPAssembleElemSolver
8: SCV volume = 0.125000; expected = 0.125
8: SCV volume = 0.125000; expected = 0.125
8: SCV volume = 0.125000; expected = 0.125
8: SCV volume = 0.125000; expected = 0.125
8: SCV volume = 0.125000; expected = 0.125
8: SCV volume = 0.125000; expected = 0.125
8: SCV volume = 0.125000; expected = 0.125
8: SCV volume = 0.125000; expected = 0.125
8: [       OK ] Hex8MeshWithNSOFields.NGPAssembleElemSolver (9 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPSharedMemData
8: [       OK ] Hex8MeshWithNSOFields.NGPSharedMemData (6 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPteamSize
8: [       OK ] Hex8MeshWithNSOFields.NGPteamSize (0 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPKernelBasic
8: [       OK ] Hex8MeshWithNSOFields.NGPKernelBasic (3 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPKernelRunAlg
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: [       OK ] Hex8MeshWithNSOFields.NGPKernelRunAlg (6 ms)
8: [ RUN      ] Hex8MeshWithNSOFields.NGPKernelExecute
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: ipNodeMap[2] = 7 (7); SCS areav[2, 0] = -0.250000
8: [       OK ] Hex8MeshWithNSOFields.NGPKernelExecute (7 ms)
8: [----------] 9 tests from Hex8MeshWithNSOFields (48 ms total)
8: 
8: [----------] 1 test from NgpMesh
8: [ RUN      ] NgpMesh.NGPMesh
8: [       OK ] NgpMesh.NGPMesh (4 ms)
8: [----------] 1 test from NgpMesh (4 ms total)
8: 
8: [----------] 4 tests from PecletFunction
8: [ RUN      ] PecletFunction.NGP_classic_double
8: [       OK ] PecletFunction.NGP_classic_double (0 ms)
8: [ RUN      ] PecletFunction.NGP_classic_simd
8: [       OK ] PecletFunction.NGP_classic_simd (0 ms)
8: [ RUN      ] PecletFunction.NGP_tanh_double
8: [       OK ] PecletFunction.NGP_tanh_double (0 ms)
8: [ RUN      ] PecletFunction.NGP_tanh_simd
8: [       OK ] PecletFunction.NGP_tanh_simd (1 ms)
8: [----------] 4 tests from PecletFunction (1 ms total)
8: 
8: [----------] 4 tests from VectorSpace
8: [ RUN      ] VectorSpace.NGP_vector_create
8: [       OK ] VectorSpace.NGP_vector_create (0 ms)
8: [ RUN      ] VectorSpace.NGP_tensor_create
8: [       OK ] VectorSpace.NGP_tensor_create (0 ms)
8: [ RUN      ] VectorSpace.NGP_vector_rotations
8: [       OK ] VectorSpace.NGP_vector_rotations (0 ms)
8: [ RUN      ] VectorSpace.NGP_vector_ops
8: [       OK ] VectorSpace.NGP_vector_ops (0 ms)
8: [----------] 4 tests from VectorSpace (0 ms total)
8: 
8: [----------] 1 test from ActuatorParsingTest
8: [ RUN      ] ActuatorParsingTest.NGP_bareMinimumParse
8: [       OK ] ActuatorParsingTest.NGP_bareMinimumParse (2 ms)
8: [----------] 1 test from ActuatorParsingTest (2 ms total)
8: 
8: [----------] 2 tests from ActuatorMeta
8: [ RUN      ] ActuatorMeta.NGP_constructor
8: [       OK ] ActuatorMeta.NGP_constructor (0 ms)
8: [ RUN      ] ActuatorMeta.NGP_copyCtor
8: [       OK ] ActuatorMeta.NGP_copyCtor (0 ms)
8: [----------] 2 tests from ActuatorMeta (0 ms total)
8: 
8: [----------] 1 test from ActuatorBulk
8: [ RUN      ] ActuatorBulk.NGP_constructor
8: [       OK ] ActuatorBulk.NGP_constructor (1 ms)
8: [----------] 1 test from ActuatorBulk (1 ms total)
8: 
8: [----------] 2 tests from ActuatorNGP
8: [ RUN      ] ActuatorNGP.NGP_testExecuteOnHostOnly
8: [       OK ] ActuatorNGP.NGP_testExecuteOnHostOnly (0 ms)
8: [ RUN      ] ActuatorNGP.NGP_testExecuteOnHostAndDevice
8: [       OK ] ActuatorNGP.NGP_testExecuteOnHostAndDevice (1 ms)
8: [----------] 2 tests from ActuatorNGP (1 ms total)
8: 
8: [----------] 4 tests from ActuatorSearchTest
8: [ RUN      ] ActuatorSearchTest.NGP_createBoundingSpheres
8: [       OK ] ActuatorSearchTest.NGP_createBoundingSpheres (0 ms)
8: [ RUN      ] ActuatorSearchTest.NGP_createElementBoxes
8: [       OK ] ActuatorSearchTest.NGP_createElementBoxes (0 ms)
8: [ RUN      ] ActuatorSearchTest.NGP_executeCoarseSearch
8: [       OK ] ActuatorSearchTest.NGP_executeCoarseSearch (1 ms)
8: [ RUN      ] ActuatorSearchTest.NGP_executeFineSearch
8: [       OK ] ActuatorSearchTest.NGP_executeFineSearch (0 ms)
8: [----------] 4 tests from ActuatorSearchTest (2 ms total)
8: 
8: [----------] 2 tests from ActuatorFunctorTests
8: [ RUN      ] ActuatorFunctorTests.NGP_testSearchAndInterpolate
8: [       OK ] ActuatorFunctorTests.NGP_testSearchAndInterpolate (4 ms)
8: [ RUN      ] ActuatorFunctorTests.NGP_testSpreadForces
8: [       OK ] ActuatorFunctorTests.NGP_testSpreadForces (3 ms)
8: [----------] 2 tests from ActuatorFunctorTests (8 ms total)
8: 
8: [----------] 2 tests from ActuatorParsingSimpleTests
8: [ RUN      ] ActuatorParsingSimpleTests.NGP_minimumRequired
8: [       OK ] ActuatorParsingSimpleTests.NGP_minimumRequired (5 ms)
8: [ RUN      ] ActuatorParsingSimpleTests.NGP_oneValueEpsilonParses
8: [       OK ] ActuatorParsingSimpleTests.NGP_oneValueEpsilonParses (5 ms)
8: [----------] 2 tests from ActuatorParsingSimpleTests (10 ms total)
8: 
8: [----------] 3 tests from ActuatorFLLC
8: [ RUN      ] ActuatorFLLC.NGP_ComputeLiftForceDistribution_G_Eq_5_3
8: [       OK ] ActuatorFLLC.NGP_ComputeLiftForceDistribution_G_Eq_5_3 (1 ms)
8: [ RUN      ] ActuatorFLLC.NGP_ComputeGradG_Eq_5_4_and_5_5
8: [       OK ] ActuatorFLLC.NGP_ComputeGradG_Eq_5_4_and_5_5 (1 ms)
8: [ RUN      ] ActuatorFLLC.NGP_ComputeInducedVelocity_Eq_5_7
8: [       OK ] ActuatorFLLC.NGP_ComputeInducedVelocity_Eq_5_7 (1 ms)
8: [----------] 3 tests from ActuatorFLLC (4 ms total)
8: 
8: [----------] 4 tests from ContinuityKernelHex8Mesh
8: [ RUN      ] ContinuityKernelHex8Mesh.NGP_inflow
8: [       OK ] ContinuityKernelHex8Mesh.NGP_inflow (15 ms)
8: [ RUN      ] ContinuityKernelHex8Mesh.NGP_open_edge
8: [       OK ] ContinuityKernelHex8Mesh.NGP_open_edge (13 ms)
8: [ RUN      ] ContinuityKernelHex8Mesh.NGP_continuity_gcl_node
8: [       OK ] ContinuityKernelHex8Mesh.NGP_continuity_gcl_node (7 ms)
8: [ RUN      ] ContinuityKernelHex8Mesh.NGP_continuity_mass_node
8: [       OK ] ContinuityKernelHex8Mesh.NGP_continuity_mass_node (6 ms)
8: [----------] 4 tests from ContinuityKernelHex8Mesh (41 ms total)
8: 
8: [----------] 4 tests from EnthalpyABLKernelHex8Mesh
8: [ RUN      ] EnthalpyABLKernelHex8Mesh.NGP_tgrad_bc
8: [       OK ] EnthalpyABLKernelHex8Mesh.NGP_tgrad_bc (16 ms)
8: [ RUN      ] EnthalpyABLKernelHex8Mesh.NGP_heat_flux_bc
8: [       OK ] EnthalpyABLKernelHex8Mesh.NGP_heat_flux_bc (15 ms)
8: [ RUN      ] EnthalpyABLKernelHex8Mesh.NGP_abl_force
8: [       OK ] EnthalpyABLKernelHex8Mesh.NGP_abl_force (8 ms)
8: [ RUN      ] EnthalpyABLKernelHex8Mesh.NGP_enthalpy_eff_diff_flux_coeff
8: [       OK ] EnthalpyABLKernelHex8Mesh.NGP_enthalpy_eff_diff_flux_coeff (8 ms)
8: [----------] 4 tests from EnthalpyABLKernelHex8Mesh (47 ms total)
8: 
8: [----------] 1 test from Hex8Mesh
8: [ RUN      ] Hex8Mesh.NGP_faceElemBasic
8: [       OK ] Hex8Mesh.NGP_faceElemBasic (22 ms)
8: [----------] 1 test from Hex8Mesh (22 ms total)
8: 
8: [----------] 5 tests from MixtureFractionKernelHex8Mesh
8: [ RUN      ] MixtureFractionKernelHex8Mesh.NGP_adv_diff_edge_tpetra
8: [       OK ] MixtureFractionKernelHex8Mesh.NGP_adv_diff_edge_tpetra (20 ms)
8: [ RUN      ] MixtureFractionKernelHex8Mesh.NGP_adv_diff_edge_tpetra_fix_pressure_at_node
8: [       OK ] MixtureFractionKernelHex8Mesh.NGP_adv_diff_edge_tpetra_fix_pressure_at_node (19 ms)
8: [ RUN      ] MixtureFractionKernelHex8Mesh.NGP_adv_diff_edge_tpetra_dirichlet
8: [       OK ] MixtureFractionKernelHex8Mesh.NGP_adv_diff_edge_tpetra_dirichlet (19 ms)
8: [ RUN      ] MixtureFractionKernelHex8Mesh.NGP_scalar_gcl_node
8: [       OK ] MixtureFractionKernelHex8Mesh.NGP_scalar_gcl_node (12 ms)
8: [ RUN      ] MixtureFractionKernelHex8Mesh.NGP_scalar_mass_node
8: [       OK ] MixtureFractionKernelHex8Mesh.NGP_scalar_mass_node (14 ms)
8: [----------] 5 tests from MixtureFractionKernelHex8Mesh (84 ms total)
8: 
8: [----------] 3 tests from WallDistKernelHex8Mesh
8: [ RUN      ] WallDistKernelHex8Mesh.NGP_wall_dist
8: [       OK ] WallDistKernelHex8Mesh.NGP_wall_dist (9 ms)
8: [ RUN      ] WallDistKernelHex8Mesh.NGP_wall_dist_shifted
8: [       OK ] WallDistKernelHex8Mesh.NGP_wall_dist_shifted (9 ms)
8: [ RUN      ] WallDistKernelHex8Mesh.NGP_wall_dist_edge
8: [       OK ] WallDistKernelHex8Mesh.NGP_wall_dist_edge (8 ms)
8: [----------] 3 tests from WallDistKernelHex8Mesh (26 ms total)
8: 
8: [----------] 1 test from ContinuityEdgeHex8Mesh
8: [ RUN      ] ContinuityEdgeHex8Mesh.NGP_advection
8: [       OK ] ContinuityEdgeHex8Mesh.NGP_advection (9 ms)
8: [----------] 1 test from ContinuityEdgeHex8Mesh (9 ms total)
8: 
8: [----------] 7 tests from MomentumEdgeHex8Mesh
8: [ RUN      ] MomentumEdgeHex8Mesh.NGP_advection_diffusion
8: [       OK ] MomentumEdgeHex8Mesh.NGP_advection_diffusion (13 ms)
8: [ RUN      ] MomentumEdgeHex8Mesh.NGP_open_edge
8: [       OK ] MomentumEdgeHex8Mesh.NGP_open_edge (19 ms)
8: [ RUN      ] MomentumEdgeHex8Mesh.NGP_mdot_calc_edge
8: [       OK ] MomentumEdgeHex8Mesh.NGP_mdot_calc_edge (13 ms)
8: [ RUN      ] MomentumEdgeHex8Mesh.NGP_mdot_rho_accum
8: [       OK ] MomentumEdgeHex8Mesh.NGP_mdot_rho_accum (8 ms)
8: [ RUN      ] MomentumEdgeHex8Mesh.NGP_mdot_open_correction
8: [       OK ] MomentumEdgeHex8Mesh.NGP_mdot_open_correction (14 ms)
8: [ RUN      ] MomentumEdgeHex8Mesh.NGP_mdot_inflow
8: [       OK ] MomentumEdgeHex8Mesh.NGP_mdot_inflow (15 ms)
8: [ RUN      ] MomentumEdgeHex8Mesh.NGP_mdot_open_edge
8: [       OK ] MomentumEdgeHex8Mesh.NGP_mdot_open_edge (15 ms)
8: [----------] 7 tests from MomentumEdgeHex8Mesh (97 ms total)
8: 
8: [----------] 22 tests from SSTKernelHex8Mesh
8: [ RUN      ] SSTKernelHex8Mesh.NGP_scalar_edge_open_solver_alg
8: [       OK ] SSTKernelHex8Mesh.NGP_scalar_edge_open_solver_alg (14 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_scalar_open_edge
8: [       OK ] SSTKernelHex8Mesh.NGP_scalar_open_edge (13 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_tke_sst_node
8: [       OK ] SSTKernelHex8Mesh.NGP_tke_sst_node (8 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_tke_sst_sust_node
8: [       OK ] SSTKernelHex8Mesh.NGP_tke_sst_sust_node (11 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_tke_sstlr_node
8: [       OK ] SSTKernelHex8Mesh.NGP_tke_sstlr_node (8 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_tke_sst_des_node
8: [       OK ] SSTKernelHex8Mesh.NGP_tke_sst_des_node (8 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_tke_sst_des_sust_node
8: [       OK ] SSTKernelHex8Mesh.NGP_tke_sst_des_sust_node (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_sdr_sst_node
8: [       OK ] SSTKernelHex8Mesh.NGP_sdr_sst_node (8 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_sdr_sst_sust_node
8: [       OK ] SSTKernelHex8Mesh.NGP_sdr_sst_sust_node (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_sdr_sstlr_node
8: [       OK ] SSTKernelHex8Mesh.NGP_sdr_sstlr_node (8 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_sdr_sst_des_node
8: [       OK ] SSTKernelHex8Mesh.NGP_sdr_sst_des_node (8 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_sdr_sst_des_sust_node
8: [       OK ] SSTKernelHex8Mesh.NGP_sdr_sst_des_sust_node (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_nodal_grad_edge
8: [       OK ] SSTKernelHex8Mesh.NGP_nodal_grad_edge (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_nodal_grad_elem
8: [       OK ] SSTKernelHex8Mesh.NGP_nodal_grad_elem (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_nodal_grad_elem_shifted
8: [       OK ] SSTKernelHex8Mesh.NGP_nodal_grad_elem_shifted (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_nodal_grad_bndry
8: [       OK ] SSTKernelHex8Mesh.NGP_nodal_grad_bndry (13 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_nodal_grad_bndry_shifted
8: [       OK ] SSTKernelHex8Mesh.NGP_nodal_grad_bndry_shifted (14 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_turb_visc_sst_alg
8: [       OK ] SSTKernelHex8Mesh.NGP_turb_visc_sst_alg (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_turb_visc_sstlr_alg
8: [       OK ] SSTKernelHex8Mesh.NGP_turb_visc_sstlr_alg (7 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_sdr_wall_lowRE
8: [       OK ] SSTKernelHex8Mesh.NGP_sdr_wall_lowRE (15 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_sdr_wall_func
8: [       OK ] SSTKernelHex8Mesh.NGP_sdr_wall_func (15 ms)
8: [ RUN      ] SSTKernelHex8Mesh.NGP_SST_Max_Length_Scale
8: [       OK ] SSTKernelHex8Mesh.NGP_SST_Max_Length_Scale (6 ms)
8: [----------] 22 tests from SSTKernelHex8Mesh (205 ms total)
8: 
8: [----------] 1 test from MomentumABLKernelHex8Mesh
8: [ RUN      ] MomentumABLKernelHex8Mesh.NGP_abl_wall_func
8: [       OK ] MomentumABLKernelHex8Mesh.NGP_abl_wall_func (16 ms)
8: [----------] 1 test from MomentumABLKernelHex8Mesh (16 ms total)
8: 
8: [----------] 8 tests from MomentumKernelHex8Mesh
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_symmetry_edge
8: [       OK ] MomentumKernelHex8Mesh.NGP_symmetry_edge (16 ms)
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_symmetry_edge_penalty
8: [       OK ] MomentumKernelHex8Mesh.NGP_symmetry_edge_penalty (15 ms)
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_momentum_mass_node
8: [       OK ] MomentumKernelHex8Mesh.NGP_momentum_mass_node (9 ms)
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_courant_reynolds
8: [       OK ] MomentumKernelHex8Mesh.NGP_courant_reynolds (8 ms)
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_nodal_grad_edge_vec
8: [       OK ] MomentumKernelHex8Mesh.NGP_nodal_grad_edge_vec (9 ms)
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_nodal_grad_elem_vec
8: [       OK ] MomentumKernelHex8Mesh.NGP_nodal_grad_elem_vec (8 ms)
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_nodal_grad_elem_shifted_vec
8: [       OK ] MomentumKernelHex8Mesh.NGP_nodal_grad_elem_shifted_vec (8 ms)
8: [ RUN      ] MomentumKernelHex8Mesh.NGP_nodal_grad_bndry_elem_vec
8: [       OK ] MomentumKernelHex8Mesh.NGP_nodal_grad_bndry_elem_vec (15 ms)
8: [----------] 8 tests from MomentumKernelHex8Mesh (88 ms total)
8: 
8: [----------] 5 tests from AMSKernelHex8Mesh
8: [ RUN      ] AMSKernelHex8Mesh.NGP_ams_diff
8: [       OK ] AMSKernelHex8Mesh.NGP_ams_diff (9 ms)
8: [ RUN      ] AMSKernelHex8Mesh.NGP_tke_ams_node
8: [       OK ] AMSKernelHex8Mesh.NGP_tke_ams_node (7 ms)
8: [ RUN      ] AMSKernelHex8Mesh.NGP_sdr_ams_node
8: [       OK ] AMSKernelHex8Mesh.NGP_sdr_ams_node (8 ms)
8: [ RUN      ] AMSKernelHex8Mesh.NGP_ams_forcing
8: cudaEventRecord(CudaInternal::constantMemReusable, cudaStream_t(cuda_instance->m_stream)) error( cudaErrorIllegalAddress): an illegal memory access was encountered /fgs/rabartl/SpackManager.base/spack-manager/spack/opt/spack/linux-rhel7-x86_64/gcc-9.3.0/trilinos-develop-wmhhsdfv3on6ahzhepsrxn7373eakdlx/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:561
8: Backtrace:
8:                        [0x65f6c03]
8:                        [0x65eddc8]
8:                        [0x65eddfb]
8:                        [0x66009ed]
8:                         [0xd90a95]
8:                         [0xa5dbf0]
8:                        [0x663292d]
8:                        [0x6620be0]
8:                        [0x6621022]
8:                        [0x6621202]
8:                        [0x66291ad]
8:                        [0x66296fb]
8:                         [0x5e6cbe]
8: __libc_start_main [0x7f50e2b72555]
8:                         [0x790c68]
8: 
8: ===================================================================================
8: =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
8: =   PID 59717 RUNNING AT ascicgpu16
8: =   EXIT CODE: 6
8: =   CLEANING UP REMAINING PROCESSES
8: =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
8: ===================================================================================
8: YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
8: This typically refers to a problem with your application.
8: Please see the FAQ page for debugging suggestions
1/1 Test #8: unitTestGPU ......................***Failed    4.75 sec

0% tests passed, 1 tests failed out of 1

Label Time Summary:
unit    =   4.75 sec*proc (1 test)

Total Test time (real) =   4.76 sec

The following tests FAILED:
	  8 - unitTestGPU (Failed)
Errors while running CTest

As described in #10954 (comment), this segfault is expected.

Therefore, the updated Trilinos branch '10954-nalu-wind-fix-cuda' would seem to be validated against Nalu-Wind. [PASSED]

@bartlettroscoe
Change to use find_package(CUDAToolkit) (trilinos#10954)
97961c7 
NOTE: This requires the matching change to the TriBITS file
tribits/core/std_tpls/FindTPLCUDA.cmake.
@bartlettroscoe
Automatic snapshot commit from tribits at 23dc20b9
082ee03 
Origin repo remote tracking branch: 'github/master'
Origin repo remote repo URL: 'github = git@github.com:TriBITSPub/TriBITS.git'
Git describe: Vera4.0-RC1-start-1292-g6d3bb5b3

At commit:

commit 23dc20b901ab55943b71e51f5e64a244ad186b5a
Author:  Roscoe A. Bartlett <rabartl@sandia.gov>
Date:    Thu Sep 29 11:46:47 2022 -0600
Summary: Change to use find_package(CUDAToolkit) (trilinos#10954)
@bartlettroscoe
Merge branch 'tribits_github_snapshot' into 10954-nalu-wind-fix-cuda (t…
a2eded0 
…rilinos#10954)
@bartlettroscoe bartlettroscoe added type: bug The primary issue is a bug in Trilinos code or tests type: enhancement Issue is an enhancement, not a bug impacting: configure or build The issue is primarily related to configuring or building TriBITS Issues with the TriBITS framework itself, not usage of the TriBITS framework PA: Framework Issues that fall under the Trilinos Framework Product Area AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration labels Oct 3, 2022
@bartlettroscoe bartlettroscoe mentioned this pull request Oct 3, 2022
Closed
@bartlettroscoe
Copy link
Member Author

bartlettroscoe commented Oct 3, 2022 •
edited
Loading

FYI: I will post my detailed testing results above shortly ...

Update: I posted the detailed testing results with complete reproduction info above.

@bartlettroscoe bartlettroscoe mentioned this pull request Oct 3, 2022
Closed
14 tasks
@bartlettroscoe bartlettroscoe self-assigned this Oct 3, 2022
rppawlo
rppawlo previously approved these changes Oct 3, 2022
View reviewed changes
@bartlettroscoe bartlettroscoe changed the base branch from master to develop October 3, 2022 22:10
@bartlettroscoe bartlettroscoe dismissed rppawlo’s stale review October 3, 2022 22:10

The base branch was changed.

@bartlettroscoe
Copy link
Member Author

@rppawlo, can you please re-approve? I accidentally had the target as 'master' instead of 'develop'. I fixed that and now it requires re-approval.

@bartlettroscoe bartlettroscoe added the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Oct 3, 2022
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 1175
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

  • Build Num: 1173
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

  • Build Num: 1172
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_intel-17.0.1

  • Build Num: 1172
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_clang-10.0.0

  • Build Num: 1188
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

  • Build Num: 1189
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL vortex
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 985
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 709
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)
  • Branch: 10954-nalu-wind-fix-cuda
  • SHA: a2eded0
  • Mode: TEST_REPO

Pull Request Author: bartlettroscoe

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 1175
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

  • Build Num: 1173
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

  • Build Num: 1172
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_intel-17.0.1

  • Build Num: 1172
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_clang-10.0.0

  • Build Num: 1188
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

  • Build Num: 1189
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL vortex
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 985
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 709
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2


CDash Test Results for PR# 11093.


Wiki: How to Reproduce PR Testing Builds and Errors.

@trilinos-autotester trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Oct 4, 2022
@bartlettroscoe
Copy link
Member Author

Well shoot, the dreaded mass jsrun failures described in #6861 caused the last PR testing iteration above. Just my luck.

@bartlettroscoe bartlettroscoe added the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Oct 4, 2022
@bartlettroscoe bartlettroscoe mentioned this pull request Oct 4, 2022
Merged
bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this pull request Oct 4, 2022
@bartlettroscoe
Merge remote-tracking branch 'rab-github/10954-nalu-wind-fix-cuda' in…
3def912 
…to 11002-enable-disable-tests-examples (trilinos#11093)

I have to merge branch '10954-nalu-wind-fix-cuda' into this branch which has
the updated snapshot of TriBITS that contains the updated FindTPLCUDA.cmake
module but does not have the matching changes to FindTPLCUBLAS.cmake and
FindTPLCUSPARSE.cmake from PR trilinos#11093 to make it work.
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 1180
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

  • Build Num: 1178
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

  • Build Num: 1177
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_intel-17.0.1

  • Build Num: 1177
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_clang-10.0.0

  • Build Num: 1193
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

  • Build Num: 1194
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL vortex
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 989
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 714
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)
  • Branch: 10954-nalu-wind-fix-cuda
  • SHA: a2eded0
  • Mode: TEST_REPO

Pull Request Author: bartlettroscoe

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 1180
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

  • Build Num: 1178
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

  • Build Num: 1177
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_intel-17.0.1

  • Build Num: 1177
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_clang-10.0.0

  • Build Num: 1193
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

  • Build Num: 1194
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL vortex
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 989
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 714
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2


CDash Test Results for PR# 11093.


Wiki: How to Reproduce PR Testing Builds and Errors.

@trilinos-autotester trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Oct 4, 2022
@bartlettroscoe
Copy link
Member Author

bartlettroscoe commented Oct 4, 2022 •
edited
Loading

Wow, incredible, the last PR testing iteration above failed again due to mass random failures as shown here failed due to the known system errors on ATS-2 'vortex':

as shown in these queries:

I will put AT: RETEST again and cross fingers.

@bartlettroscoe bartlettroscoe added the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Oct 4, 2022
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 1192
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

  • Build Num: 1190
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

  • Build Num: 1189
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_intel-17.0.1

  • Build Num: 1189
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_clang-10.0.0

  • Build Num: 1205
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

  • Build Num: 1206
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL vortex
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 999
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 726
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)
  • Branch: 10954-nalu-wind-fix-cuda
  • SHA: a2eded0
  • Mode: TEST_REPO

Pull Request Author: bartlettroscoe

@bartlettroscoe
Copy link
Member Author

bartlettroscoe commented Oct 5, 2022 •
edited
Loading

After this last PR iteration started above completes (and likely fails), I am going to stop putting AT: RETEST on this PR. Instead, I am going to just put AT: RETEST on the PR #11099 since that PR has this PR branch merged into it. Therefore, when PR #11099 merges, so will this PR.

@bartlettroscoe bartlettroscoe mentioned this pull request Oct 5, 2022
Closed
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_PR_gcc-8.3.0

  • Build Num: 1192
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-8.3.0-openmpi-1.10.1-openmp_release-debug_static_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-serial

  • Build Num: 1190
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_gcc-7.2.0-debug

  • Build Num: 1189
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_intel-17.0.1

  • Build Num: 1189
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_clang-10.0.0

  • Build Num: 1205
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL trilinos-any
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-10.1.243

  • Build Num: 1206
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME ats2_cuda-10.1.243-gnu-8.3.1-spmpi-rolling_release_static_Volta70_Power9_no-asan_no-complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL vortex
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_python3

  • Build Num: 999
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-gnu-7.2.0-anaconda3-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_no-mpi_no-pt_no-rdc_no-uvm_deprecated-on_pr-framework
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL ascic
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2

Build Information

Test Name: Trilinos_PR_cuda-11.4.2-uvm-off

  • Build Num: 726
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
FORCE_CLEAN true
GENCONFIG_BUILD_NAME rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables
PR_LABELS type: bug;type: enhancement;impacting: configure or build;AT: RETEST;AT: AUTOMERGE;TriBITS;PA: Framework
PULLREQUESTNUM 11093
PULLREQUEST_CDASH_TRACK Pull Request
TEST_REPO_ALIAS TRILINOS
TRILINOS_NODE_LABEL GPU
TRILINOS_SOURCE_BRANCH 10954-nalu-wind-fix-cuda
TRILINOS_SOURCE_REPO https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA a2eded0
TRILINOS_SRN_CONFIG true
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 85a5ff2


CDash Test Results for PR# 11093.

@trilinos-autotester trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Oct 5, 2022
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ rppawlo tasmith4 ]!

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

@trilinos-autotester trilinos-autotester merged commit 78c0387 into trilinos:develop Oct 5, 2022
@trilinos-autotester
Copy link
Contributor

Merge on Pull Request# 11093: IS A SUCCESS - Pull Request successfully merged

@trilinos-autotester trilinos-autotester removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Oct 5, 2022
Trilinos TriBITS Refactor automation moved this from In Review to Done Oct 5, 2022
@bartlettroscoe bartlettroscoe mentioned this pull request Oct 6, 2022
Closed
11 tasks
@bartlettroscoe bartlettroscoe mentioned this pull request Dec 6, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rppawlo rppawlo rppawlo approved these changes

@tasmith4 tasmith4 tasmith4 approved these changes

@ibaned ibaned Awaiting requested review from ibaned

@e10harvey e10harvey Awaiting requested review from e10harvey

Assignees

@bartlettroscoe bartlettroscoe

Labels
impacting: configure or build The issue is primarily related to configuring or building PA: Framework Issues that fall under the Trilinos Framework Product Area TriBITS Issues with the TriBITS framework itself, not usage of the TriBITS framework type: bug The primary issue is a bug in Trilinos code or tests type: enhancement Issue is an enhancement, not a bug
Projects
Trilinos TriBITS Refactor
Status: Done
Trilinos TriBITS Refactor
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@bartlettroscoe @trilinos-autotester @rppawlo @tasmith4