Fix compilation warnings regarding std::fn on device by bartgol · Pull Request #70 · E3SM-Project/EKAT

bartgol · 2020-10-21T17:57:29Z

Motivation

On CUDA, the compilation of pack math functions caused a bunch of warnings, regarding std::blah functions not being a device fcn but being called inside a __host__ __device__ function. I am not 100% sure why this warning generated, and yet was not a real problem.

This PR fixes the warning, by providing 3 implementation of all the math functions, for double, float, and integer types.

Related Issues

Closes Fix warning regarding calling host function from device in ekat_pack.hpp #8

Testing

These modification affect only pack operations implementations, which continue to be tested as they were before.

The error appeared to be benign, but it generated several warnings. To get around it, we added the MathFcn class, which dispatches the function call to the proper implementation (cuda vs std).

For some reason this never caused compilation errors, but it started after the previous commit.

E3SM-Bot · 2020-10-21T18:13:57Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

E3SM-Bot · 2020-10-21T18:17:34Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Build Num: 94
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`13d6b89`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	AT: AUTOMERGE;Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Build Num: 89
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`13d6b89`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	AT: AUTOMERGE;Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Build Num: 12
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`13d6b89`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	AT: AUTOMERGE;Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Using Repos:

Repo: EKAT (E3SM-Project/EKAT)

Pull Request Author: bartgol

E3SM-Bot · 2020-10-21T18:24:26Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Build Num: 94
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`13d6b89`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	AT: AUTOMERGE;Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Build Num: 89
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`13d6b89`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	AT: AUTOMERGE;Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Build Num: 12
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`13d6b89`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	AT: AUTOMERGE;Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

E3SM-Bot · 2020-10-21T18:24:44Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

E3SM-Bot · 2020-10-21T18:24:51Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

bartgol · 2020-10-21T18:52:32Z

@jgfouca @ambrad I pinged you two cause you implemented most of the pack stuff. Feel free to take your name off if you'd rather pass.

ambrad · 2020-10-21T18:55:55Z

This moves some code and then for testing only replaces a macro with a function? Or is there more?

bartgol · 2020-10-21T19:00:59Z

This moves some code and then for testing only replaces a macro with a function? Or is there more?

I moved the pack math overloads to their own file, cause ekat_pack.hpp was a bit long. For testing, nothing has changed, it's the same as before. What changed is how the different math fcns overloads are generated: the macro instantiating the pack overloads now calls MathFcn<ScalarT>::fcn, which takes care of doing a different impl (std::blah vs blah or blahf) depending on a) ScalarT, and b) whether we're inside a CUDA kernel.

ambrad · 2020-10-21T19:12:53Z

Sorry for being dense, but is this introducing one extra level of function call for things like pow?

E3SM-Bot · 2020-10-21T19:13:56Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO REVIEWS HAVE BEEN PERFORMED ON THIS PULL REQUEST!

E3SM-Bot · 2020-10-21T19:14:03Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

bartgol · 2020-10-21T19:26:16Z

@ambrad yes, although it should be inlined (I may replace KOKKOS_INLINE_FUNCTION with KOKKOS_FORCEINLINE_FUNCTION if you prefer).

ambrad · 2020-10-21T19:38:20Z

Ok. Yes, please at least use KOKKOS_FORCEINLINE_FUNCTION. We have clear evidence from studies ~2 years ago that if one isn't very careful, inline isn't enough and non-inlined function calls can destroy vectorization.

Also, I don't see why an extra fn call is necessary. Why not just use macros whose def depends on CUDA_ARCH? This is super low-level code, so worrying about things like fn calls destroying vectorization is important.

bartgol · 2020-10-21T19:39:34Z

To give more context: things were also (almost) working with something like this


#ifdef __CUDA_ARCH__
#define ekat_pack_gen_unary_stdfn(fn)               \
  template <typename ScalarT, int N>                \
  KOKKOS_INLINE_FUNCTION                            \
  Pack<ScalarT,N> fn (const Pack<ScalarT,N>& p) {   \
    Pack<ScalarT,N> s;                              \
    vector_simd                                     \
    for (int i = 0; i < N; ++i) {                   \
      s[i] = fn(p[i]);                              \
    }                                               \
    return s;                                       \
  }
#else
#define ekat_pack_gen_unary_stdfn(fn)               \
  template <typename ScalarT, int N>                \
  KOKKOS_INLINE_FUNCTION                            \
  Pack<ScalarT,N> fn (const Pack<ScalarT,N>& p) {   \
    Pack<ScalarT,N> s;                              \
    vector_simd                                     \
    for (int i = 0; i < N; ++i) {                   \
      s[i] = std::fn(p[i]);                         \
    }                                               \
    return s;                                       \
  }
#endif

since cuda overloads blah for both double and float. However, the compilation would crap out when trying to test on int/long.

Edit: that's why I was asking on slack about ints. The only workaround I could find was to explicitly cast integers to double, but to do that without messing up the sp build, I had to do multiple impl. And partial specialization wasn't possible, so in the end I used a helper struct for the specialization of the core function on integer/floating.

jgfouca · 2020-10-21T19:43:18Z

Just a semi-related question, if KOKKOS_FORCEINLINE_FUNCTION is more reliable than KOKKOS_INLINE_FUNCTION, should we just always use KOKKOS_FORCEINLINE? If not, when to prefer one over the other?

E3SM-Bot · 2020-10-21T19:43:56Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ jgfouca ]!

E3SM-Bot · 2020-10-21T19:44:03Z

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

bartgol · 2020-10-21T19:50:10Z

Just a semi-related question, if KOKKOS_FORCEINLINE_FUNCTION is more reliable than KOKKOS_INLINE_FUNCTION, should we just always use KOKKOS_FORCEINLINE? If not, when to prefer one over the other?

I think the point is that you don't want to inline the whole library. For key functions, you want to be sure it is inlined, e.g., for vectorization reasons, as Andrew pointed out. But in other less crucial cases, it's probably best to let the compiler decide.

ambrad · 2020-10-21T19:54:44Z

@jgfouca, I agree strongly with Luca. FORCEINLINE must be used very carefully. It's pretty much meant only for dealing with things like this.

- use FORCEINLINE instead of INLINE (crucial for vectorization) - use only 2 specializations, since blah(x) on CUDA works for for both double and float.

E3SM-Bot · 2020-10-21T20:14:26Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

E3SM-Bot · 2020-10-21T20:17:29Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Build Num: 95
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`11c524b`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Build Num: 90
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`11c524b`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Build Num: 13
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`11c524b`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Using Repos:

Repo: EKAT (E3SM-Project/EKAT)

Pull Request Author: bartgol

E3SM-Bot · 2020-10-21T20:24:05Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Build Num: 95
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`11c524b`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Build Num: 90
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`11c524b`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Build Num: 13
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`11c524b`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

E3SM-Bot · 2020-10-21T20:24:23Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

E3SM-Bot · 2020-10-21T20:24:29Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

bartgol · 2020-10-21T20:40:24Z

Ok, I removed testing of math functions for int/long. Math fcns calls on packs of integers will result in a compiler error on cuda (except for abs, which is overloaded for int/long). This also removed the need for MathFcn, and the discussion on FORCEINLINE.

Note: I did however replace INLINE with FORCEINLINE on the max/min functions in ekat_math_utils.hpp, since it is possible they might get called inside vectorizable loops.

This comes at the (small) price of not providing overloads of standard math functions for packs of integers.

E3SM-Bot · 2020-10-21T20:43:57Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

E3SM-Bot · 2020-10-21T20:44:49Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Build Num: 96
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`d9a0710`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Build Num: 91
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`d9a0710`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Build Num: 14
Status: STARTED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`d9a0710`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Using Repos:

Repo: EKAT (E3SM-Project/EKAT)

Pull Request Author: bartgol

E3SM-Bot · 2020-10-21T20:51:20Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Build Num: 96
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`d9a0710`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Build Num: 91
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`d9a0710`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Build Num: 14
Status: PASSED

Jenkins Parameters

Parameter Name	Value
EKAT_SOURCE_BRANCH	Fix-8
EKAT_SOURCE_REPO	https://github.com/E3SM-Project/EKAT
EKAT_SOURCE_SHA	`d9a0710`
EKAT_TARGET_BRANCH	master
EKAT_TARGET_REPO	https://github.com/E3SM-Project/EKAT
EKAT_TARGET_SHA	`b66923d`
PR_LABELS	Packs;code cleanup
PULLREQUESTNUM	70
TEST_REPO_ALIAS	EKAT

E3SM-Bot · 2020-10-21T20:51:37Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ jgfouca ambrad ]!

E3SM-Bot · 2020-10-21T20:51:45Z

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

bartgol added 2 commits October 21, 2020 11:50

Fixing nvcc warning regarding std::fn on device

b2353b1

The error appeared to be benign, but it generated several warnings. To get around it, we added the MathFcn class, which dispatches the function call to the proper implementation (cuda vs std).

Fixed use of Unmanaged in kokkos utils

13d6b89

For some reason this never caused compilation errors, but it started after the previous commit.

bartgol added AT: AUTOMERGE pkg: pack Related to the pack package code cleanup labels Oct 21, 2020

bartgol self-assigned this Oct 21, 2020

bartgol requested review from ambrad and jgfouca October 21, 2020 18:51

bartgol removed the AT: AUTOMERGE label Oct 21, 2020

jgfouca previously approved these changes Oct 21, 2020

View reviewed changes

bartgol marked this pull request as draft October 21, 2020 19:47

bartgol dismissed jgfouca’s stale review via 11c524b October 21, 2020 20:04

Couple of changes to MathFcn

11c524b

- use FORCEINLINE instead of INLINE (crucial for vectorization) - use only 2 specializations, since blah(x) on CUDA works for for both double and float.

bartgol mentioned this pull request Oct 21, 2020

update ekat E3SM-Project/scream#696

Merged

bartgol marked this pull request as ready for review October 21, 2020 20:40

Removed MathFcn struct

d9a0710

This comes at the (small) price of not providing overloads of standard math functions for packs of integers.

ambrad approved these changes Oct 21, 2020

View reviewed changes

jgfouca approved these changes Oct 21, 2020

View reviewed changes

bartgol merged commit 20ab2e7 into master Oct 21, 2020

bartgol deleted the Fix-8 branch October 21, 2020 20:57

Conversation

bartgol commented Oct 21, 2020

Motivation

Related Issues

Testing

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Jenkins Parameters

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

Uh oh!

E3SM-Bot commented Oct 21, 2020

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Jenkins Parameters

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Jenkins Parameters

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

bartgol commented Oct 21, 2020

Uh oh!

ambrad commented Oct 21, 2020

Uh oh!

bartgol commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ambrad commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

bartgol commented Oct 21, 2020

Uh oh!

ambrad commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartgol commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgfouca commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

bartgol commented Oct 21, 2020

Uh oh!

ambrad commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Uh oh!

E3SM-Bot commented Oct 21, 2020

Build Information

Test Name: EKAT_PullRequest_Autotester_Blake

Jenkins Parameters

Build Information

Test Name: EKAT_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: EKAT_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

Uh oh!

bartgol commented Oct 21, 2020 •

edited

Loading

ambrad commented Oct 21, 2020 •

edited

Loading

bartgol commented Oct 21, 2020 •

edited

Loading