Cuda 9 and intrepid2 calls from Panzer.

[bathmatt]

I'm getting kokkos errors I've never seen compile intrepid2 code with cuda 9. I'm not sure if this is a known issue and not sure where to document this..

It is dying on a panzer file compile. Has anyone seen this? I can't get cuda-gdb to work with panzer so I updated to 9 and now I can't compile trilinos

/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:130: error: expansion pattern ‘Kokkos::LayoutStride’ contains no argument packs/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:148: error: expansion pattern ‘ExecSpaceType’ contains no argument packs
/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:151: error: template argument 2 is invalid
/home/mbetten/Trilinos/Trilinos/packages/intrepid2/refactor/src/Discretization/Basis/Intrepid2_Basis.hpp:325:151: error: template argument 3 is invalid

is part of the actual error stream,

[ibaned]

@bathmatt I recommend you open a Trilinos version of this issue and mention @rppawlo or other Panzer developers. I don't think CUDA 9 is part of our integration testing for Trilinos, in fact we have an open issue #1174 to start testing just Kokkos with it nightly.

[rppawlo]

@mperego

[ibaned]

Moved this to trilinos/Trilinos#1928 until we determine that Kokkos is the one that needs to change. If thats the case, please reopen this issue.

[ndellingwood]

@kyungjoo-kim created an isolated reproducer requiring only Kokkos of the issue seen in Intrepid2, documented in these Trilinos issues: trilinos/Trilinos#1976 and trilinos/Trilinos#1928

With the code pasted below from @kyungjoo-kim , the same compilation failure seen in Intrepid2 can be reproduced by compiling with a Cuda 9.0 and gcc/4.9.3 compiler (for example using the matching devpack on Hansen) along with the master branch of Kokkos.
This code compiles properly when using the develop branch of Kokkos.

#include "Kokkos_Core.hpp"
#include "Kokkos_DynRankView.hpp"

template<typename SpT, typename VT>
struct my_fake_basis {
  typedef Kokkos::DynRankView<VT,SpT> output_view_type;
  typedef Kokkos::DynRankView<VT,SpT> input_view_type;

  my_fake_basis() = default;
  void get_values(output_view_type A) {
    printf("this is okay\n\n");
  }

// after uncomment the following codes, it shows compiler error
//  void get_values(output_view_type A,
//                  const input_view_type B) {
//    printf("this is NOT okay with cuda 9\n\n");
//  }
};

template<typename SpT, typename VT>
void my_test() {
  my_fake_basis<SpT,VT> basis;
}

int main(int argc, char *argv[]) {
  Kokkos::initialize();
  my_test<Kokkos::Serial,double>();
  Kokkos::finalize();
  return 0;
}

// error message
// error: expansion pattern ‘SpT’ contains no argument packs

@crtrott @ibaned @mhoemmen @bathmatt

[ndellingwood]

Intrepid2 will compile with Cuda 9 and gcc/4.9.3 when using the current Kokkos develop branch (sha f27d189) along with some modifications to Sacado to remove DynRankView-related items from the Experimental namespace, reflecting similar updates to Kokkos tracked by kokkos/kokkos#1293. Quite a few warning outputs.

A patch of the Sacado changes is attached (created using git diff from the Trilinos base directory, using git apply command with the patch should apply them).

The Kokkos develop branch was tested by doing the following:

1. Add a symbolic link to the kokkos repo (with develop branch checked out) in Trilinos' base directory
2. Add the following line to configure file to override the kokkos package in Trilinos:
   -D Kokkos_SOURCE_DIR_OVERRIDE:STRING=kokkos \
3. Apply the attached patch to remove the Experimental namespace from Sacado's DynRankView codes.
   sacado_rm_exp_drv.patch.txt

[ndellingwood]

Though the above comment is related to Trilinos, the attached patch is relevant for the next Kokkos promotion testing.

[ibaned]

This looks related to #1306

[kyungjoo-kim]

@ndellingwood Thank you for checking this. It is lucky that it compiles with the Kokkos develop branch.

When we examine the difference between the master and develop branches, we are not able to find out obvious difference between them, which means that we are just lucky and unfortunately this problem may happen again when some white spaces in the code are modified.

It looks like that higher version of gcc + cuda 9 works fine. Maybe Intrepid2 cmake setup should include a check to see if the above code is compiled. If not, recommend to use higher version of compilers.

[ibaned]

@ndellingwood thanks for the Sacado patch, that will definitely come in handy later.