Tpetra::MultiVector unit tests fail for Scalar=complex<double>, likely due to kokkos-kernels changes

[mhoemmen]

https://testing.sandia.gov/cdash/testDetails.php?test=55163267&build=3970561

The following tests FAILED:
    1. MultiVector_double_default_local_ordinal_type_default_global_ordinal_type_Kokkos_Compat_KokkosSerialWrapperNode_ComplexDotOneColumn_UnitTest ... 
    56. MultiVector_int_int_std_complex0double0_Kokkos_Compat_KokkosSerialWrapperNode_NonContigView_UnitTest ... 

Further inspection shows large relative errors, so this isn't just an issue of needing to adjust the tolerance for complex numbers. When I run the tests locally on a Mac, I get a segfault.

I'm curious if this is related to the recent kokkos-kernels update.

@trilinos/tpetra

Motivation and Context

Salinas depends on Tpetra with complex Scalar types.

[mhoemmen]

Bisection: This commit passes the test:

commit 69de6b8e9c07b26be0c71afd2b6f076dacb111ed (HEAD)
Author: Mark Hoemmen <mhoemme@sandia.gov>
Date:   Wed Sep 19 12:02:20 2018 -0600

    Belos: Fix "native" Tpetra GMRES solver for Teuchos_ENABLE_COMPLEX=OFF

    @trilinos/belos

[mhoemmen]

This commit fails; it is the next commit in develop right after the one mentioned above:

commit fbd9a3c4577a661331847a51620795e1b228e946 (HEAD)
Merge: 64396a9c7b f6d4f8a351
Author: kyungjoo-kim <kyukim@sandia.gov>
Date:   Wed Sep 19 13:54:29 2018 -0500

    Merge pull request #3458 from kyungjoo-kim/issue-3438

    KokkosKernels - remove cusparse testing in spgemm.

[kyungjoo-kim]

Really ? As far as I remembered, I just changed testing scope in kokkoskernels testing directory. Oh.... I cannot believe how this could possibly affect other testings.

[kyungjoo-kim]

FYI

[mhoemmen]

@kyungjoo-kim It looks like there's something off with the dot products -- does kokkos-kernels test with complex scalar types normally?

[ndellingwood]

@mhoemmen The changes made in PR #3438 (SHA fbd9a3c of failed commit above) removed cusparse testing in kokkos-kernels spgemm unit tests, there were no internal code changes.

[kyungjoo-kim]

@mhoemmen I looked up cdash with complex keywords and I cannot find a running test with complex. Could you double check if there is any cdash test with complex for KokkosKernels ?

@crtrott Do we have KK jenkins tests running with complex ? I cannot access jenkins-son.sandia.gov to check that.

[mhoemmen]

@kyungjoo-kim Here is a build with complex enabled:

https://testing.sandia.gov/cdash/testDetails.php?test=55163267&build=3970561

Here are the build details:

https://testing.sandia.gov/cdash/buildSummary.php?buildid=3970561

I was able to get the test to fail on my Mac laptop with Clang and OpenMPI 1.10.x, with just TpetraCore enabled and Trilinos_ENABLE_COMPLEX_DOUBLE=ON.

[kyungjoo-kim]

@mhoemmen Yes I see the test but I did not see a complex test from KokkosKernels. It only includes Tpetra tests. Checking other tests, I cannot find a KokkosKernels complex test running on cdash.

[mhoemmen]

@kyungjoo-kim Hm, are there kokkos-kernels tests at all? I'm worried about lack of complex arithmetic coverage for Trilinos tests in general, esp. given that Salinas depends on it. Does EMPIRE also care?

[mhoemmen]

Is this relevant? #3336

[srajama1]

@ndellingwood : Can we enable complex on Kokkos jenkins tests. I don't think we do.

This is the first time I hear Salinas using Tpetra and thereby Kokkos Kernels. Is this a new dependency ?

[ndellingwood]

@srajama1 yeah, I'll work with Christian to learn how to do this later this week when he has a bit of time.

[mhoemmen]

Sierra has been building with complex arithmetic for years, due only to Salinas. Also I'd really not like to have correctness regressions :)

[mhoemmen]

@prwolfe @rrdrake Please don't update your version of Trilinos until we figure this out. It's likely this will break Salinas.

@micahahoward FYI; I don't think this affects your code but it may be a correctness issue.

[mhoemmen]

@kddevin @kyungjoo-kim

I did some more investigation into the ComplexDotOneColumn test in tpetra/core/test/MultiVector/MultiVector_UnitTests.cpp. When I replace the following call to KokkosBlas::dot in multiVectorSingleColumnDot with a hand-rolled loop, I got that test to pass:

Trilinos/packages/tpetra/core/src/Tpetra_MultiVector_def.hpp

Line 1917 in eaed9de

lclDot = KokkosBlas::dot (x_1d, y_1d);

I noticed the following build warnings too:

.../Trilinos/packages/kokkos-kernels/src/impl/tpls/KokkosBlas1_dot_tpl_spec_decl.hpp:54:33: warning: 'zdotu_' has C-linkage specified, but returns user-defined type 'std::complex<double>' which is
      incompatible with C [-Wreturn-type-c-linkage]
extern "C" std::complex<double> zdotu_( const int* N, const std::complex<double>* x, const int* x_inc,
                                ^
.../Trilinos/packages/kokkos-kernels/src/impl/tpls/KokkosBlas1_dot_tpl_spec_decl.hpp:56:33: warning: 'cdotu_' has C-linkage specified, but returns user-defined type 'std::complex<float>' which is
      incompatible with C [-Wreturn-type-c-linkage]
extern "C" std::complex<float>  cdotu_( const int* N, const std::complex<float>* x, const int* x_inc,
                                ^

Normally, these warnings are harmless. However, they reminded me of some issues we saw several years ago in Teuchos, relating to calling Fortran functions that return complex numbers. Fortran functions don't necessarily return complex numbers in the way that the C ABI expects. This has nothing to do with input arrays of complex; it just has to do with functions that return a complex number.

There are at least two possible fixes:

1. Wrap the Fortran functions in a Fortran wrapper that "returns" the complex result as an output argument.
2. Try to guess the Fortran ABI for returning complex numbers.
3. Don't call these Fortran functions; call the generic implementations in the KokkosBlas instead.

Option (1) requires a Fortran compiler. Option (2) is risky, because there may be different conventions for different compilers and/or systems. Option (3) is easy, but may lose some performance.