icb_arpack_cpp test FAILs on ppc64le on Fedora

[rathann]

Expected behavior

All tests in the testsuite should PASS.

Actual behavior

icb_arpack_cpp test FAILs on ppc64le on Fedora.

Where/how to reproduce the problem

• arpack-ng: 3.7.0
• OS: Fedora 32/rawhide
• compiler: gcc 9.2.1
• environment:

+ CFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mcpu=power8 -mtune=power8 -fasynchronous-unwind-tables -fstack-clash-protection'
+ export CFLAGS
+ CXXFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mcpu=power8 -mtune=power8 -fasynchronous-unwind-tables -fstack-clash-protection'
+ export CXXFLAGS
+ FFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mcpu=power8 -mtune=power8 -fasynchronous-unwind-tables -fstack-clash-protection -I/usr/lib64/gfortran/modules'
+ export FFLAGS
+ FCFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mcpu=power8 -mtune=power8 -fasynchronous-unwind-tables -fstack-clash-protection -I/usr/lib64/gfortran/modules'
+ export FCFLAGS
+ LDFLAGS='-Wl,-z,relro   -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'
+ export LDFLAGS

• configure:

./configure --build=ppc64le-redhat-linux-gnu --host=ppc64le-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-static --with-blas=-lopenblasp --with-lapack=-lopenblasp --enable-icb

• input data: N/A

Steps to reproduce the problem

make check

Error message

...
ERROR: iparam[4] 9, nev 9, info -8
terminate called after throwing an instance of 'std::domain_error'
  what():  Error inside ARPACK routines
FAIL icb_arpack_cpp (exit status: 134)

Traces

~/arpack-ng> tail -n 300 test-suite.log

Callstack

...
ERROR: iparam[4] 1, nev 9, info 1
terminate called after throwing an instance of 'std::domain_error'
  what():  Error inside ARPACK routines

Thread 1 "lt-icb_arpack_c" received signal SIGABRT, Aborted.
0x00007ffff6ba6ed8 in raise () from /lib64/libc.so.6
 libgcc-9.2.1-1.fc32.3.ppc64le libgfortran-9.2.1-1.fc32.3.ppc64le libquadmath-9.2.1-1.fc32.3.ppc64le libstdc++-9.2.1-1.fc32.3.ppc64le openblas-threads-0.3.7-1.fc31.ppc64le
(gdb) where
Missing separate debuginfos, use: dnf debuginfo-install#0  0x00007ffff6ba6ed8 in raise () from /lib64/libc.so.6
#1  0x00007ffff6b87b4c in abort () from /lib64/libc.so.6
#2  0x00007ffff6fa39e8 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6
#3  0x00007ffff6f9ed04 in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
#4  0x00007ffff6f9edc0 in std::terminate() () from /lib64/libstdc++.so.6
#5  0x00007ffff6f9f290 in __cxa_throw () from /lib64/libstdc++.so.6
#6  0x0000000100003064 in complex_symmetric_runner<float> () at /usr/include/c++/9/ostream:621
#7  0x0000000100001984 in main () at icb_arpack_cpp.cpp:205
(gdb)

Notes, remarks

Callstack is from a diffent machine than the build log.

[fghoussen]

ERROR: iparam[4] 9, nev 9, info -8

info -8 <=> lapack trown from lapack. Can you try with another lapack ?

[fghoussen]

BTW, this "fix" may make the test PASS (same check but no more than icb_arpack_c checks)

~\arpack-ng> git diff
diff --git a/TESTS/icb_arpack_cpp.cpp b/TESTS/icb_arpack_cpp.cpp
index 0e669a6..b467f39 100644
--- a/TESTS/icb_arpack_cpp.cpp
+++ b/TESTS/icb_arpack_cpp.cpp
@@ -73,7 +73,7 @@ void real_symmetric_runner(double const & tol_check) {
   }

   // check number of ev found by arpack.
-  if (iparam[4] < nev /*arpack may succeed to compute more EV than expected*/ || info != 0) {
+  if (iparam[4] < nev /*arpack may succeed to compute more EV than expected*/) {
     std::cout << "ERROR: iparam[4] " << iparam[4] << ", nev " << nev << ", info " << info << std::endl;
     throw std::domain_error("Error inside ARPACK routines");
   }
@@ -154,7 +154,7 @@ void complex_symmetric_runner(double const & tol_check) {
   }

   // check number of ev found by arpack.
-  if (iparam[4] < nev /*arpack may succeed to compute more EV than expected*/ || info != 0) {
+  if (iparam[4] < nev /*arpack may succeed to compute more EV than expected*/) {
     std::cout << "ERROR: iparam[4] " << iparam[4] << ", nev " << nev << ", info " << info << std::endl;
     throw std::domain_error("Error inside ARPACK routines");
   }

[opoplawski]

I'm not sure the initial report error is correct. With the latest build we are seeing:

ERROR: iparam[4] 1, nev 9, info 1
terminate called after throwing an instance of 'std::domain_error'
  what():  Error inside ARPACK routines
FAIL icb_arpack_cpp (exit status: 134)

with both the regular and the ILP64 build.

See https://kojipkgs.fedoraproject.org//packages/arpack/3.7.0/3.fc32/data/logs/ppc64le/build.log for full logs.

[rathann]

The error message (different from yours) in the report was from one of my koji test builds and the call stack (showing same error as yours) was from a mock build on the ppc64le Fedora test machine. Anyway, does this mean the real issue is in openblas?

[rathann]

Reading arpack code, I found this:

c  INFO    Integer.  (INPUT/OUTPUT)
c          If INFO .EQ. 0, a randomly initial residual vector is used.
c          If INFO .NE. 0, RESID contains the initial residual vector,
c                          possibly from a previous run.
c          Error flag on output.
c          =  0: Normal exit.
c          =  1: Maximum number of iterations taken.
c                All possible eigenvalues of OP has been found. IPARAM(5)
c                returns the number of wanted converged Ritz values.

Doesn't this mean that info == 1 is not actually an error? nev in the error message matches the expected value of 9, so what's the issue?

[fghoussen]

@rathann : I guess you are rigth, info == 1 is no error but special case. I think you can get rid of testing info (#230 (comment)) : if the result is tested and is found to be accurate enough (the test case does this), it's OK (at least to me !) even if lapack had to throw an error on the way. I guess this fix may some problems (likely not all) : did you test that fix ?

[fghoussen]

I'm not sure the initial report error is correct. With the latest build we are seeing:

ERROR: iparam[4] 1, nev 9, info 1
terminate called after throwing an instance of 'std::domain_error'
  what():  Error inside ARPACK routines
FAIL icb_arpack_cpp (exit status: 134)

with both the regular and the ILP64 build.

See https://kojipkgs.fedoraproject.org//packages/arpack/3.7.0/3.fc32/data/logs/ppc64le/build.log for full logs.

@opoplawski: right, this is real error here. Killing info does not hurt anyway : make things more simple but may not fix anything...

You also may try to increase ncv to ease convergence. Basically, you would need that fix (not tested):

>> git diff
diff --git a/TESTS/icb_arpack_cpp.cpp b/TESTS/icb_arpack_cpp.cpp
index 0e669a6..d02f3f0 100644
--- a/TESTS/icb_arpack_cpp.cpp
+++ b/TESTS/icb_arpack_cpp.cpp
@@ -32,6 +32,7 @@ void real_symmetric_runner(double const & tol_check) {
   a_int const nev = 9;
 
   a_int const ncv = 2 * nev + 1;
+  ncv *= 3; /*ease CV: increase space dimension*/
   a_int const ldv = N;
 
   a_int const ldz = N + 1;
@@ -112,6 +113,7 @@ void complex_symmetric_runner(double const & tol_check) {
   a_int const nev = 9;
 
   a_int const ncv = 2 * nev + 1;
+  ncv *= 3; /*ease CV: increase space dimension*/
   a_int const ldv = N;
 
   a_int const ldz = N + 1;

[fghoussen]

Did you try to compile with cmake ? If not, give a try to cmake : the problem may hide in autotools.
With autotools, you may try this patch 2e287a1.