Failing tests on s390c, ppc64le, and MIPS

[yarda]

We are trying to rebase gnuradio to 3.9.0.0 in Fedora, but we found out that the volk-2.4.1 testsuite doesn't pass on s390x and ppc64le. We build with standard distro flags plus -fno-strict-aliasing. The following tests are failing:

92 - qa_volk_32fc_s32fc_multiply_32fc (Failed)
93 - qa_volk_32fc_s32fc_rotatorpuppet_32fc (Failed)
102 - qa_volk_32fc_x2_s32fc_multiply_conjugate_add_32fc (Failed)

Build logs:
https://jskarvad.fedorapeople.org/volk/ppc64le_build_failure.txt
https://jskarvad.fedorapeople.org/volk/s390x_build_failure.txt

[jdemel]

VOLK does not provide optimizations for these architectures. Thus, only the generic kernels are used. We ran tests on these archs with TravisCI but need to stop it because of TravisCI policy changes.

Can you run your build with ctest -V? That should tell us more about these failing tests. I can only assume that the error is the same we saw with TravisCI, i.e. a segfault in a generic kernel. I can't test on ppc64le and s390x. Can you analyze the issue and suggest a fix? Or share more insight, e.g. a gdb backtrace?

[yarda]

You are right, it's segfaulting:

92: Test command: /usr/bin/sh "/root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib/volk_32fc_s32fc_multiply_32fc_test.sh" "/root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib"
92: Test timeout computed to be: 10000000
92: RUN_VOLK_TESTS: volk_32fc_s32fc_multiply_32fc(131071,1)
92: /root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib/volk_32fc_s32fc_multiply_32fc_test.sh: line 6: 20289 Segmentation fault      (core dumped) volk_test_all volk_32fc_s32fc_multiply_32fc
 92/134 Test  #92: qa_volk_32fc_s32fc_multiply_32fc .....................***Failed    0.14 sec
test 93
        Start  93: qa_volk_32fc_s32fc_rotatorpuppet_32fc

93: Test command: /usr/bin/sh "/root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib/volk_32fc_s32fc_rotatorpuppet_32fc_test.sh" "/root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib"
93: Test timeout computed to be: 10000000
93: RUN_VOLK_TESTS: volk_32fc_s32fc_rotatorpuppet_32fc(131071,1)
93: /root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib/volk_32fc_s32fc_rotatorpuppet_32fc_test.sh: line 6: 20293 Segmentation fault      (core dumped) volk_test_all volk_32fc_s32fc_rotatorpuppet_32fc
 93/134 Test  #93: qa_volk_32fc_s32fc_rotatorpuppet_32fc ................***Failed    0.17 sec
test 94
        Start  94: qa_volk_32fc_s32fc_x2_rotator_32fc

102: Test command: /usr/bin/sh "/root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib/volk_32fc_x2_s32fc_multiply_conjugate_add_32fc_test.sh" "/root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib"
102: Test timeout computed to be: 10000000
102: RUN_VOLK_TESTS: volk_32fc_x2_s32fc_multiply_conjugate_add_32fc(131071,1)
102: Volk warning: no arch found, returning generic impl
102: /root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib/volk_32fc_x2_s32fc_multiply_conjugate_add_32fc_test.sh: line 6: 20322 Segmentation fault      (core dumped) volk_test_all volk_32fc_x2_s32fc_multiply_conjugate_add_32fc
102/134 Test #102: qa_volk_32fc_x2_s32fc_multiply_conjugate_add_32fc ....***Failed    0.18 sec
test 103
        Start 103: qa_volk_32fc_x2_square_dist_32f

I will run it through the debugger.

[yarda]

bt for the volk_32fc_s32fc_multiply_32fc_test.sh, compiled with -g -O0 -fno-strict-aliasing:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  volk_32fc_s32fc_multiply_32fc_generic (cVector=0x7fff86b80010, aVector=0x7fff86a70010, scalar=327 + 0i, num_points=4049594328)
    at /root/rpmbuild/BUILD/volk-2.4.1/kernels/volk/volk_32fc_s32fc_multiply_32fc.h:235
235	        *cPtr++ = (*aPtr++) * scalar;
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.32-1.fc33.ppc64le libgcc-10.2.1-9.fc33.ppc64le libstdc++-10.2.1-9.fc33.ppc64le orc-0.4.31-3.fc33.ppc64le
(gdb) bt
#0  volk_32fc_s32fc_multiply_32fc_generic (cVector=0x7fff86b80010, aVector=0x7fff86a70010, scalar=327 + 0i, num_points=4049594328)
    at /root/rpmbuild/BUILD/volk-2.4.1/kernels/volk/volk_32fc_s32fc_multiply_32fc.h:235
#1  0x00007fff874cbb68 in volk_32fc_s32fc_multiply_32fc_manual (cVector=0x7fff86b80010, aVector=0x7fff86a70010, scalar=327 + 0i, 
    num_points=4049594328, impl_name=0x7ffff15fe7c8 "\330\347_\361\377\177")
    at /root/rpmbuild/BUILD/volk-2.4.1/ppc64le-redhat-linux-gnu/lib/volk.c:7784
#2  0x0000000138b611a0 in run_cast_test2_s32fc (func=0x7fff874cbaa4 <volk_32fc_s32fc_multiply_32fc_manual>, 
    buffs=std::vector of length 2, capacity 2 = {...}, scalar=..., vlen=131071, iter=0, arch="generic")
    at /root/rpmbuild/BUILD/volk-2.4.1/lib/qa_utils.cc:335
#3  0x0000000138b5d664 in run_volk_tests (desc=..., manual_func=0x7fff874cbaa4 <volk_32fc_s32fc_multiply_32fc_manual>, 
    name="volk_32fc_s32fc_multiply_32fc", tol=9.99999997e-07, scalar=..., vlen=131071, iter=1, results=0x7ffff15ff048, 
    puppet_master_name="NULL", absolute_mode=false, benchmark_mode=true) at /root/rpmbuild/BUILD/volk-2.4.1/lib/qa_utils.cc:644
#4  0x0000000138b5c73c in run_volk_tests (desc=..., manual_func=0x7fff874cbaa4 <volk_32fc_s32fc_multiply_32fc_manual>, 
    name="volk_32fc_s32fc_multiply_32fc", test_params=..., results=0x7ffff15ff048, puppet_master_name="NULL")
    at /root/rpmbuild/BUILD/volk-2.4.1/lib/qa_utils.cc:498
#5  0x0000000138b52cf0 in main (argc=2, argv=0x7ffff15ff568) at /root/rpmbuild/BUILD/volk-2.4.1/lib/testqa.cc:58

[yarda]

To be honest I have no idea why there are non sense values at #1. Valgrind showed that the values weren't crippled before. It was gcc 10.2.1, I will try with older gcc.

[yarda]

The same problem occurred with gcc 9.3.1.

[yarda]

The problem is caused by lv_32fc_t when linking C and C++ code, I am completing minimal reproducer. Currently I am not sure whether it's gcc bug.

[yarda]

Stripped down reproducer:

$ cat Makefile
test1: test1.o test11.o
	g++ -o test1 test11.o test1.o

test1.o: test1.c cpl.h
	gcc -c -o test1.o test1.c

test11.o: test1.cc cpl.h
	g++ -c -o test11.o test1.cc

$ cat cpl.h
ifdef __cplusplus

#include <complex>

typedef std::complex<float> lv_32fc_t;

#else

#include <complex.h>

typedef float complex lv_32fc_t;

#endif

typedef void (*volk_fn_2arg_s32fc)(lv_32fc_t, unsigned int, const char*);

$ cat test1.c
#include "cpl.h"
#include <stdio.h>

void manual(const lv_32fc_t scalar, unsigned int num_points, const char* impl_name)
{
    printf("%u, %f, %f, %s\n", num_points, __real__ scalar, __imag__ scalar, impl_name);
}

$ cat test1.cc
#include <stdio.h>
#include <string>
#include <complex>
#include "cpl.h"

extern "C" void manual(const lv_32fc_t  scalar, unsigned int  num_points, const char* impl_name);

unsigned int vlen = 131071;
lv_32fc_t scalar(327.0f, 2.0f);
std::string s("manual");
volk_fn_2arg_s32fc manual_func = (volk_fn_2arg_s32fc) manual;

int main(int argc, char *argv[])
{
  manual_func(scalar, vlen, s.c_str());

  return 0;
}

Both std::complex<float> and float complex seems to be 8 bytes. They are interchangeable on x86, but it seems on ppc64le and s390x they aren't, moreover they crippled all the following args. I don't know why, I will try opening gcc bug.

[yarda]

I created gcc bug and let's see :) https://bugzilla.redhat.com/show_bug.cgi?id=1918519

[marcusmueller]

Thank you, @yarda!

[jwakely]

They are interchangeable on x86, but it seems on ppc64le and s390x they aren't, moreover they crippled all the following args.

The standard doesn't say they're interchangeable so you're relying on undefined behaviour. All the standard says is that they have the same memory layout, so you can access std::complex<float>* and float _Complex* like an array of two floats and access the real and imaginary parts that way. That doesn't mean that function arguments of type std::complex<float> and float _Complex are passed the same way. The calling conventions for a given architecture might pass a std::complex<float> in a single 64-bit register, but might pass float _Complex by address, the same way a float[2] would be passed (decaying to a pointer).

[jwakely]

It works if you just pass the complex number by address instead of by value, see https://bugzilla.redhat.com/show_bug.cgi?id=1918519#c3

[jdemel]

@jwakely Thanks for the thorough explanation.

Now it becomes quite obvious why all those three _s32fc_ kernels fail. Also, we need to fix this. Obviously, we can't just change the function signature. I assume, we need to change the signature and add a shim that still exposes the old signature as well. After all I'm quite sure at least gnuradio relies on the old signature. And this might make gnuradio fail on ppc64le and s390x even with a fix.
Is there a way to fix this without breaking our API?

[marcusmueller]

@jdemel, well, the only way for current/previously released C++ software that uses lv_fc32_t as std::complex<float> not to break API is to leave it be std::complex<float>, that's the API.

However, not all is bleak:

We can fix this for GR master, 3.9.0.1, 3.8.2.1 (or whatever the next 3.8 is) by actually not relying on it being std::complex.

For people with un-upgradeable software that links against VOLK 3, you could just add a #ifdef __cpluspus #ifdef VOLK_COMPAT_LV_32FCT_STD_COMPLEX… to restore the old API, and write into the release notes that if you want to use old C++ software with new VOLK, you'll a) won't be able on s390x and ppc64, and b) you'll need to -DVOLK_COMPAT….

[jwakely]

You could add overloads on the C++ side which take std::complex<float> by value, then obtain float* from it, call a C function that takes the float* and the creates a new float _Complex from the values in the float* and calls the original C function with that new float _Complex.

That would allow the C++ code to keep calling the function and passing complex<float> by value, but have it safely convert that to the right type for the underlying C API.

But I don't see how to fix existing C++ code that has already been compiled and passes a std::complex<float> by value to the symbol manual (or whatever the real C function is called). That code is already calling the underlying C function with the wrong arguments and can't be fixed without a recompile on the targets where the calling convention differs.

[marcusmueller]

@jwakely exactly, all you can do is have a compatibility mode for this legacy software activated with a #define; afterwards, you can make the new API as sensible as you want.

creates a new float _Complex from the values

Application-wise, copying the data even once is not an option; VOLK is library for SIMD-accelerated direct operation on uniform vectors in memory, and copying that nullifies nearly every benefit you get from it. Therefore, you really need to cast the pointers passed around here, not the values, in C-land. While I'm certain we could trick enough compilers to at least accept that a const std::complex newone(old.real, old.imag) should be done without a copy, we cannot do the same for mutable objects – aliasing rules and everything.

So, yeah, I think we shouldn't ever had a C++ API that looks 90% like the C API but isn't. Makes little sense, usage-wise, anyway.