Self Tests Fail (L2/L3) When Using IBM Compilers

[hughes-c]

The self and link tests fail when using the IBM build tools but work fine on the same system with gcc 5.4.0. I have not tried with MASS or without OpenMP; nor have I compared the assembly produced by the different compilers yet. I have tried everything from O0 to O5 with and without the strict flag and the results are the same.

I have included a file with the test output below and will update the issue when I know more.

System Details:

• RHEL 7 Little Endian
• IBM Power8 (8335-GTA) (dual socket 8PE/8thread per socket)
• xlc-14..1(beta 6)/xlf-16.1(beta 6)

Make/Compilation:
VERBOSE=1 USE_OPENMP=1 TARGET=POWER8 make tests
xlc -Ofast -q64 -qstrict -qarch=pwr8 -qtune=pwr8 -qalign=linuxppc -DUSE_OPENMP -qsmp=omp
xlf90_r -q64 -qpic=large -O2 -q64 -qstrict -qfixed -qarch=pwr8 -qtune=pwr8 -qalign=bindc=linuxppc -DUSE_OPENMP -qsmp=omp
I have also tried including -L${XLF_ROOT}/lib -L${XLC_ROOT}/lib -lxlf90_r -lxlopt -lxlsmp -lxl -lxlfmath

build-tests.txt

[martin-frbg]

Can you test with 0.2.20 or current "develop" snapshot rather than 0.2.19 ? There have been several fixes for unwanted side effects of Power8 assembly routines recently. (If RHEL on Power8 uses something other than glibc please use "develop", otherwise 0.2.20 and develop are practically identical)

[hughes-c]

Working from the develop branch:

xlc -c -Ofast -q64 -qstrict -qarch=pwr8 -qtune=pwr8 -qalign=linuxppc -DUSE_OPENMP -qsmp=omp -DMAX_STACK_ALLOC=2048 -DF_INTERFACE_IBM -qpic=large -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=128 -DASMNAME=dtrmm_oltucopy -DASMFNAME=dtrmm_oltucopy -DNAME=dtrmm_oltucopy -DCNAME=dtrmm_oltucopy -DCHAR_NAME=\"dtrmm_oltucopy\" -DCHAR_CNAME=\"dtrmm_oltucopy\" -DNO_AFFINITY -I.. -DDOUBLE  -UCOMPLEX  -DDOUBLE -UCOMPLEX -DOUTER -DLOWER -DUNIT generic/trmm_ltcopy_4.c -o dtrmm_oltucopy.o                                                                                                                                                                                                                                     
In file included from ../kernel/power/sdot.c:39:                                                                                                                                                                                               
../kernel/power/sdot_microk_power8.c:169:8: error: unknown register name 'vs32' in asm                                                                                                                                                         
       "vs32","vs33","vs34","vs35","vs36","vs37","vs38","vs39",                                                                                                                                                                                
       ^                                                                                                                                                                                                                                       
1 error generated.                                                                                                                                                                                                                             
Error while processing ../kernel/power/sdot.c.                                                                                                                                                                                                 
make[1]: *** [sdot_k.o] Error 1                                                                                                                                                                                                                
make[1]: *** Waiting for unfinished jobs....                                                                                                                                                                                                   
In file included from ../kernel/power/scopy.c:39:                                                                                                                                                                                              
../kernel/power/scopy_microk_power8.c:110:8: error: unknown register name 'vs40' in asm                                                                                                                                                        
       "vs40","vs41","vs42","vs43","vs44","vs45","vs46","vs47"

It keeps going like that for some time.

[martin-frbg]

That's bad, these vs registers were added to the assembly clobber lists as part of the fix. This now looks like the clang compiler problem #1120 so could be the file uses some gnu assembler idiom for register naming.

[martin-frbg]

Things I found on the 'net suggest compiler options like -qaltivec or -maltivec might be needed for xlc to enable compiler support for vector data types and instructions. No idea if that is true...

[hughes-c]

It looks like a problem with the clobber list. I had a similar issue before where the list was self-referencing (e.g. %0, %1) and xlc produced warnings about register names. I did try adding -qasm=gcc to the compile flags with no success.

I'll try -qaltivec this afternoon but when i tried adding it on Friday, it gave me a lot more errors.

[martin-frbg]

I haven't used xlc since the days of the RS6000 so I cannot really help. All I managed to find was this discussion https://redmine.gromacs.org/issues/2102 which has a link to the manual of a somewhat older xlc version. (Perhaps knowing what the other errors were would help others to identify the problem)

[brada4]

If it builds with gcc (Like found at fedora epel) you can call functions from xlc? ASM is extension to C standards and in absence of documentation impossible to guess what goes wrong.

[hughes-c]

Yup. I can build it with gcc, including MASS, with no problem. However, I would like to be able to use IBM's toolchain, so I'll keep working on this and report back when I have an update.

[brada4]

Please run 'make clean' before new build and record output with 'script'
Fo not run 'make tests' before 'make' completes (ideally plain 'make' runs tests already and you can not mix up build options...)
when i last ran xlc on pSeries it overhauled assemblies at high optimisation levels. Try like '-O1' for starters.

[brada4]

Also add MAKE_NB_JOBS=1 (very slow 1-cpu build) to sync commands with error outputs.
curent output shows _trmm compilation command next to _dot assembly error....

[hughes-c]

I've attached the output that you requested.

power8_build.txt

[brada4]

Maybe the other shadow registers need to be saved go figure.

[martin-frbg]

So still no luck finding an xlc option that makes it accept vsx in clobber lists ?

[brada4]

maybe -qvsx iff it exists?