Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failures on Xeon Cascade lake introduced in 0.3.19 #3505

Closed
nalimilan opened this issue Jan 1, 2022 · 11 comments · Fixed by #3510
Closed

Test failures on Xeon Cascade lake introduced in 0.3.19 #3505

nalimilan opened this issue Jan 1, 2022 · 11 comments · Fixed by #3510

Comments

@nalimilan
Copy link

I see test failures when running OpenBLAS 0.3.19 tests on a Xeon Cascade lake (see below), which did not appear using 0.3.18 (full build output here):

 Real CBLAS Test Program Results
 Test of subprogram number  1         CBLAS_SDOT     
                                    ----- PASS -----
 Test of subprogram number  2         CBLAS_SAXPY    
                                    ----- PASS -----
 Test of subprogram number  3         CBLAS_SROTG    
                                    ----- PASS -----
 Test of subprogram number  4         CBLAS_SROT     
                                    ----- PASS -----
 Test of subprogram number  5         CBLAS_SCOPY    
                                    ----- PASS -----
 Test of subprogram number  6         CBLAS_SSWAP    
                                    ----- PASS -----
 Test of subprogram number  7         CBLAS_SNRM2    
                                    ----- PASS -----
 Test of subprogram number  8         CBLAS_SASUM    
                                    ----- PASS -----
 Test of subprogram number  9         CBLAS_SSCAL    
                                    ----- PASS -----
 Test of subprogram number 10         CBLAS_ISAMAX   
                                    ----- PASS -----
 Test of subprogram number 11         CBLAS_SROTM    
                                    ----- PASS -----
OPENBLAS_NUM_THREADS=2 ./xdcblat1
 Real CBLAS Test Program Results
 Test of subprogram number  1         CBLAS_DDOT     
                                    ----- PASS -----
 Test of subprogram number  2         CBLAS_DAXPY    
                                    ----- PASS -----
 Test of subprogram number  3         CBLAS_DROTG    
                                    ----- PASS -----
 Test of subprogram number  4         CBLAS_DROT     
                                    ----- PASS -----
 Test of subprogram number  5         CBLAS_DCOPY    
                                    ----- PASS -----
 Test of subprogram number  6         CBLAS_DSWAP    
                                    ----- PASS -----
 Test of subprogram number  7         CBLAS_DNRM2    
                                    ----- PASS -----
 Test of subprogram number  8         CBLAS_DASUM    
                                    ----- PASS -----
 Test of subprogram number  9         CBLAS_DSCAL    
                                    ----- PASS -----
 Test of subprogram number 10         CBLAS_IDAMAX   
                                    ----- PASS -----
 Test of subprogram number 11         CBLAS_DROTM    
                                    ----- PASS -----
OPENBLAS_NUM_THREADS=2 ./xccblat1
 Complex CBLAS Test Program Results
 Test of subprogram number  1         CBLAS_CDOTC    
                                    ----- PASS -----
 Test of subprogram number  2         CBLAS_CDOTU    
                                    ----- PASS -----
 Test of subprogram number  3         CBLAS_CAXPY    
                                    ----- PASS -----
 Test of subprogram number  4         CBLAS_CCOPY    
                                    ----- PASS -----
 Test of subprogram number  5         CBLAS_CSWAP    
                                    ----- PASS -----
 Test of subprogram number  6         CBLAS_SCNRM2   
                                    ----- PASS -----
 Test of subprogram number  7         CBLAS_SCASUM   
                                    ----- PASS -----
 Test of subprogram number  8         CBLAS_CSCAL    
                                    ----- PASS -----
 Test of subprogram number  9         CBLAS_CSSCAL   
                                    ----- PASS -----
 Test of subprogram number 10         CBLAS_ICAMAX   
                                    ----- PASS -----
OPENBLAS_NUM_THREADS=2 ./xzcblat1
 Complex CBLAS Test Program Results
 Test of subprogram number  1         CBLAS_ZDOTC    
                                    ----- PASS -----
 Test of subprogram number  2         CBLAS_ZDOTU    
                                    ----- PASS -----
 Test of subprogram number  3         CBLAS_ZAXPY    
                                    ----- PASS -----
 Test of subprogram number  4         CBLAS_ZCOPY    
                                    ----- PASS -----
 Test of subprogram number  5         CBLAS_ZSWAP    
                                    ----- PASS -----
 Test of subprogram number  6         CBLAS_DZNRM2   
                                    ----- PASS -----
 Test of subprogram number  7         CBLAS_DZASUM   
                                    ----- PASS -----
 Test of subprogram number  8         CBLAS_ZSCAL    
                                    ----- PASS -----
 Test of subprogram number  9         CBLAS_ZDSCAL   
                                    ----- PASS -----
 Test of subprogram number 10         CBLAS_IZAMAX   
                                    ----- PASS -----
OPENBLAS_NUM_THREADS=2 ./xscblat2 < sin2
 TESTS OF THE REAL             LEVEL 2 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   0     1     2     3     5     9    63
   FOR K                   0     1     2     4
   FOR INCX AND INCY       1     2    -1    -2
   FOR ALPHA             0.0   1.0   0.7
   FOR BETA              0.0   1.0   0.9
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  1.2E-07
 cblas_sgemv  PASSED THE TESTS OF ERROR-EXITS
 cblas_sgemv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_sgemv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_sgbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_sgbmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_sgbmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_ssymv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ssymv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_ssymv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_ssbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ssbmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_ssbmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_sspmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_sspmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_sspmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_strmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_strmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_strmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_stbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_stbmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_stbmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_stpmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_stpmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_stpmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_strsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_strsv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_strsv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_stbsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_stbsv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_stbsv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_stpsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_stpsv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_stpsv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_sger   PASSED THE TESTS OF ERROR-EXITS
 cblas_sger   PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   484 CALLS)
 cblas_sger   PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   484 CALLS)
 cblas_ssyr   PASSED THE TESTS OF ERROR-EXITS
 cblas_ssyr   PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   145 CALLS)
 cblas_ssyr   PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   145 CALLS)
 cblas_sspr   PASSED THE TESTS OF ERROR-EXITS
 cblas_sspr   PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   145 CALLS)
 cblas_sspr   PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   145 CALLS)
 cblas_ssyr2  PASSED THE TESTS OF ERROR-EXITS
 cblas_ssyr2  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   577 CALLS)
 cblas_ssyr2  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   577 CALLS)
 cblas_sspr2  PASSED THE TESTS OF ERROR-EXITS
 cblas_sspr2  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   577 CALLS)
 cblas_sspr2  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   577 CALLS)
 END OF TESTS
OPENBLAS_NUM_THREADS=2 ./xdcblat2 < din2
 TESTS OF THE DOUBLE PRECISION LEVEL 2 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   0     1     2     3     5     9    63
   FOR K                   0     1     2     4
   FOR INCX AND INCY       1     2    -1    -2
   FOR ALPHA             0.0   1.0   0.7
   FOR BETA              0.0   1.0   0.9
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  2.2D-16
 cblas_dgemv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dgemv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_dgemv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_dgbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dgbmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_dgbmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_dsymv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dsymv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_dsymv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_dsbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dsbmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_dsbmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_dspmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dspmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_dspmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_dtrmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dtrmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dtrmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dtbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dtbmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_dtbmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_dtpmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dtpmv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dtpmv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dtrsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dtrsv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dtrsv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dtbsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dtbsv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_dtbsv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_dtpsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_dtpsv  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dtpsv  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   289 CALLS)
 cblas_dger   PASSED THE TESTS OF ERROR-EXITS
 cblas_dger   PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   484 CALLS)
 cblas_dger   PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   484 CALLS)
 cblas_dsyr   PASSED THE TESTS OF ERROR-EXITS
 cblas_dsyr   PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   145 CALLS)
 cblas_dsyr   PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   145 CALLS)
 cblas_dspr   PASSED THE TESTS OF ERROR-EXITS
 cblas_dspr   PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   145 CALLS)
 cblas_dspr   PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   145 CALLS)
 cblas_dsyr2  PASSED THE TESTS OF ERROR-EXITS
 cblas_dsyr2  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   577 CALLS)
 cblas_dsyr2  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   577 CALLS)
 cblas_dspr2  PASSED THE TESTS OF ERROR-EXITS
 cblas_dspr2  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (   577 CALLS)
 cblas_dspr2  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (   577 CALLS)
 END OF TESTS
OPENBLAS_NUM_THREADS=2 ./xccblat2 < cin2
 TESTS OF THE COMPLEX          LEVEL 2 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   0     1     2     3     5     9    63
   FOR K                   0     1     2     4
   FOR INCX AND INCY       1     2    -1    -2
   FOR ALPHA          ( 0.0, 0.0)  ( 1.0, 0.0)  ( 0.7,-0.9)  
   FOR BETA           ( 0.0, 0.0)  ( 1.0, 0.0)  ( 1.3,-1.1)  
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  1.2E-07
 cblas_cgemv  PASSED THE TESTS OF ERROR-EXITS
 cblas_cgemv  PASSED THE COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_cgemv  PASSED THE COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_cgbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_cgbmv  PASSED THE COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_cgbmv  PASSED THE COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_chemv  PASSED THE TESTS OF ERROR-EXITS
 cblas_chemv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_chemv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_chbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_chbmv  PASSED THE COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_chbmv  PASSED THE COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_chpmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_chpmv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_chpmv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_ctrmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctrmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ctrmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ctbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctbmv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ctbmv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ctpmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctpmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ctpmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ctrsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctrsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ctrsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ctbsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctbsv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ctbsv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ctpsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctpsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ctpsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_cgeru  PASSED THE TESTS OF ERROR-EXITS
 cblas_cgerc  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_cgerc  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_cgeru  PASSED THE TESTS OF ERROR-EXITS
 cblas_cgeru  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_cgeru  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_cher   PASSED THE TESTS OF ERROR-EXITS
 cblas_cher   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_cher   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_chpr   PASSED THE TESTS OF ERROR-EXITS
 cblas_chpr   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_chpr   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_cher2  PASSED THE TESTS OF ERROR-EXITS
 cblas_cher2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 cblas_cher2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 cblas_chpr2  PASSED THE TESTS OF ERROR-EXITS
 cblas_chpr2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 cblas_chpr2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 END OF TESTS
OPENBLAS_NUM_THREADS=2 ./xzcblat2 < zin2
 TESTS OF THE COMPLEX*16      LEVEL 2 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   0     1     2     3     5     9    63
   FOR K                   0     1     2     4
   FOR INCX AND INCY       1     2    -1    -2
   FOR ALPHA          ( 0.0, 0.0)  ( 1.0, 0.0)  ( 0.7,-0.9)  
   FOR BETA           ( 0.0, 0.0)  ( 1.0, 0.0)  ( 1.3,-1.1)  
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  2.2E-16
 cblas_zgemv  PASSED THE TESTS OF ERROR-EXITS
 cblas_zgemv  PASSED THE COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_zgemv  PASSED THE COMPUTATIONAL TESTS (  4324 CALLS)
 cblas_zgbmv  PASSED THE TESTS OF ERROR-EXITS
OPENBLAS_NUM_THREADS=2 ./xscblat3 < sin3
 TESTS OF THE REAL             LEVEL 3 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   0     1     2     3     5     9    35
   FOR ALPHA             0.0   1.0   0.7
   FOR BETA              0.0   1.0   1.3
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  1.2E-07
 cblas_sgemm  PASSED THE TESTS OF ERROR-EXITS
 cblas_zgbmv  PASSED THE COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_zgbmv  PASSED THE COMPUTATIONAL TESTS ( 17284 CALLS)
 cblas_zhemv  PASSED THE TESTS OF ERROR-EXITS
 cblas_zhemv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_sgemm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS ( 27783 CALLS)
 cblas_zhemv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_zhbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_zhbmv  PASSED THE COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_zhbmv  PASSED THE COMPUTATIONAL TESTS (  6913 CALLS)
 cblas_zhpmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_zhpmv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_zhpmv  PASSED THE COMPUTATIONAL TESTS (  1729 CALLS)
 cblas_ztrmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztrmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ztrmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ztbmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztbmv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ztbmv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ztpmv  PASSED THE TESTS OF ERROR-EXITS
 cblas_sgemm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 27783 CALLS)
 cblas_ssymm  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztpmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ztpmv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ztrsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztrsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ztrsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ztbsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ssymm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_ztbsv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ztbsv  PASSED THE COMPUTATIONAL TESTS (  1153 CALLS)
 cblas_ztpsv  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztpsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_ssymm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_strmm  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztpsv  PASSED THE COMPUTATIONAL TESTS (   289 CALLS)
 cblas_zgeru  PASSED THE TESTS OF ERROR-EXITS
 cblas_zgerc  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_strmm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_zgerc  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_zgeru  PASSED THE TESTS OF ERROR-EXITS
 cblas_zgeru  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_strmm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_strsm  PASSED THE TESTS OF ERROR-EXITS
 cblas_zgeru  PASSED THE COMPUTATIONAL TESTS (   484 CALLS)
 cblas_zher   PASSED THE TESTS OF ERROR-EXITS
 cblas_zher   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_zher   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_zhpr   PASSED THE TESTS OF ERROR-EXITS
 cblas_zhpr   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_zhpr   PASSED THE COMPUTATIONAL TESTS (   145 CALLS)
 cblas_zher2  PASSED THE TESTS OF ERROR-EXITS
 cblas_strsm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_zher2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 cblas_zher2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 cblas_zhpr2  PASSED THE TESTS OF ERROR-EXITS
 cblas_zhpr2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 cblas_strsm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_ssyrk  PASSED THE TESTS OF ERROR-EXITS
 cblas_zhpr2  PASSED THE COMPUTATIONAL TESTS (   577 CALLS)
 END OF TESTS
 cblas_ssyrk  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  2646 CALLS)
 cblas_ssyrk  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  2646 CALLS)
 cblas_ssyr2k PASSED THE TESTS OF ERROR-EXITS
 cblas_ssyr2k PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  2646 CALLS)
 cblas_ssyr2k PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  2646 CALLS)
 END OF TESTS
OPENBLAS_NUM_THREADS=2 ./xdcblat3 < din3
 TESTS OF THE DOUBLE PRECISION LEVEL 3 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   1     2     3     5     7     9    35
   FOR ALPHA             0.0   1.0   0.7
   FOR BETA              0.0   1.0   1.3
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  2.2D-16
 cblas_dgemm  PASSED THE TESTS OF ERROR-EXITS
 cblas_dgemm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS ( 27783 CALLS)
 cblas_dgemm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 27783 CALLS)
 cblas_dsymm  PASSED THE TESTS OF ERROR-EXITS
 ******* FATAL ERROR - COMPUTED RESULT IS LESS THAN HALF ACCURATE *******
           EXPECTED RESULT   COMPUTED RESULT
       1     -0.572029          0.486094    
      THESE ARE THE RESULTS FOR COLUMN   1
 ******* cblas_dsymm  FAILED ON CALL NUMBER:
    238: cblas_dsymm ( CblasColMajor,    CblasRight,    CblasUpper,
                      1, 35, 1.0, A, 36, B,  2, 0.0, C,  2).
 ******* cblas_dsymm  FAILED ON CALL NUMBER:
      1: cblas_dsymm ( CblasRowMajor,     CblasLeft,    CblasUpper,
                      1,  1, 0.0, A,  2, B,  2, 0.0, C,  2).
 ******* FATAL ERROR - TESTS ABANDONED *******
OPENBLAS_NUM_THREADS=2 ./xccblat3 < cin3
 TESTS OF THE COMPLEX          LEVEL 3 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   0     1     2     3     5     9
   FOR ALPHA          ( 0.0, 0.0)  ( 1.0, 0.0)  ( 0.7,-0.9)  
   FOR BETA           ( 0.0, 0.0)  ( 1.0, 0.0)  ( 1.3,-1.1)  
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  1.2E-07
 cblas_cgemm  PASSED THE TESTS OF ERROR-EXITS
 cblas_cgemm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS ( 17496 CALLS)
 cblas_cgemm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 17496 CALLS)
 cblas_chemm  PASSED THE TESTS OF ERROR-EXITS
 cblas_chemm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_chemm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_csymm  PASSED THE TESTS OF ERROR-EXITS
 cblas_csymm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_csymm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_ctrmm  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctrmm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  2592 CALLS)
 cblas_ctrmm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  2592 CALLS)
 cblas_ctrsm  PASSED THE TESTS OF ERROR-EXITS
 cblas_ctrsm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  2592 CALLS)
 cblas_ctrsm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  2592 CALLS)
 cblas_cherk  PASSED THE TESTS OF ERROR-EXITS
 cblas_cherk  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_cherk  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_csyrk  PASSED THE TESTS OF ERROR-EXITS
 cblas_csyrk  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_csyrk  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_cher2k PASSED THE TESTS OF ERROR-EXITS
 cblas_cher2k PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_cher2k PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_csyr2k PASSED THE TESTS OF ERROR-EXITS
 cblas_csyr2k PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1296 CALLS)
 cblas_csyr2k PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1296 CALLS)
 END OF TESTS
OPENBLAS_NUM_THREADS=2 ./xzcblat3 < zin3
TESTS OF THE COMPLEX*16        LEVEL 3 BLAS
 THE FOLLOWING PARAMETER VALUES WILL BE USED:
   FOR N                   0     1     2     3     5     9    35
   FOR ALPHA          ( 0.0, 0.0)  ( 1.0, 0.0)  ( 0.7,-0.9)  
   FOR BETA           ( 0.0, 0.0)  ( 1.0, 0.0)  ( 1.3,-1.1)  
 ROUTINES PASS COMPUTATIONAL TESTS IF TEST RATIO IS LESS THAN   16.00
 COLUMN-MAJOR AND ROW-MAJOR DATA LAYOUTS ARE TESTED
 RELATIVE MACHINE PRECISION IS TAKEN TO BE  2.2E-16
 cblas_zgemm  PASSED THE TESTS OF ERROR-EXITS
 cblas_zgemm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS ( 27783 CALLS)
 cblas_zgemm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS ( 27783 CALLS)
 cblas_zhemm  PASSED THE TESTS OF ERROR-EXITS
 cblas_zhemm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zhemm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zsymm  PASSED THE TESTS OF ERROR-EXITS
 cblas_zsymm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zsymm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_ztrmm  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztrmm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_ztrmm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_ztrsm  PASSED THE TESTS OF ERROR-EXITS
 cblas_ztrsm  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_ztrsm  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  3528 CALLS)
 cblas_zherk  PASSED THE TESTS OF ERROR-EXITS
 cblas_zherk  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zherk  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zsyrk  PASSED THE TESTS OF ERROR-EXITS
 cblas_zsyrk  PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zsyrk  PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zher2k PASSED THE TESTS OF ERROR-EXITS
 cblas_zher2k PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zher2k PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zsyr2k PASSED THE TESTS OF ERROR-EXITS
 cblas_zsyr2k PASSED THE COLUMN-MAJOR COMPUTATIONAL TESTS (  1764 CALLS)
 cblas_zsyr2k PASSED THE ROW-MAJOR    COMPUTATIONAL TESTS (  1764 CALLS)
 END OF TESTS

Details about the CPU are here. The exrcept above is for the libopenblasp64.so library, which is built with TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NUM_THREADS=128 USE_THREAD=1 INTERFACE64=1. But the build log contains error for other variants too (we build many, notably with INTERFACE64=0 and USE_THREAD=0).

This triggers an incredible number of failures when running Julia tests.

Unfortunately I do not have direct access to that machine, I can only trigger builds of the RPM package. I can try a few build or runtime options though if that's useful.

Might be related to #3494. See also https://bugzilla.redhat.com/show_bug.cgi?id=1982856.
@martin-frbg
Copy link
Collaborator

martin-frbg commented Jan 1, 2022 •
edited

can you try #3498 please ? unfortunately my only avx512 host at the moment is a lowly quadcore where I could not reproduce the (other) problem

@nalimilan
Copy link
Author

I've just started a build applying that fix (see the sed -i command), but unfortunately I still see FATAL ERRORs in the log: https://kojipkgs.fedoraproject.org//work/tasks/9866/80719866/build.log

Note that we build using DYNAMIC_ARCH=1, but AFAICT this fix isn't used in that case, right?

@nalimilan
Copy link
Author

Actually I can reproduce the failures on a machine to which I have SSH access, so feel free to suggest things to try.

@martin-frbg
Copy link
Collaborator

the full patch from #3498 would remove the special handling of DYNAMIC_ARCH (the revert to the other, older dgemm kernel - though i do not recall earlier errors from that).
if you have direct access, you could try a build without DYNAMIC_ARCH with the one-liner sed as you have it now

@nalimilan
Copy link
Author

I'm unable to reproduce the error if I drop DYNAMIC_ARCH=1 and use TARGET=SKYLAKEX instead (even without the sed call). Does that make sense?

BTW, why don't make and make tests return a non-zero value when tests fail? This would have caught the error in Fedora before we ship it to users.

@nalimilan
Copy link
Author

Though #3498 with DYNAMIC_ARCH=1 fixes the errors!

@martin-frbg
Copy link
Collaborator

Makes some sense as the error observed in #3494 may be specific to something in Ubuntu 18.04 (binutils or their snapshot/build of gcc 7.5.0). The switch to a different dgemm kernel for DYNAMIC_ARCH builds was added to work around an older issue where builds made on Sandybridge were observed to fail on Skylake. Possibly the old 4x8 DGEMM does not coexist well with the small-matrix kernels added recently)
The tests do not return non-zero currently as they are copied from the netlib reference implementation, and it is at least theoretically possible for them to fail in a non-fatal way due to limited accuracy. (And it does not help that the test failures seen - in retrospect after #3494 - in the CI runs with Intel SDE were intermittent. At least PR #3498 has them always passing - that is what I tested with the repeated closing and reopening of the PR)

@nalimilan
Copy link
Author

Makes some sense as the error observed in #3494 may be specific to something in Ubuntu 18.04 (binutils or their snapshot/build of gcc 7.5.0). The switch to a different dgemm kernel for DYNAMIC_ARCH builds was added to work around an older issue where builds made on Sandybridge were observed to fail on Skylake. Possibly the old 4x8 DGEMM does not coexist well with the small-matrix kernels added recently)

OK. What do you think we should do in the short term to fix the failures on Fedora?

The tests do not return non-zero currently as they are copied from the netlib reference implementation, and it is at least theoretically possible for them to fail in a non-fatal way due to limited accuracy. (And it does not help that the test failures seen - in retrospect after #3494 - in the CI runs with Intel SDE were intermittent. At least PR #3498 has them always passing - that is what I tested with the repeated closing and reopening of the PR)

I see. If/once they are relatively reliable, it would be useful to have a way to make them fail. It's easy to disable that if failures happen at random, but catching bugs early could be useful for Fedora IMO.

BTW, a related question I have: have you considered using a versioning scheme which allows distinguishing pure bugfix releases (which should be backported in Fedora) from feature releases (which should not)? Ideally this kind of bug wouldn't have been backported to existing Fedora versions so that we have more time to handle it.

@martin-frbg
Copy link
Collaborator

OK. What do you think we should do in the short term to fix the failures on Fedora?

go with the patch from #3498

a versioning scheme which allows distinguishing pure bugfix releases (...) from feature releases

not doable I think, given the number of architectures supported. (unless you expect me to do a release whenever anything bad pops up anywhere, or more frequent releases in general)

@nalimilan
Copy link
Author

go with the patch from #3498

OK, thanks!

not doable I think, given the number of architectures supported. (unless you expect me to do a release whenever anything bad pops up anywhere, or more frequent releases in general)

My idea was not really to have more frequent releases or more backports, but to communicate more clearly to packagers and users what to expect from a given version. This is just semantic versioning, which many projects follow. If most OpenBLAS releases mix new features/improvements and bug fixes, it would be clearer IMHO to increase the minor version number rather than only the patch number. If you do that, nothing forces you to make bugfix/patch releases, but if at some point a situation happens when e.g. a serious regression was introduced in the last release, shipping a patch in a bugfix release will make it clear to packagers and distributors that they should upgrade and can do it with minimal risks of breakage.

@brada4
Copy link
Contributor

brada4 commented Jan 2, 2022

API does not change through decades, maybe it is worth noting PR-s where bugs are fixed so those at backporting can pick them up.

@martin-frbg martin-frbg mentioned this issue Jan 8, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix recent SkylakeX/DYNAMIC_ARCH DGEMM breakage
3 participants
@nalimilan @martin-frbg @brada4