Segfault with eigen in R

[kyleam]

Hello,

The Guix package manager contains an OpenBLAS recipe. When used with
R, calling eigen with a matrix over some size results in a segfault:

R --vanilla -d valgrind
> x <- eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))
==12518== Invalid read of size 8
==12518==    at 0x8E400E0: dgemv_t_SANDYBRIDGE (in /gnu/store/hw9p1zyn1nh8pbm1cl69nm0i391lk6c7-openblas-0.2.15/lib/libopenblasp-r0.2.15.so)
==12518==    by 0x16BAED48: dlatrd_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==12518==    by 0x16C61F92: dsytrd_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==12518==    by 0x16CB9540: dsyevr_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==12518==    by 0x19F42D5E: La_rs (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/modules/lapack.so)
==12518==    by 0x19F45B96: mod_do_lapack (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/modules/lapack.so)
==12518==    by 0x4F35635: bcEval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==12518==    by 0x4F432DF: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==12518==    by 0x4F48F4B: Rf_applyClosure (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==12518==    by 0x4F4345E: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==12518==    by 0x4F46BBD: do_set (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==12518==    by 0x4F4367C: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==12518==  Address 0xfb0 is not stack'd, malloc'd or (recently) free'd
==12518==

 *** caught segfault ***
address 0xfb0, cause 'memory not mapped'

Traceback:
 1: eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))

This does not occur when R is built without OpenBLAS.

System details:

• Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz
• GNU/Linux (GuixSD and Guix-build on Arch Linux)
• gcc 4.9

OpenBLAS has been configured with NO_LAPACK=1,DYNAMIC_ARCH=1.

R is built with the following flags:

• --with-blas=openblas
• --with-lapack
• --with-cairo
• --with-libpng
• --with-jpeglib
• --with-libtiff
• --with-ICU
• --enable-R-shlib
• --enable-BLAS-shlib
• --with-system-zlib
• --with-system-bzlib
• --with-system-pcre
• --with-system-tre
• --with-system-xz

Any ideas on what the source of this problem might be? (While
discussing this on the Guix ticket, it was suggested that the
issue may be on the R side.)

Thanks in advance.

[brada4]

Does it happen with OMP_NUM_THREADS=1?
It could happen that your valgrind does not understand your pthread library

[brada4]

By default valgrind assumes 2MB stack, common defaults on Linux systems are 4..8..10..16MB
You must rise it to $(ulimit -s) * 1024 via --max-stackframe= parameter for valgrind.
It would be of value to capture FULL R output under valgrind (starting with R banner):
$ valgrind --max-stackframe=8388608 --trace-children=yes R --vanilla 2>&1 | tee console.out

[kyleam]

Thanks for your comments.

Does it happen with OMP_NUM_THREADS=1?
It could happen that your valgrind does not understand your pthread library

I still get the error if I set this as an environmental variable
before starting R.

If I run this without debugging with valgrind,

R --vanilla 2>&1 | tee console.out.3

I get

R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> x <- eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))

 *** caught segfault ***
address 0xfb0, cause 'memory not mapped'

Traceback:
 1: eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
aborting ...

Apparently, I haven't set things up to do a core dump. On the Guix ticket,
someone reproduced the problem and included a core dump.

You must rise it to $(ulimit -s) * 1024 via --max-stackframe= parameter for valgrind.
It would be of value to capture FULL R output under valgrind (starting with R banner):
$ valgrind --max-stackframe=8388608 --trace-children=yes R --vanilla 2>&1 | tee console.out

Setting --max-stackframe based on my system's ulimit output, I still
get the error. When I run

valgrind --max-stackframe=8388608 --trace-children=yes R --vanilla 2>&1 | tee console.out

this is the output captured in console.out:

==2120== Memcheck, a memory error detector
==2120== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==2120== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==2120== Command: /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/bin/R --vanilla
==2120==
==2122== Memcheck, a memory error detector
==2122== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==2122== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==2122== Command: /gnu/store/q6b4jg9nhsxb6kvn87nzr2w6f2vi1gx3-coreutils-8.24/bin/uname -m
==2122==
==2122==
==2122== HEAP SUMMARY:
==2122==     in use at exit: 0 bytes in 0 blocks
==2122==   total heap usage: 3 allocs, 3 frees, 116 bytes allocated
==2122==
==2122== All heap blocks were freed -- no leaks are possible
==2122==
==2122== For counts of detected and suppressed errors, rerun with: -v
==2122== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 1 from 1)
==2120== Memcheck, a memory error detector
==2120== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==2120== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==2120== Command: /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/bin/exec/R --vanilla
==2120==

R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> x <- eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))
==2120== Invalid read of size 8
==2120==    at 0x8E400E0: dgemv_t_SANDYBRIDGE (in /gnu/store/hw9p1zyn1nh8pbm1cl69nm0i391lk6c7-openblas-0.2.15/lib/libopenblasp-r0.2.15.so)
==2120==    by 0x20BB2D48: dlatrd_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==2120==    by 0x20C65F92: dsytrd_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==2120==    by 0x20CBD540: dsyevr_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==2120==    by 0x23F46D5E: La_rs (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/modules/lapack.so)
==2120==    by 0x23F49B96: mod_do_lapack (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/modules/lapack.so)
==2120==    by 0x4F35635: bcEval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==2120==    by 0x4F432DF: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==2120==    by 0x4F48F4B: Rf_applyClosure (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==2120==    by 0x4F4345E: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==2120==    by 0x4F46BBD: do_set (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==2120==    by 0x4F4367C: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==2120==  Address 0xfb0 is not stack'd, malloc'd or (recently) free'd
==2120==

 *** caught segfault ***
address 0xfb0, cause 'memory not mapped'

Traceback:
 1: eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
aborting ...
==2132== Memcheck, a memory error detector
==2132== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==2132== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==2132== Command: /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash -c rm\ -rf\ /tmp/RtmpkBr2xO
==2132==
==2132== Conditional jump or move depends on uninitialised value(s)
==2132==    at 0x4E20B6: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0x496D83: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0x4970DD: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0x4EBAEA: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0x4EDF5E: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0x4EF307: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0x468544: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0x402FE8: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132==    by 0xFFF000497: ???

[Edit: Truncated long, unhelpful output.]

[brada4]

That was huge. You should not abort R. It just aborts that process and valgrind does not have chance to see the memory access error. Just make normal exit and cancel.
Please keep away bash errors - they are clear sign your stack size does not match one traced by valgrind.

{lease just post valgrind findings after you cancel SIGABRT and related to R and blas only.

[kyleam]

Sorry. Thanks for your patience. Here is the output if I kill the
process with ABRT.

==5727== Memcheck, a memory error detector
==5727== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==5727== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==5727== Command: /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/bin/R --vanilla
==5727==
==5729== Memcheck, a memory error detector
==5729== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==5729== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==5729== Command: /gnu/store/q6b4jg9nhsxb6kvn87nzr2w6f2vi1gx3-coreutils-8.24/bin/uname -m
==5729==
==5729==
==5729== HEAP SUMMARY:
==5729==     in use at exit: 0 bytes in 0 blocks
==5729==   total heap usage: 3 allocs, 3 frees, 116 bytes allocated
==5729==
==5729== All heap blocks were freed -- no leaks are possible
==5729==
==5729== For counts of detected and suppressed errors, rerun with: -v
==5729== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 1 from 1)
==5727== Memcheck, a memory error detector
==5727== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==5727== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==5727== Command: /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/bin/exec/R --vanilla
==5727==

R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> x <- eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))
==5727== Invalid read of size 8
==5727==    at 0x8E400E0: dgemv_t_SANDYBRIDGE (in /gnu/store/hw9p1zyn1nh8pbm1cl69nm0i391lk6c7-openblas-0.2.15/lib/libopenblasp-r0.2.15.so)
==5727==    by 0x20DE2D48: dlatrd_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==5727==    by 0x20E95F92: dsytrd_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==5727==    by 0x20EED540: dsyevr_ (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libRlapack.so)
==5727==    by 0x24176D5E: La_rs (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/modules/lapack.so)
==5727==    by 0x24179B96: mod_do_lapack (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/modules/lapack.so)
==5727==    by 0x4F35635: bcEval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x4F432DF: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x4F48F4B: Rf_applyClosure (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x4F4345E: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x4F46BBD: do_set (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x4F4367C: Rf_eval (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==  Address 0xfb0 is not stack'd, malloc'd or (recently) free'd
==5727==

 *** caught segfault ***
address 0xfb0, cause 'memory not mapped'

Traceback:
 1: eigen(crossprod(matrix(rnorm(50 * 500), 50, 500)))

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: ==5727==
==5727== Process terminating with default action of signal 6 (SIGABRT)
==5727==    at 0x5D2A313: ??? (in /gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/lib/libc-2.22.so)
==5727==    by 0x502829E: R_SelectEx (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x50285C3: R_checkActivityEx (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x5028B04: Rstd_ReadConsole (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x4F69219: sigactionSegv (in /gnu/store/jb11p396a277rndb52da20ygdksccji8-r-3.2.2/lib/R/lib/libR.so)
==5727==    by 0x5A3BDCF: ??? (in /gnu/store/qv7bk62c22ms9i11dhfl71hnivyc82k2-glibc-2.22/lib/libpthread-2.22.so)
==5727==    by 0x8E400DF: dgemv_t_SANDYBRIDGE (in /gnu/store/hw9p1zyn1nh8pbm1cl69nm0i391lk6c7-openblas-0.2.15/lib/libopenblasp-r0.2.15.so)
==5727==
==5727== HEAP SUMMARY:
==5727==     in use at exit: 40,549,043 bytes in 12,076 blocks
==5727==   total heap usage: 31,025 allocs, 18,949 frees, 71,666,249 bytes allocated
==5727==
==5727== LEAK SUMMARY:
==5727==    definitely lost: 0 bytes in 0 blocks
==5727==    indirectly lost: 0 bytes in 0 blocks
==5727==      possibly lost: 2,128 bytes in 7 blocks
==5727==    still reachable: 40,546,915 bytes in 12,069 blocks
==5727==         suppressed: 0 bytes in 0 blocks
==5727== Rerun with --leak-check=full to see details of leaked memory
==5727==
==5727== For counts of detected and suppressed errors, rerun with: -v
==5727== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)

[martin-frbg]

This reminds me of #644 (mixup of C and FORTRAN-style array index counting) do we know what R/rlapack does between calls to BLAS/lapack functions ? I think we would need a debuggable build of OpenBLAS here to determine if the access is just before or just behind an array, and perhaps even run
valgrind with the --db-attach=yes option to start a gdb debugger session at the point of failure.

[brada4]

First we need debuggable glibc and clean mark on bash running simple R startup script.
After that we can toss the coin whose fault is it.

[brada4]

Your R build parameters are wrong
--with-blas=openblas
it is command line parameter like -lopenblas, configure just drops it
--with-lapack
Use system lapack (something provided by arch linux, not openblas or guix)
--enable-BLAS-shlib
build R's own libRblas.so from netlib source
I'd leave just latest and symlink to openblas if needed.
On top of GUIX being not production ready and arch linux providing official R-3.2.2 and openblas 0.2.14 builds+ as bonus production quality bash and valgrind packages.

[kyleam]

Thanks for the comment.

I'm aware of the state of Guix. My interest is in fixing its R
build. That'd be great if the issue ends up being in the build
configuration rather than somewhere upstream.

[brada4]

I dont see how openblas (or tux racer for instance) could help there:
You have broken valgrind making all debug attempts useless (or you want to report a problem of your miscompilation to bash maintainers?).
You said building without openblas works - if it is blas-shlib version - good - you found the way most distributions package R, then user can choose right BLAS for their CPU, be it openblas or atlas or mkl or acml trending on their system at that moment.

[martin-frbg]

Should be easy to see then if the problem is gone when one builds R with blas-shlib and then makes sure that it uses OpenBLAS at runtime. (Also not sure why we would want a debugable libc rather than debugable OpenBLAS and where the conviction that valgrind is broken rather than the stack trashed comes from ?)

[brada4]

Here:
==2132== Command: /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash -c rm\ -rf\ /tmp/RtmpkBr2xO
==2132==
==2132== Conditional jump or move depends on uninitialised value(s)
==2132== at 0x4E20B6: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)
==2132== by 0x496D83: ??? (in /gnu/store/7jhakv1r1nbs2sr2f7ammq256w7niarh-bash-static-4.3.39/bin/bash)

[rekado]

@brada4

The documentation makes clear that --with-blas=openblas is a configure time parameter. It is an instruction to link with -lopenblas.

--with-lapack links against the available lapack at build time. The documentation says:

Please do bear in mind that using --with-lapack is ‘definitely not recommended’: it is provided only because it is necessary on some platforms and because some users want to experiment with claimed performance improvements. Reporting problems where it is used unnecessarily will simply irritate the R helpers.

We could drop it in our R build, but I'd first like to know if this is related to the reported error.

You have broken valgrind making all debug attempts useless (or you want to report a problem of your miscompilation to bash maintainers?).
...
On top of GUIX being not production ready and arch linux providing official R-3.2.2 and openblas 0.2.14 builds+ as bonus production quality bash and valgrind packages.

I find your aggressiveness irritating and unhelpful. Please try to be less antagonistic.

[brada4]

No need to spit flames. 1 screen down the document you posted TLDR you see proper flag to link against openblas.
Valgrind is built with some -O99 flag. It just cannot happen that bash has corrupt stack running 1k sized script that runs on million of other linux computers just fine,and 2 openblas threads first read memory, then write same place meaning compiler used for valgrind and/opr blas and/or R optimizes out pthread mutex.
It is clear miscompilation, and me not judge where it is.

[brada4]

Latest build log for R-3.2.2 shows you link to both libRblas.so - old Netlib BLAS included in R and libopenblas.so (minus lapack/e), and in turn seconds later you build libRlapack.so and link to both libraries.
Later on your R build does not include pthread support, i'd bet it crashes in 3/4 of tries at eigen().

split R "example" in 5 lines, so you see for yourself if it is eigen crashing first ot other (backtrace does not match R eigen() call)

Tell me if you want me to continue on your build logs while i scrolled to best places in them.