Skip to content
This repository has been archived by the owner on Sep 11, 2023. It is now read-only.

Solving of eigenvalue problems yield bogus results with OpenBLAS-0.3.5 on SkyLake X CPUs #1426

Closed
thempel opened this issue Jun 27, 2019 · 3 comments

Comments

@thempel
Copy link
Member

thempel commented Jun 27, 2019

Apparently, there is a problem with some dependencies that cause the eigendecomposition of sparse matrices to fail. In our debug-session, @marscher figured out that at least with newer versions of scipy / openblas, unit tests with sparse matrices fail. Eigendecompositions cannot be computed or produce random results. The issue will be reported upstream.

A conda environment yaml file that reproduces the issue is attached here.
However, the issue could be reproduced with any clean installation of pyemma.

The problem can be avoided by using dense matrices (default behavior).

@thempel
Copy link
Member Author

thempel commented Jun 28, 2019

We pinned down the problem to a specific version of OpenBLAS, namely 3.8 from conda-forge. It seems like build number 7 is broken, while number 10 has been tested to work with PyEMMA.

libblas                   3.8.0                7_openblas    conda-forge
libcblas                  3.8.0                7_openblas    conda-forge

@thempel
Copy link
Member Author

thempel commented Jun 28, 2019

Could be fixed with

>>> conda install blas
The following NEW packages will be INSTALLED:

  blas               conda-forge/linux-64::blas-2.10-mkl
  liblapacke         conda-forge/linux-64::liblapacke-3.8.0-10_mkl

The following packages will be UPDATED:

  libblas                                  3.8.0-7_openblas --> 3.8.0-10_mkl
  libcblas                                 3.8.0-7_openblas --> 3.8.0-10_mkl
  liblapack                                3.8.0-7_openblas --> 3.8.0-10_mkl

@thempel thempel closed this as completed Jun 28, 2019
@marscher
Copy link
Member

marscher commented Jul 2, 2019

quoting @isuruf

openblas 0.3.5 had issues with SkyLake X cpus. dgemm was broken there.
I'll move 0.3.5 openblas to broken

This means, that we will not get any more broken installations, which is very good. But I have no clue how to deal with the problem, that some installations are already broken. We can not reach people properly and in the meantime they will have bogus results.

@marscher marscher changed the title sparse matrices may be broken Solutions of eigenvalue problems yield bogus results Jul 2, 2019
@marscher marscher changed the title Solutions of eigenvalue problems yield bogus results Solving of eigenvalue problems yield bogus results with OpenBLAS Jul 2, 2019
@marscher marscher pinned this issue Jul 2, 2019
@marscher marscher changed the title Solving of eigenvalue problems yield bogus results with OpenBLAS Solving of eigenvalue problems yield bogus results with OpenBLAS-0.3.5 on SkyLake X CPUs Jul 15, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants