Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anaconda's scipy.test() fails with "OMP: Error #100: Fatal system error detected #785

Closed
evfro opened this issue Aug 5, 2016 · 19 comments
Closed

Comments

@evfro
Copy link

evfro commented Aug 5, 2016

  • A brief description

SciPy is a python package for scientific computing. Normally it's installed with Anaconda distribution, that also ships Intel's MKL library (with LAPACK and BLAS routines).

  • Expected results

Executing scipy.test() from within python completes with the report on all errors and failures discovered.

  • Actual results (with terminal output if applicable)

scipy.test() doesn't complete and terminates python with the following error:

OMP: Error #100: Fatal system error detected.
OMP: System error #22: Invalid argument
Aborted (core dumped)
  • Your Windows build number

Microsoft Windows [Version 10.0.14393]

  • Steps / commands required to reproduce the error

Run from terminal:

python -c 'import scipy; scipy.test()'
  • Strace of the failing command

https://gist.github.com/Evfro/2d0d83745a92d3f6e2f52ce3675a273d

The last thing before error there is the following line:
sched_getaffinity(0, 4, 0x7fc7757a7e00) = -1 EINVAL (Invalid argument)

  • Required packages and commands to install

Install latest miniconda (Python 2.7):

wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
bash Miniconda*.sh

Update apt-get (prevents some errors):

sudo apt-get update -qq

Install scipy, numpy and nose (for running tests):

conda install numpy scipy nose

This will additionally install MKL and libgfortran-3.
Note: test doesn't fail if nomkl versions of numpy and scipy are installed, however, MKL (especially its LAPACK and BLAS) is required for some other packages, so it's not really an option.

@deto
Copy link

deto commented Aug 5, 2016

Which packages is MKL required for? As far as I know, it was only added recently to their free-tier Linux installer so most things should work fine with nomkl.

@evfro
Copy link
Author

evfro commented Aug 5, 2016

For example, this ones fails at compilation time with exactly the same error (and it's not possible to install w/o MKL):
https://github.com/oseledets/ttpy
I guess, there could be many more scientific computing projects, that also rely on MKL's BLAS and LAPACK.

@evfro
Copy link
Author

evfro commented Aug 5, 2016

Sorry, my last comment was wrong. TTpy packages doesn't fail at compile time, but it fails when some functions are called (for example, tt.tensor).

What I've found is that setting environmental variable KMP_AFFINITY=disabled resolves the problem. However, I have no idea what implications it might have and not sure whether it's a proper workaround.

And it actually means, that the problem is not with MKL itself, but with OpenMP becoming faulty with any other value of KMP_AFFINITY (I tried all).

@srinath29
Copy link

Yeah.. And all the packages built on top of it. I was using statsmodels.api.GLSAR and it was also failing with the same problem. I opened a bug in statsmodels
statsmodels/statsmodels#3142

@srinath29
Copy link

Any solutions ? I am little anxious as one of my work depends on it.

@deto
Copy link

deto commented Aug 7, 2016

If nobody has a fix and you really need it, you might want to just run
Ubuntu in a VM. I've installed Anaconda with MKL doing this before. Not
as optimal as Bash on Windows but it'll get you moving.

You can also run much of the data science stack in Windows (not Bash) just
fine using Anaconda.

On Aug 7, 2016 12:29 AM, "srinath" notifications@github.com wrote:

Any solutions ? I am little anxious as one of my work depends on it.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx0us8t2TFzKCfsjh6LgOmZUWQCX5ks5qdYlSgaJpZM4Jd6On
.

@srinath29
Copy link

Thanks for the suggestion. I am already doing that. Actually I am in final stages of development of a benchmark https://github.com/srinath29/UltimateAnalyticsBenchmark
I want to test it on Bash on windows and I have score of running it natively on windows also. So I am bit curious.

@evfro
Copy link
Author

evfro commented Aug 7, 2016

Have you tried setting KMP_AFFINITY=disabled? Does it really kills performance? I haven't tested yet, would be interesting to see.

@srinath29
Copy link

srinath29 commented Aug 7, 2016

@evfro
Hey... It is working!!!!!
These are the results
cmd
bash

See the score at the bottom. Bash is performing better than windows. Now I am not sure it is because making it disabled or it's bash's power. Can you do the same test on your system? I am very curious to see how it is working.
I think this should be labeled as workaround .

@evfro
Copy link
Author

evfro commented Aug 9, 2016

@srinath29
I haven't closely looked into your code, but the thing to keep in mind when measuring time is that some python time-measuring functions behave differently on Linux and Windows. The most reliable way I found is to use

from timeit import default_timer

Now, that said, I can confirm that there are performance differences in certain types of operations.
For example, this simple code

a = numpy.random.random((10000, 10000))
%timeit a.mean()

gives me on average 52ms on bash and 133ms on windows.

However, more involved calculations like computing sparse SVD (scipy.sparse.linalg.svds) give me roughly equal results: 151ms on bash against 156ms on windows (for 6040 by 3706 matrix with 99% sparsity).

And while this KMP_AFFINITY thing does the trick, I would really like to hear from core developers team of BashOnWindows whether this is an appropriate modification (and whether it can be made default without additional user actions). From Intel's website: "Thread affinity is supported on Windows systems and versions of Linux systems that have kernel support for thread affinity..." So the problem can be either due to a bug or due to a kernel-level limitation.

@davidghiurco
Copy link

does anyone know if there's a fix other than disabling MKL affinity? I'm using a numpy/scipy stack for one of my classes and i develop on Bash for windows and am wondering if there's any fix for this or if I still have to do it the hacky way (and can rest assured that unless professor runs BashOnWindows too, he won't get this error)

@drhirsch
Copy link

suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04

@davidghiurco
Copy link

how to get this build for Bash for Windows? or is it installed automatically with Windows update?

@drhirsch
Copy link

You need Windows 14936 or newer, and no it will not autoupgrade, even when you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder) so backup anything you might need.

@deto
Copy link

deto commented Nov 19, 2016

Can't you also run do-release-upgrade? So you don't lose all your files?

On Nov 19, 2016 2:42 PM, "Michael Hirsch" notifications@github.com wrote:

You need Windows 14936 or newer, and no it will not autoupgrade, even when
you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command
Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder)
so backup anything you might need.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx8YV96Cxh8PsNLNj8_GrV5e1QiZqks5q_3tUgaJpZM4Jd6On
.

@drhirsch
Copy link

Some people do that, I wanted a clean start.

@bklooste
Copy link

Same with CNTK ., KMP_AFFINITY=disabled made it work

@stehufntdev
Copy link
Collaborator

Closing this per the comment above, please let us know if there are any regressions in this scenario:

"suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04"

@grayskripko
Copy link

@scienceopen it is possible to upgrade Ubuntu 14 to 16
According to https://media.readthedocs.org/pdf/wsl-guide/latest/wsl-guide.pdf

You can also upgrade to the latest version of Ubuntu with the following command (caution, this will take quite some time)!:
$ sudo -S apt-mark hold procps strace sudo
$ sudo -S env RELEASE_UPGRADER_NO_SCREEN=1 do-release-upgrade

I've got 16.04 on my machine after these steps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants