Anaconda's scipy.test() fails with "OMP: Error #100: Fatal system error detected #785

Closed
Evfro opened this Issue Aug 5, 2016 · 18 comments

Projects

None yet

8 participants

@Evfro
Evfro commented Aug 5, 2016 edited
  • A brief description

SciPy is a python package for scientific computing. Normally it's installed with Anaconda distribution, that also ships Intel's MKL library (with LAPACK and BLAS routines).

  • Expected results

Executing scipy.test() from within python completes with the report on all errors and failures discovered.

  • Actual results (with terminal output if applicable)

scipy.test() doesn't complete and terminates python with the following error:

OMP: Error #100: Fatal system error detected.
OMP: System error #22: Invalid argument
Aborted (core dumped)
  • Your Windows build number

Microsoft Windows [Version 10.0.14393]

  • Steps / commands required to reproduce the error

Run from terminal:

python -c 'import scipy; scipy.test()'
  • Strace of the failing command

https://gist.github.com/Evfro/2d0d83745a92d3f6e2f52ce3675a273d

The last thing before error there is the following line:
sched_getaffinity(0, 4, 0x7fc7757a7e00) = -1 EINVAL (Invalid argument)

  • Required packages and commands to install

Install latest miniconda (Python 2.7):

wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
bash Miniconda*.sh

Update apt-get (prevents some errors):

sudo apt-get update -qq

Install scipy, numpy and nose (for running tests):

conda install numpy scipy nose

This will additionally install MKL and libgfortran-3.
Note: test doesn't fail if nomkl versions of numpy and scipy are installed, however, MKL (especially its LAPACK and BLAS) is required for some other packages, so it's not really an option.

@deto
deto commented Aug 5, 2016

Which packages is MKL required for? As far as I know, it was only added recently to their free-tier Linux installer so most things should work fine with nomkl.

@Evfro
Evfro commented Aug 5, 2016

For example, this ones fails at compilation time with exactly the same error (and it's not possible to install w/o MKL):
https://github.com/oseledets/ttpy
I guess, there could be many more scientific computing projects, that also rely on MKL's BLAS and LAPACK.

@Evfro
Evfro commented Aug 5, 2016 edited

Sorry, my last comment was wrong. TTpy packages doesn't fail at compile time, but it fails when some functions are called (for example, tt.tensor).

What I've found is that setting environmental variable KMP_AFFINITY=disabled resolves the problem. However, I have no idea what implications it might have and not sure whether it's a proper workaround.

And it actually means, that the problem is not with MKL itself, but with OpenMP becoming faulty with any other value of KMP_AFFINITY (I tried all).

@srinath29 srinath29 referenced this issue in statsmodels/statsmodels Aug 6, 2016
Closed

On Bash on Ubuntu on Windows 10, fit() is crashing #3142

@srinath29

Yeah.. And all the packages built on top of it. I was using statsmodels.api.GLSAR and it was also failing with the same problem. I opened a bug in statsmodels
statsmodels/statsmodels#3142

@srinath29

Any solutions ? I am little anxious as one of my work depends on it.

@deto
deto commented Aug 7, 2016

If nobody has a fix and you really need it, you might want to just run
Ubuntu in a VM. I've installed Anaconda with MKL doing this before. Not
as optimal as Bash on Windows but it'll get you moving.

You can also run much of the data science stack in Windows (not Bash) just
fine using Anaconda.

On Aug 7, 2016 12:29 AM, "srinath" notifications@github.com wrote:

Any solutions ? I am little anxious as one of my work depends on it.

โ€”
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx0us8t2TFzKCfsjh6LgOmZUWQCX5ks5qdYlSgaJpZM4Jd6On
.

@srinath29

Thanks for the suggestion. I am already doing that. Actually I am in final stages of development of a benchmark https://github.com/srinath29/UltimateAnalyticsBenchmark
I want to test it on Bash on windows and I have score of running it natively on windows also. So I am bit curious.

@Evfro
Evfro commented Aug 7, 2016

Have you tried setting KMP_AFFINITY=disabled? Does it really kills performance? I haven't tested yet, would be interesting to see.

@srinath29
srinath29 commented Aug 7, 2016 edited

@Evfro
Hey... It is working!!!!!
These are the results
cmd
bash

See the score at the bottom. Bash is performing better than windows. Now I am not sure it is because making it disabled or it's bash's power. Can you do the same test on your system? I am very curious to see how it is working.
I think this should be labeled as workaround .

@Evfro
Evfro commented Aug 9, 2016

@srinath29
I haven't closely looked into your code, but the thing to keep in mind when measuring time is that some python time-measuring functions behave differently on Linux and Windows. The most reliable way I found is to use

from timeit import default_timer

Now, that said, I can confirm that there are performance differences in certain types of operations.
For example, this simple code

a = numpy.random.random((10000, 10000))
%timeit a.mean()

gives me on average 52ms on bash and 133ms on windows.

However, more involved calculations like computing sparse SVD (scipy.sparse.linalg.svds) give me roughly equal results: 151ms on bash against 156ms on windows (for 6040 by 3706 matrix with 99% sparsity).

And while this KMP_AFFINITY thing does the trick, I would really like to hear from core developers team of BashOnWindows whether this is an appropriate modification (and whether it can be made default without additional user actions). From Intel's website: "Thread affinity is supported on Windows systems and versions of Linux systems that have kernel support for thread affinity..." So the problem can be either due to a bug or due to a kernel-level limitation.

@davidghiurco

does anyone know if there's a fix other than disabling MKL affinity? I'm using a numpy/scipy stack for one of my classes and i develop on Bash for windows and am wondering if there's any fix for this or if I still have to do it the hacky way (and can rest assured that unless professor runs BashOnWindows too, he won't get this error)

@jackchammons jackchammons added the python label Oct 25, 2016
@jhunkeler jhunkeler referenced this issue in spacetelescope/JWSTUserTraining2016 Nov 8, 2016
Open

check_env core dumps on Ubuntu on Windows 10 #21

@scienceopen

suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04

@davidghiurco

how to get this build for Bash for Windows? or is it installed automatically with Windows update?

@scienceopen

You need Windows 14936 or newer, and no it will not autoupgrade, even when you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder) so backup anything you might need.

@deto
deto commented Nov 19, 2016

Can't you also run do-release-upgrade? So you don't lose all your files?

On Nov 19, 2016 2:42 PM, "Michael Hirsch" notifications@github.com wrote:

You need Windows 14936 or newer, and no it will not autoupgrade, even when
you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command
Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder)
so backup anything you might need.

โ€”
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx8YV96Cxh8PsNLNj8_GrV5e1QiZqks5q_3tUgaJpZM4Jd6On
.

@scienceopen

Some people do that, I wanted a clean start.

@bklooste

Same with CNTK ., KMP_AFFINITY=disabled made it work

@stehufntdev stehufntdev added the fixed label Jan 6, 2017
@stehufntdev
Collaborator

Closing this per the comment above, please let us know if there are any regressions in this scenario:

"suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04"

@stehufntdev stehufntdev closed this Jan 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment