New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anaconda's scipy.test() fails with "OMP: Error #100: Fatal system error detected #785

Closed
Evfro opened this Issue Aug 5, 2016 · 19 comments

Comments

Projects
None yet
9 participants
@Evfro

Evfro commented Aug 5, 2016

  • A brief description

SciPy is a python package for scientific computing. Normally it's installed with Anaconda distribution, that also ships Intel's MKL library (with LAPACK and BLAS routines).

  • Expected results

Executing scipy.test() from within python completes with the report on all errors and failures discovered.

  • Actual results (with terminal output if applicable)

scipy.test() doesn't complete and terminates python with the following error:

OMP: Error #100: Fatal system error detected.
OMP: System error #22: Invalid argument
Aborted (core dumped)
  • Your Windows build number

Microsoft Windows [Version 10.0.14393]

  • Steps / commands required to reproduce the error

Run from terminal:

python -c 'import scipy; scipy.test()'
  • Strace of the failing command

https://gist.github.com/Evfro/2d0d83745a92d3f6e2f52ce3675a273d

The last thing before error there is the following line:
sched_getaffinity(0, 4, 0x7fc7757a7e00) = -1 EINVAL (Invalid argument)

  • Required packages and commands to install

Install latest miniconda (Python 2.7):

wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
bash Miniconda*.sh

Update apt-get (prevents some errors):

sudo apt-get update -qq

Install scipy, numpy and nose (for running tests):

conda install numpy scipy nose

This will additionally install MKL and libgfortran-3.
Note: test doesn't fail if nomkl versions of numpy and scipy are installed, however, MKL (especially its LAPACK and BLAS) is required for some other packages, so it's not really an option.

@deto

This comment has been minimized.

Show comment
Hide comment
@deto

deto Aug 5, 2016

Which packages is MKL required for? As far as I know, it was only added recently to their free-tier Linux installer so most things should work fine with nomkl.

deto commented Aug 5, 2016

Which packages is MKL required for? As far as I know, it was only added recently to their free-tier Linux installer so most things should work fine with nomkl.

@Evfro

This comment has been minimized.

Show comment
Hide comment
@Evfro

Evfro Aug 5, 2016

For example, this ones fails at compilation time with exactly the same error (and it's not possible to install w/o MKL):
https://github.com/oseledets/ttpy
I guess, there could be many more scientific computing projects, that also rely on MKL's BLAS and LAPACK.

Evfro commented Aug 5, 2016

For example, this ones fails at compilation time with exactly the same error (and it's not possible to install w/o MKL):
https://github.com/oseledets/ttpy
I guess, there could be many more scientific computing projects, that also rely on MKL's BLAS and LAPACK.

@Evfro

This comment has been minimized.

Show comment
Hide comment
@Evfro

Evfro Aug 5, 2016

Sorry, my last comment was wrong. TTpy packages doesn't fail at compile time, but it fails when some functions are called (for example, tt.tensor).

What I've found is that setting environmental variable KMP_AFFINITY=disabled resolves the problem. However, I have no idea what implications it might have and not sure whether it's a proper workaround.

And it actually means, that the problem is not with MKL itself, but with OpenMP becoming faulty with any other value of KMP_AFFINITY (I tried all).

Evfro commented Aug 5, 2016

Sorry, my last comment was wrong. TTpy packages doesn't fail at compile time, but it fails when some functions are called (for example, tt.tensor).

What I've found is that setting environmental variable KMP_AFFINITY=disabled resolves the problem. However, I have no idea what implications it might have and not sure whether it's a proper workaround.

And it actually means, that the problem is not with MKL itself, but with OpenMP becoming faulty with any other value of KMP_AFFINITY (I tried all).

@srinath29

This comment has been minimized.

Show comment
Hide comment
@srinath29

srinath29 Aug 6, 2016

Yeah.. And all the packages built on top of it. I was using statsmodels.api.GLSAR and it was also failing with the same problem. I opened a bug in statsmodels
statsmodels/statsmodels#3142

srinath29 commented Aug 6, 2016

Yeah.. And all the packages built on top of it. I was using statsmodels.api.GLSAR and it was also failing with the same problem. I opened a bug in statsmodels
statsmodels/statsmodels#3142

@srinath29

This comment has been minimized.

Show comment
Hide comment
@srinath29

srinath29 Aug 7, 2016

Any solutions ? I am little anxious as one of my work depends on it.

srinath29 commented Aug 7, 2016

Any solutions ? I am little anxious as one of my work depends on it.

@deto

This comment has been minimized.

Show comment
Hide comment
@deto

deto Aug 7, 2016

If nobody has a fix and you really need it, you might want to just run
Ubuntu in a VM. I've installed Anaconda with MKL doing this before. Not
as optimal as Bash on Windows but it'll get you moving.

You can also run much of the data science stack in Windows (not Bash) just
fine using Anaconda.

On Aug 7, 2016 12:29 AM, "srinath" notifications@github.com wrote:

Any solutions ? I am little anxious as one of my work depends on it.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx0us8t2TFzKCfsjh6LgOmZUWQCX5ks5qdYlSgaJpZM4Jd6On
.

deto commented Aug 7, 2016

If nobody has a fix and you really need it, you might want to just run
Ubuntu in a VM. I've installed Anaconda with MKL doing this before. Not
as optimal as Bash on Windows but it'll get you moving.

You can also run much of the data science stack in Windows (not Bash) just
fine using Anaconda.

On Aug 7, 2016 12:29 AM, "srinath" notifications@github.com wrote:

Any solutions ? I am little anxious as one of my work depends on it.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx0us8t2TFzKCfsjh6LgOmZUWQCX5ks5qdYlSgaJpZM4Jd6On
.

@srinath29

This comment has been minimized.

Show comment
Hide comment
@srinath29

srinath29 Aug 7, 2016

Thanks for the suggestion. I am already doing that. Actually I am in final stages of development of a benchmark https://github.com/srinath29/UltimateAnalyticsBenchmark
I want to test it on Bash on windows and I have score of running it natively on windows also. So I am bit curious.

srinath29 commented Aug 7, 2016

Thanks for the suggestion. I am already doing that. Actually I am in final stages of development of a benchmark https://github.com/srinath29/UltimateAnalyticsBenchmark
I want to test it on Bash on windows and I have score of running it natively on windows also. So I am bit curious.

@Evfro

This comment has been minimized.

Show comment
Hide comment
@Evfro

Evfro Aug 7, 2016

Have you tried setting KMP_AFFINITY=disabled? Does it really kills performance? I haven't tested yet, would be interesting to see.

Evfro commented Aug 7, 2016

Have you tried setting KMP_AFFINITY=disabled? Does it really kills performance? I haven't tested yet, would be interesting to see.

@srinath29

This comment has been minimized.

Show comment
Hide comment
@srinath29

srinath29 Aug 7, 2016

@Evfro
Hey... It is working!!!!!
These are the results
cmd
bash

See the score at the bottom. Bash is performing better than windows. Now I am not sure it is because making it disabled or it's bash's power. Can you do the same test on your system? I am very curious to see how it is working.
I think this should be labeled as workaround .

srinath29 commented Aug 7, 2016

@Evfro
Hey... It is working!!!!!
These are the results
cmd
bash

See the score at the bottom. Bash is performing better than windows. Now I am not sure it is because making it disabled or it's bash's power. Can you do the same test on your system? I am very curious to see how it is working.
I think this should be labeled as workaround .

@Evfro

This comment has been minimized.

Show comment
Hide comment
@Evfro

Evfro Aug 9, 2016

@srinath29
I haven't closely looked into your code, but the thing to keep in mind when measuring time is that some python time-measuring functions behave differently on Linux and Windows. The most reliable way I found is to use

from timeit import default_timer

Now, that said, I can confirm that there are performance differences in certain types of operations.
For example, this simple code

a = numpy.random.random((10000, 10000))
%timeit a.mean()

gives me on average 52ms on bash and 133ms on windows.

However, more involved calculations like computing sparse SVD (scipy.sparse.linalg.svds) give me roughly equal results: 151ms on bash against 156ms on windows (for 6040 by 3706 matrix with 99% sparsity).

And while this KMP_AFFINITY thing does the trick, I would really like to hear from core developers team of BashOnWindows whether this is an appropriate modification (and whether it can be made default without additional user actions). From Intel's website: "Thread affinity is supported on Windows systems and versions of Linux systems that have kernel support for thread affinity..." So the problem can be either due to a bug or due to a kernel-level limitation.

Evfro commented Aug 9, 2016

@srinath29
I haven't closely looked into your code, but the thing to keep in mind when measuring time is that some python time-measuring functions behave differently on Linux and Windows. The most reliable way I found is to use

from timeit import default_timer

Now, that said, I can confirm that there are performance differences in certain types of operations.
For example, this simple code

a = numpy.random.random((10000, 10000))
%timeit a.mean()

gives me on average 52ms on bash and 133ms on windows.

However, more involved calculations like computing sparse SVD (scipy.sparse.linalg.svds) give me roughly equal results: 151ms on bash against 156ms on windows (for 6040 by 3706 matrix with 99% sparsity).

And while this KMP_AFFINITY thing does the trick, I would really like to hear from core developers team of BashOnWindows whether this is an appropriate modification (and whether it can be made default without additional user actions). From Intel's website: "Thread affinity is supported on Windows systems and versions of Linux systems that have kernel support for thread affinity..." So the problem can be either due to a bug or due to a kernel-level limitation.

@davidghiurco

This comment has been minimized.

Show comment
Hide comment
@davidghiurco

davidghiurco Oct 18, 2016

does anyone know if there's a fix other than disabling MKL affinity? I'm using a numpy/scipy stack for one of my classes and i develop on Bash for windows and am wondering if there's any fix for this or if I still have to do it the hacky way (and can rest assured that unless professor runs BashOnWindows too, he won't get this error)

davidghiurco commented Oct 18, 2016

does anyone know if there's a fix other than disabling MKL affinity? I'm using a numpy/scipy stack for one of my classes and i develop on Bash for windows and am wondering if there's any fix for this or if I still have to do it the hacky way (and can rest assured that unless professor runs BashOnWindows too, he won't get this error)

@drhirsch

This comment has been minimized.

Show comment
Hide comment
@drhirsch

drhirsch Nov 19, 2016

suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04

drhirsch commented Nov 19, 2016

suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04

@davidghiurco

This comment has been minimized.

Show comment
Hide comment
@davidghiurco

davidghiurco Nov 19, 2016

how to get this build for Bash for Windows? or is it installed automatically with Windows update?

davidghiurco commented Nov 19, 2016

how to get this build for Bash for Windows? or is it installed automatically with Windows update?

@drhirsch

This comment has been minimized.

Show comment
Hide comment
@drhirsch

drhirsch Nov 19, 2016

You need Windows 14936 or newer, and no it will not autoupgrade, even when you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder) so backup anything you might need.

drhirsch commented Nov 19, 2016

You need Windows 14936 or newer, and no it will not autoupgrade, even when you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder) so backup anything you might need.

@deto

This comment has been minimized.

Show comment
Hide comment
@deto

deto Nov 19, 2016

Can't you also run do-release-upgrade? So you don't lose all your files?

On Nov 19, 2016 2:42 PM, "Michael Hirsch" notifications@github.com wrote:

You need Windows 14936 or newer, and no it will not autoupgrade, even when
you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command
Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder)
so backup anything you might need.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx8YV96Cxh8PsNLNj8_GrV5e1QiZqks5q_3tUgaJpZM4Jd6On
.

deto commented Nov 19, 2016

Can't you also run do-release-upgrade? So you don't lose all your files?

On Nov 19, 2016 2:42 PM, "Michael Hirsch" notifications@github.com wrote:

You need Windows 14936 or newer, and no it will not autoupgrade, even when
you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command
Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder)
so backup anything you might need.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx8YV96Cxh8PsNLNj8_GrV5e1QiZqks5q_3tUgaJpZM4Jd6On
.

@drhirsch

This comment has been minimized.

Show comment
Hide comment
@drhirsch

drhirsch Nov 19, 2016

Some people do that, I wanted a clean start.

drhirsch commented Nov 19, 2016

Some people do that, I wanted a clean start.

@bklooste

This comment has been minimized.

Show comment
Hide comment
@bklooste

bklooste Nov 20, 2016

Same with CNTK ., KMP_AFFINITY=disabled made it work

bklooste commented Nov 20, 2016

Same with CNTK ., KMP_AFFINITY=disabled made it work

@stehufntdev stehufntdev added the fixed label Jan 6, 2017

@stehufntdev

This comment has been minimized.

Show comment
Hide comment
@stehufntdev

stehufntdev Jan 6, 2017

Collaborator

Closing this per the comment above, please let us know if there are any regressions in this scenario:

"suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04"

Collaborator

stehufntdev commented Jan 6, 2017

Closing this per the comment above, please let us know if there are any regressions in this scenario:

"suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04"

@grayskripko

This comment has been minimized.

Show comment
Hide comment
@grayskripko

grayskripko May 25, 2017

@scienceopen it is possible to upgrade Ubuntu 14 to 16
According to https://media.readthedocs.org/pdf/wsl-guide/latest/wsl-guide.pdf

You can also upgrade to the latest version of Ubuntu with the following command (caution, this will take quite some time)!:
$ sudo -S apt-mark hold procps strace sudo
$ sudo -S env RELEASE_UPGRADER_NO_SCREEN=1 do-release-upgrade

I've got 16.04 on my machine after these steps

grayskripko commented May 25, 2017

@scienceopen it is possible to upgrade Ubuntu 14 to 16
According to https://media.readthedocs.org/pdf/wsl-guide/latest/wsl-guide.pdf

You can also upgrade to the latest version of Ubuntu with the following command (caution, this will take quite some time)!:
$ sudo -S apt-mark hold procps strace sudo
$ sudo -S env RELEASE_UPGRADER_NO_SCREEN=1 do-release-upgrade

I've got 16.04 on my machine after these steps

@RomainBrault RomainBrault referenced this issue Jul 19, 2017

Open

error? #4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment