New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect whether LLVM has SVML enabled and override SVML setting if not present #3006

Closed
seibert opened this Issue May 31, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@seibert
Contributor

seibert commented May 31, 2018

The bug in #2998 (and likely the one reported in the discussion of #2938) has revealed a situation where array expressions will give the wrong answer. Triggering this bug requires:

  • The ICC runtime (specifically libsvml) is present in the user's environment
  • The user is using an llvmlite statically linked against a version of LLVM that has not been patched with SVML support.
  • [Edit: Forgot one more condition] The platform is 64-bit.

We incorrectly assumed that this combination would be extremely unlikely, and if did happen, LLVM would raise an error. This was a mistake, as it seems:

  1. There are a number of Windows users getting llvmlite from the Chris Gohlke packages or from conda-forge, which do not apply all of our LLVM patches. (Note that anyone who gets llvmlite from the Numba channel on Anaconda Cloud, the default channel from Anaconda, or pip installs the wheels posted to PyPI are unaffected by this problem.)
  2. Many Windows users of Numba have the ICC runtime present as Intel Fortran is commonly used to compile SciPy for Windows due to challenges getting other Fortran compilers to work on Windows.
  3. Unpatched LLVM does not raise an error when we enable SVML autovectorization, but rather seems to generate incorrect code instead.

To correct this, we're going to add a function to llvmlite to verify if the LLVM being used has the patch. (Note, as LLVM is statically linked to llvmlite, it does not matter what LLVM is present on the end user system, only which LLVM is used to build llvmlite.) Numba will look for this function, and if it is not present or returns false (SVML support not available), it will disable the SVML code generation on that platform, regardless of whether the ICC runtime is present.

Although it affects a small number of users, we view this as a critical bug, and will be making patch releases of Numba 0.38 and llvmlite 0.23 outside of our normal release cycle.

@seibert

This comment has been minimized.

Contributor

seibert commented May 31, 2018

Note that as mentioned in the linked issue, the workaround before these new releases is to set:

NUMBA_DISABLE_INTEL_SVML=1

in your environment.

@cgohlke

This comment has been minimized.

Contributor

cgohlke commented Jun 1, 2018

FWIW, I rebuilt the llvmlite binaries at https://www.lfd.uci.edu/~gohlke/pythonlibs/#llvmlite using "the patches".

stuartarchibald added a commit to stuartarchibald/numba that referenced this issue Jun 1, 2018

Augment SVML detection with llvmlite SVML patch detection.
This adds defensive behaviour to SVML detection following changes
to llvmlite that added a function to declare whether SVML was enabled,
via patching LLVM, at compile time. This is a critical patch that
corrects erroneous behaviour, for context see numba#3006.

Closes numba#3006
Fixes numba#2998

stuartarchibald added a commit to stuartarchibald/numba that referenced this issue Jun 1, 2018

Augment SVML detection with llvmlite SVML patch detection.
This adds defensive behaviour to SVML detection following changes
to llvmlite that added a function to declare whether SVML was enabled,
via patching LLVM, at compile time. This is a critical patch that
corrects erroneous behaviour, for context see numba#3006.

Closes numba#3006
Fixes numba#2998

@seibert seibert closed this in #3008 Jun 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment