compiler error building the _sha3 module with Intel 2018 compilers

[wscullin]

BPO	33174
Nosy	@tiran, @ned-deily, @wscullin, @boegel, @edmondac

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-06-20.18:52:42.485>
created_at = <Date 2018-03-28.20:01:01.971>
labels = ['build', 'expert-installation']
title = 'compiler error building the _sha3 module with Intel 2018 compilers'
updated_at = <Date 2018-06-20.19:01:43.649>
user = 'https://github.com/wscullin'

bugs.python.org fields:

activity = <Date 2018-06-20.19:01:43.649>
actor = 'boegel'
assignee = 'none'
closed = True
closed_date = <Date 2018-06-20.18:52:42.485>
closer = 'ned.deily'
components = ['Installation']
creation = <Date 2018-03-28.20:01:01.971>
creator = 'wscullin'
dependencies = []
files = []
hgrepos = []
issue_num = 33174
keywords = []
message_count = 10.0
messages = ['314619', '315884', '317591', '319941', '319943', '319945', '319948', '320100', '320104', '320106']
nosy_count = 5.0
nosy_names = ['christian.heimes', 'ned.deily', 'wscullin', 'boegel', 'edmondac']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue33174'
versions = ['Python 3.6']

[wscullin]

When building Python 3.6.X and later with icc (18.0.0.128 or
18.0.1.163), there's an error building the _sha3 module with any
optimization level other than -O0:

building '_sha3' extension
icc -pthread -fPIC -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3
-Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-parameter
-Wno-missing-field-initializers -fp-model strict -I./Include -I.
-I/usr/local/include
-I/derp/Python-3.6.4/Include
-I/derp/Python-3.6.4 -c
/derp/Python-3.6.4/Modules/_sha3/
sha3module.c -o
build/temp.linux-x86_64-3.6/derp/Python-3.6.4/Modules/_sha3/sha3module.o
": internal error: ** The compiler has encountered an unexpected problem.
** Segmentation violation signal raised. **
Access violation or stack overflow. Please contact Intel Support for assistance.

compilation aborted for
/derp/Python-3.6.4/Modules/_sha3/sha3module.c
(code 4)
...
[ jlselogin2: Python-3.6.4 ]$

if I drop to -O0, compilation works every time. I haven't found
disabling any particular set of optimizations to be useful in
obtaining a successful build.
...
[ jlselogin2: Python-3.6.4 ]$

dropping to -O0, compilation works every time. I haven't found
disabling any particular set of optimizations to be useful in
obtaining a successful build with icc.

Intel has been notified and a bug filed as this is really a compiler bug. On the Python side, it does not appear possible to use Modules/Setup to drop the optimization level for just _sha3 and I'm hunting for a workaround.

[boegel]

Willian: on which OS and (Intel?) processor generation are you seeing this?

We're not seeing this on CentOS 7.4.1708 with Python 3.6.4 and icc 18.0.1.163 on Intel Haswell or Intel Skylake.
We have seen this with other software on Intel Skylake though...

Have you tried with icc 18.0.2 (which was released March 21st 2018)?

[wscullin]

Hi Kenneth:

I can recreate this issue on Haswell, Skylake-X, and KNL running SLES 12, Clear Linux, and Centos 7.5. I've tried 18.1 and 18.2. It's still present across the board with Intel C/C++ 18.0.2.199. On the Centos 7.5.1804 image, the build will seemingly complete, but since _sha3 didn't actually compile, the regression tests will fail with errors of the form:

$ /dev/shm/testinstall/icc/python/3.6.5/bin/python3 -m test
ERROR:root:code for hash sha3_224 was not found.
Traceback (most recent call last):
  File "/dev/shm/testinstall/icc/python/3.6.5/lib/python3.6/hashlib.py", line 121, in __get_openssl_constructor
    f = getattr(_hashlib, 'openssl_' + name)
AttributeError: module '_hashlib' has no attribute 'openssl_sha3_224'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/dev/shm/testinstall/icc/python/3.6.5/lib/python3.6/hashlib.py", line 243, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/dev/shm/testinstall/icc/python/3.6.5/lib/python3.6/hashlib.py", line 128, in __get_openssl_constructor
    return __get_builtin_constructor(name)
  File "/dev/shm/testinstall/icc/python/3.6.5/lib/python3.6/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha3_224
ERROR:root:code for hash sha3_256 was not found.
Traceback (most recent call last):
  File "/dev/shm/testinstall/icc/python/3.6.5/lib/python3.6/hashlib.py", line 121, in __get_openssl_constructor
    f = getattr(_hashlib, 'openssl_' + name)
AttributeError: module '_hashlib' has no attribute 'openssl_sha3_256'

...

Searching about, it looks like you may have seen something similar in EasyBuild:

easybuilders/easybuild-easyconfigs#5086

but it doesn't look like the root cause was sussed out.

• William

[boegel]

@william: someone in the EasyBuild community seems to have figured out the culprit, and came up with a patch to work around the problem, see https://github.com/easybuilders/easybuild-easyconfigs/pull/6447/files#diff-bdbfca2206414b3b37794b77c0abb8e8 .

Any feedback on that?

[tiran]

I read the patch. The manual loop unrolling is ugly but mostly ok. But
I'm -1 on the "-O0" option and won't accept the patch unless it works with standard optimization. sha3 is already slow. Without optimization, the algorithm grinds to a halt.

Did anybody contact Intel and reported the bug? It's clearly a bug in ICC. GCC, clang and MSVC compile the code just fine.

[edmondac]

I made the patch. I'm not sure it's suitable to commit back as it is, as it unrolls a macro but there seems to be a choice of macro in the code (depending on certain conditions).

The problem is that icc can't handle a line that is so long.

I didn't log the bug with Intel - but I agree that it's a compiler bug and not a Python bug. I'll log it now...

-Ed

[edmondac]

See https://software.intel.com/en-us/forums/intel-c-compiler/topic/780574

[boegel]

After a bit of back and forth, we have figured out the underlying cause for the problem with the Intel compiler: if the stack limit is not set to a specific value (see output of "ulimit -s"), the Intel compilers fail with a segfault or Interal Compiler Error (ICE).

But if the stack limit is set to 'unlimited' (e.g. via "ulimit -s unlimited"), then the compilation works fine...

See also easybuilders/easybuild-easyconfigs#6484 and our (optional) workaround in easybuilders/easybuild-easyblocks#1441

[ned-deily]

Thanks for all the investigate work and thanks for the finding what appears to be an easy workaround, @boegel. Clearly, though, this is not a Python problem and, with an external workaround (e.g. increasing the process stack size via ulimit), there is no need for a workaround in Python itself so I'm closing this issue. Suggest making sure that the workaround is brought to the Intel compiler team's attention.

[boegel]

Fully agreed that this is not something that should be fixed in the Python code base.

Moreover, the problem has been fixed in Intel compilers 2018 update 3 as far as I can tell (tested with both Python 3.6.4 and 3.7.0rc1).

I can't reproduce the problem anymore with that version of the Intel compilers, even after setting a hard stack limit (e.g. "ulimit -s 10240"), while it was easy to reproduce with the 2018 update 1 version.