cannot import hashlib when openssl is missing

[xdegaye]

BPO	36544
Nosy	@vstinner, @xdegaye, @miss-islington
PRs	bpo-36544 : Fix regression introduced in bpo-36146 #12708 bpo-36544: setup.py reports missing _hashlib #12750

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-09.15:46:15.787>
created_at = <Date 2019-04-06.16:18:21.048>
labels = ['type-bug', '3.8', 'build']
title = 'cannot import hashlib when openssl is missing'
updated_at = <Date 2019-04-09.17:24:44.150>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2019-04-09.17:24:44.150>
actor = 'xdegaye'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-09.15:46:15.787>
closer = 'xdegaye'
components = ['Build']
creation = <Date 2019-04-06.16:18:21.048>
creator = 'xdegaye'
dependencies = []
files = []
hgrepos = []
issue_num = 36544
keywords = ['patch']
message_count = 8.0
messages = ['339536', '339537', '339538', '339569', '339571', '339762', '339773', '339790']
nosy_count = 3.0
nosy_names = ['vstinner', 'xdegaye', 'miss-islington']
pr_nums = ['12708', '12750']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36544'
versions = ['Python 3.8']

[xdegaye]

Python is built natively in a docker container (based on ubuntu bionic) that lacks openssl.

pydev@979e9e009b08:~/build/python-native$ ./python
Python 3.8.0a3+ (heads/master-dirty:d6bf6f2, Apr  6 2019, 14:43:30)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
ERROR:root:code for hash blake2b was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type blake2b
ERROR:root:code for hash blake2s was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type blake2s
ERROR:root:code for hash sha3_224 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha3_224
ERROR:root:code for hash sha3_256 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha3_256
ERROR:root:code for hash sha3_384 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha3_384
ERROR:root:code for hash sha3_512 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha3_512
ERROR:root:code for hash shake_128 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type shake_128
ERROR:root:code for hash shake_256 was not found.
Traceback (most recent call last):
  File "/home/pydev/cpython/Lib/hashlib.py", line 244, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/home/pydev/cpython/Lib/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type shake_256
>>>

[xdegaye]

Changeset 5ec33a1 appends to the new detect_openssl_hashlib() method, the detection for the _sha256, _sha512, _md5, ... extension modules. These modules were detected unconditionnaly before this change and are not detected anymore now when openssl is not supported by the platform.

[xdegaye]

Working on a PR

[xdegaye]

Steps to reproduce (requires docker support, about 500 MB and only few minutes):

1. Create the file named 'Dockerfile' whose content is:

####################################
FROM ubuntu

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
bash \
build-essential \
git \
sed \
&& rm -rf /var/lib/apt/lists/*

# Run as user 'pydev'.
RUN groupadd pydev && useradd --no-log-init -m -g pydev pydev
USER pydev:pydev
ENV HOME=/home/pydev
WORKDIR $HOME
####################################

2. Create the ubuntu:bpo-36544 image

    $ docker build --tag=ubuntu:bpo-36544 .

3. Run the image and enters bash as user pydev

    $ docker run -it ubuntu:bpo-36544

As user pydev, build python and import hashlib:

    pydev@5a8ffc33955d:~$ git clone -b master --single-branch --depth 1 https://github.com/python/cpython.git
    pydev@70e0722a76f2:~$ cd cpython/ && ./configure && make
    pydev@70e0722a76f2:~/cpython$ ./python -c "import hashlib"

[xdegaye]

Check that PR 12708 fixes the bug:

Run the ubuntu:bpo-36544 image and enters bash as user pydev

    $ docker run -it ubuntu:bpo-36544

As user pydev, clone python and fetch [PR 12708](https://github.com/python/cpython/pull/12708), build python and import hashlib:

    pydev@e4cfd6f74c2b:~$ git clone -b master --single-branch --depth 1 https://github.com/python/cpython.git
    pydev@e4cfd6f74c2b:~$ cd cpython/ && ./configure && make
    pydev@e4cfd6f74c2b:~/cpython$ git fetch --depth 1 origin pull/12708/head:pr_12708 && git checkout pr_12708
    pydev@e4cfd6f74c2b:~/cpython$ make
    pydev@e4cfd6f74c2b:~/cpython$ ./python -c "import hashlib"

[miss-islington]

New changeset 2ee077f by Miss Islington (bot) (xdegaye) in branch 'master':
bpo-36544 : Fix regression introduced in bpo-36146 (GH-12708)
2ee077f

[vstinner]

Xavier de Gaye: Oops! I also saw such errors but I didn't understand them. Thanks for the fix!

I wrote PR 12750 to fix another minor issue which isn't directly related to your fix (nor to my previous refactoring).

[xdegaye]

Thanks Victor for the refactoring of setup.py. That was badly needed indeed.