compileall hangs when accessing urandom even if number of workers is 1

[virtuald]

BPO	29877
Nosy	@brettcannon, @pitrou, @vstinner, @tiran, @methane, @PCManticore, @virtuald, @miss-islington
PRs	bpo-29877: compileall: import ProcessPoolExecutor only when needed #4856 [3.7] bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856) #10686 [3.6] bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856) #10687

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-24.01:37:57.669>
created_at = <Date 2017-03-22.05:57:16.434>
labels = ['3.7', 'library', 'performance']
title = 'compileall hangs when accessing urandom even if number of workers is 1'
updated_at = <Date 2018-11-24.01:37:57.668>
user = 'https://github.com/virtuald'

bugs.python.org fields:

activity = <Date 2018-11-24.01:37:57.668>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-24.01:37:57.669>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2017-03-22.05:57:16.434>
creator = 'virtuald'
dependencies = []
files = []
hgrepos = []
issue_num = 29877
keywords = ['patch']
message_count = 11.0
messages = ['289973', '308649', '308651', '308654', '308655', '308688', '308927', '330290', '330342', '330345', '330346']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'pitrou', 'vstinner', 'christian.heimes', 'methane', 'Claudiu.Popa', 'virtuald', 'miss-islington']
pr_nums = ['4856', '10686', '10687']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue29877'
versions = ['Python 3.6', 'Python 3.7']

[virtuald]

Found on Python 3.6 on a low-resource platform (NI RoboRIO), it seems that this occurs only because the ProcessPoolExecutor is being imported. A proposed fix would only import ProcessPoolExecutor if -j > 1. Stacktrace follows:

 /usr/local/bin/python3 -m compileall -j 1 /home/lvuser/py
^CTraceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/compileall.py", line 20, in <module>
    from concurrent.futures import ProcessPoolExecutor
  File "/usr/local/lib/python3.6/concurrent/futures/__init__.py", line 17, in <module>
    from concurrent.futures.process import ProcessPoolExecutor
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 53, in <module>
    import multiprocessing
  File "/usr/local/lib/python3.6/multiprocessing/__init__.py", line 16, in <module>
    from . import context
  File "/usr/local/lib/python3.6/multiprocessing/context.py", line 5, in <module>
    from . import process
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 311, in <module>
    _current_process = _MainProcess()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 298, in __init__
    self._config = {'authkey': AuthenticationString(os.urandom(32)),

[pitrou]

So calling urandom() hangs on your device because there's not enough entropy? What OS is this running? Modern Linux, at least, should never block in urandom() (AFAIK).

[vstinner]

Found on Python 3.6 on a low-resource platform (NI RoboRIO), ...

Hum, it looks like an "embedded device", likely with no entropy source.

To check if os.urandom() will block, try to call the following function in your main script:

>>> import os; os.getrandom(1, flags=os.GRND_NONBLOCK)
b'\xd1'

os.urandom() blocks at system startup until the entropy pool is initialized with 128 bytes of entropy. Once it's initialized, os.urandom() will never block again.

You may also check the number of entropy bytes available to /dev/random (Python uses /dev/urandom which is different):

$ cat /proc/sys/kernel/random/entropy_avail 
3748

For a longer explanation, see the PEP-524.

[pitrou]

Le 19/12/2017 à 15:50, STINNER Victor a écrit :

Hum, it looks like an "embedded device", likely with no entropy source.

If it does I/O (which it probably does, being used for robotics), it
should certainly be able to extract entropy from the outside world.

If it doesn't, it's an OS implementation issue.

Of course it probably doesn't hurt to import concurrent.futures lazily.
However, people should generally not expect us to never call urandom()
when importing stdlib modules.

[virtuald]

I'm sure that the platform (a RT linux customized by National Instruments) has issues related to urandom, as this has reared it's ugly head with other PEP-525 related issues also: https://bugs.python.org/issue29208

I'll have to try those suggestions to investigate the os.urandom issue.

[vstinner]

In kernel logs (look at dmesg), I found two messages:

vstinner@apu$ journalctl -k|grep random
déc. 14 23:10:28 apu kernel: random: fast init done
déc. 14 23:10:32 apu kernel: random: crng init done

Sadly, I don't know which one means that urandom is initialized.

See also https://wiki.archlinux.org/index.php/Random_number_generation

[virtuald]

Finally got around to looking at this.

# cat /proc/sys/kernel/random/entropy_avail
183

.. ran python3 -c 'import os; os.getrandom(1, flags=os.GRND_NONBLOCK)' a few times, but didn't run into a hang. Seems like the entropy_avail is staying at about the same level?

It's worth noting that I ran into this error with last year's firmware image, and I haven't actually tested this year's firmware image for this particular issue... just that I had it on my TODO list to fix for this year. :)

I think it's still a good change to make regardless.

[methane]

I am not sure PR 4856 is enhancement or bugfix.
But I think it's worth enough to backport to 3.6 and 3.7.
It caused a real UX problem.

[miss-islington]

New changeset 1d817e4 by Miss Islington (bot) (Dustin Spicuzza) in branch 'master':
bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
1d817e4

[miss-islington]

New changeset 903a3e8 by Miss Islington (bot) in branch '3.7':
bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
903a3e8

[miss-islington]

New changeset 879f5f3 by Miss Islington (bot) in branch '3.6':
bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
879f5f3