Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Failed to build numpy on Android #21026

Closed
GalaxySnail opened this issue Feb 10, 2022 · 0 comments · Fixed by #21027
Closed

BUG: Failed to build numpy on Android #21026

GalaxySnail opened this issue Feb 10, 2022 · 0 comments · Fixed by #21027
Labels
00 - Bug

Comments

@GalaxySnail
Copy link
Contributor

GalaxySnail commented Feb 10, 2022 •
edited

Describe the issue:

Because Android doesn't support sem_open, python on Android doesn't support multiprocessing.Lock. Therefore, multiprocessing.Queue, multiprocessing.pool.Pool, concurrent.futures.ProcessPoolExecutor, and everything based on multiprocessing.Lock are unavailable on Android.

In numpy.distutils.ccompiler it uses multiprocessing.pool.ThreadPool to build parallel. But the ThreadPool inherits from multiprocessing.pool.Pool, it requires multiprocessing.SimpleQueue in its __init__ method, which is not necessary. Also, multiprocessing.pool.ThreadPool is not recommanded in the document. It will be fine to use concurrent.futures.ThreadPoolExecutor instead of multiprocessing.pool.ThreadPool.

Reproduce the code example:

pip download numpy==1.22.2
SETUPTOOLS_USE_DISTUTILS=local pip wheel --no-cache-dir ./numpy-1.22.2.zip

Error message:

creating build/temp.linux-aarch64-3.10/build/src.linux-aarch64-3.10/numpy/core/src/npymath
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.10/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.10/lib-dynload/_multiprocessing.cpython-310.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/data/data/com.termux/files/usr/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/data/data/com.termux/files/usr/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 230, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
    self.run_setup()
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 267, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 158, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 461, in <module>
    setup_package()
  File "setup.py", line 453, in setup_package
    setup(**metadata)
  File "/data/data/com.termux/files/usr/tmp/pip-req-build-8iir_yy8/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/data/data/com.termux/files/usr/tmp/pip-req-build-8iir_yy8/numpy/distutils/command/build.py", line 61, in run
    old_build.run(self)
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/data/data/com.termux/files/usr/tmp/pip-build-env-4f_9lkdl/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/data/data/com.termux/files/usr/tmp/pip-req-build-8iir_yy8/numpy/distutils/command/build_clib.py", line 167, in run
    self.build_libraries(self.libraries)
  File "/data/data/com.termux/files/usr/tmp/pip-req-build-8iir_yy8/numpy/distutils/command/build_clib.py", line 186, in build_libraries
    self.build_a_library(build_info, lib_name, libraries)
  File "setup.py", line 228, in build_a_library
    build_clib.build_a_library(self, build_info, lib_name, libraries)
  File "/data/data/com.termux/files/usr/tmp/pip-req-build-8iir_yy8/numpy/distutils/command/build_clib.py", line 365, in build_a_library
    objects += compiler.compile(
  File "/data/data/com.termux/files/usr/tmp/pip-req-build-8iir_yy8/numpy/distutils/ccompiler.py", line 89, in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
  File "/data/data/com.termux/files/usr/tmp/pip-req-build-8iir_yy8/numpy/distutils/ccompiler.py", line 359, in CCompiler_compile
    pool = multiprocessing.pool.ThreadPool(jobs)
  File "/data/data/com.termux/files/usr/lib/python3.10/multiprocessing/pool.py", line 927, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/data/data/com.termux/files/usr/lib/python3.10/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/data/data/com.termux/files/usr/lib/python3.10/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/data/com.termux/files/usr/lib/python3.10/multiprocessing/queues.py", line 342, in __init__
    self._rlock = ctx.Lock()
  File "/data/data/com.termux/files/usr/lib/python3.10/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/data/com.termux/files/usr/lib/python3.10/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

NumPy/Python version information:

>>> print(sys.version)
3.10.2 (main, Jan 16 2022, 02:57:51) [Clang 12.0.8 (https://android.googlesource.com/toolchain/llvm-project c935d99d7

Packaging script: https://github.com/termux/termux-packages/blob/master/packages/python/build.sh
GalaxySnail added a commit to GalaxySnail/numpy that referenced this issue Feb 10, 2022
@GalaxySnail
Use ThreadPoolExecutor instead of ThreadPool
dd98459 
Use `concurrent.futures.ThreadPoolExecutor` in distutils
instead of `multiprocessing.pool.ThreadPool`.

Fix numpy#21026
GalaxySnail added a commit to GalaxySnail/numpy that referenced this issue Feb 10, 2022
@GalaxySnail
BUG: use ThreadPoolExecutor instead of ThreadPool
204f931 
Use `concurrent.futures.ThreadPoolExecutor` in distutils
instead of `multiprocessing.pool.ThreadPool`.

Fix numpy#21026
@GalaxySnail GalaxySnail mentioned this issue Feb 10, 2022
Merged
charris pushed a commit to charris/numpy that referenced this issue Mar 2, 2022
@GalaxySnail @charris
BUG: use ThreadPoolExecutor instead of ThreadPool
23517b9 
Use `concurrent.futures.ThreadPoolExecutor` in distutils
instead of `multiprocessing.pool.ThreadPool`.

Fix numpy#21026
charris pushed a commit to charris/numpy that referenced this issue Mar 2, 2022
@GalaxySnail @charris
BUG: use ThreadPoolExecutor instead of ThreadPool
a794a6f 
Use `concurrent.futures.ThreadPoolExecutor` in distutils
instead of `multiprocessing.pool.ThreadPool`.

Fix numpy#21026
@charris charris mentioned this issue Mar 2, 2022
Merged
melissawm pushed a commit to melissawm/numpy that referenced this issue Apr 12, 2022
@GalaxySnail @melissawm
BUG: use ThreadPoolExecutor instead of ThreadPool
f2a965c 
Use `concurrent.futures.ThreadPoolExecutor` in distutils
instead of `multiprocessing.pool.ThreadPool`.

Fix numpy#21026
seberg pushed a commit to seberg/numpy that referenced this issue Apr 24, 2022
@GalaxySnail @seberg
BUG: use ThreadPoolExecutor instead of ThreadPool
ded9011 
Use `concurrent.futures.ThreadPoolExecutor` in distutils
instead of `multiprocessing.pool.ThreadPool`.

Fix numpy#21026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug
Projects
None yet
Milestone
No milestone
1 participant
@GalaxySnail