Issue installing via pip inside docker container for linux/arm64

[kykyi]

Outline

I am getting cython related errors when attempting to pip install sparse_dot_topn inside my dockerfile. I am unsure how the error is arising as I can see the files in the sparse_dot_topn repo and they have not changed in 2 years! Thanks in advance.

I am on an Apple M1 with macOS 13.2, unsure if that is relevant here though as we are inside the container.

Problem

FROM python:3.9.7-slim

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y build-essential cmake git wget
RUN pip install pip --upgrade

RUN pip install sparse_dot_topn==0.3.0 --no-cache-dir

Running the command:

docker buildx build --platform linux/amd64,linux/arm64 .

I get the following error but only in linux/arm64:

[linux/arm64 8/9] RUN pip install sparse_dot_topn==0.3.0 --no-cache-dir:                                                                                                                  
0.301 Collecting sparse_dot_topn==0.3.0                                                                                                                                                      
0.332   Downloading sparse_dot_topn-0.3.0.tar.gz (15 kB)                                                                                                                                     
0.338   Installing build dependencies: started                                                                                                                                               
1.371   Installing build dependencies: finished with status 'done'                                                                                                                           
1.372   Getting requirements to build wheel: started                                                                                                                                         
1.463   Getting requirements to build wheel: finished with status 'done'                                                                                                                     
1.466   Installing backend dependencies: started                                                                                                                                             
6.541   Installing backend dependencies: finished with status 'done'                                                                                                                         
6.542   Preparing metadata (pyproject.toml): started                                                                                                                                         
6.794   Preparing metadata (pyproject.toml): finished with status 'done'                                                                                                                     
6.798 Requirement already satisfied: setuptools>=42 in /usr/local/lib/python3.9/site-packages (from sparse_dot_topn==0.3.0) (57.5.0)
6.799 Requirement already satisfied: cython>=0.29.15 in /usr/local/lib/python3.9/site-packages (from sparse_dot_topn==0.3.0) (3.0.0)
6.799 Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.9/site-packages (from sparse_dot_topn==0.3.0) (1.25.1)
6.800 Requirement already satisfied: scipy>=1.2.3 in /usr/local/lib/python3.9/site-packages (from sparse_dot_topn==0.3.0) (1.11.1)
6.828 Building wheels for collected packages: sparse_dot_topn
6.829   Building wheel for sparse_dot_topn (pyproject.toml): started
8.972   Building wheel for sparse_dot_topn (pyproject.toml): finished with status 'error'
8.982   error: subprocess-exited-with-error
8.982   
8.982   × Building wheel for sparse_dot_topn (pyproject.toml) did not run successfully.
8.982   │ exit code: 1
8.982   ╰─> [166 lines of output]
8.982       running bdist_wheel
8.982       running build
8.982       running build_py
8.982       creating build
8.982       creating build/lib.linux-aarch64-cpython-39
8.982       creating build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       copying sparse_dot_topn/__init__.py -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       copying sparse_dot_topn/awesome_cossim_topn.py -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       running egg_info
8.982       writing sparse_dot_topn.egg-info/PKG-INFO
8.982       writing dependency_links to sparse_dot_topn.egg-info/dependency_links.txt
8.982       writing requirements to sparse_dot_topn.egg-info/requires.txt
8.982       writing top-level names to sparse_dot_topn.egg-info/top_level.txt
8.982       reading manifest file 'sparse_dot_topn.egg-info/SOURCES.txt'
8.982       reading manifest template 'MANIFEST.in'
8.982       adding license file 'LICENSE'
8.982       writing manifest file 'sparse_dot_topn.egg-info/SOURCES.txt'
8.982       copying sparse_dot_topn/array_wrappers.pxd -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       copying sparse_dot_topn/array_wrappers.pyx -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       copying sparse_dot_topn/sparse_dot_topn.pyx -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       copying sparse_dot_topn/sparse_dot_topn_parallel.h -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       copying sparse_dot_topn/sparse_dot_topn_source.h -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       copying sparse_dot_topn/sparse_dot_topn_threaded.pyx -> build/lib.linux-aarch64-cpython-39/sparse_dot_topn
8.982       running build_ext
8.982       /tmp/pip-build-env-qpr73hsx/normal/lib/python3.9/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-ac4zogpb/sparse-dot-topn_e59d95f11f06430da4966db7104429ac/sparse_dot_topn/array_wrappers.pxd
8.982         tree = Parsing.p_module(s, pxd, full_module_name)
8.982       Compiling ./sparse_dot_topn/array_wrappers.pyx because it changed.
8.982       [1/1] Cythonizing ./sparse_dot_topn/array_wrappers.pyx
8.982       building 'sparse_dot_topn.array_wrappers' extension
8.982       creating build/temp.linux-aarch64-cpython-39
8.982       creating build/temp.linux-aarch64-cpython-39/sparse_dot_topn
8.982       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -I/tmp/pip-build-env-qpr73hsx/normal/lib/python3.9/site-packages/numpy/core/include -c ./sparse_dot_topn/array_wrappers.cpp -o build/temp.linux-aarch64-cpython-39/./sparse_dot_topn/array_wrappers.o -std=c++0x -pthread -O3
8.982       g++ -shared -Wl,--strip-all build/temp.linux-aarch64-cpython-39/./sparse_dot_topn/array_wrappers.o -L/usr/local/lib -o build/lib.linux-aarch64-cpython-39/sparse_dot_topn/array_wrappers.cpython-39-aarch64-linux-gnu.so
8.982       /tmp/pip-build-env-qpr73hsx/normal/lib/python3.9/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-ac4zogpb/sparse-dot-topn_e59d95f11f06430da4966db7104429ac/sparse_dot_topn/sparse_dot_topn.pyx
8.982         tree = Parsing.p_module(s, pxd, full_module_name)
8.982       
8.982       Error compiling Cython file:
8.982       ------------------------------------------------------------
8.982       ...
8.982       # April 14, 2021
8.982       
8.982       # distutils: language = c++
8.982       
8.982       from libcpp.vector cimport vector
8.982       from array_wrappers cimport ArrayWrapper_int, ArrayWrapper_double
8.982       ^
8.982       ------------------------------------------------------------
8.982       
8.982       sparse_dot_topn/sparse_dot_topn.pyx:23:0: 'array_wrappers.pxd' not found
8.982       
8.982       Error compiling Cython file:
8.982       ------------------------------------------------------------
8.982       ...
8.982       # April 14, 2021
8.982       
8.982       # distutils: language = c++
8.982       
8.982       from libcpp.vector cimport vector
8.982       from array_wrappers cimport ArrayWrapper_int, ArrayWrapper_double
8.982       ^
8.982       ------------------------------------------------------------
8.982       
8.982       sparse_dot_topn/sparse_dot_topn.pyx:23:0: 'array_wrappers/ArrayWrapper_int.pxd' not found
8.982       
8.982       Error compiling Cython file:
8.982       ------------------------------------------------------------
8.982       ...
8.982       # April 14, 2021
8.982       
8.982       # distutils: language = c++
8.982       
8.982       from libcpp.vector cimport vector
8.982       from array_wrappers cimport ArrayWrapper_int, ArrayWrapper_double
8.982       ^
8.982       ------------------------------------------------------------
8.982       
8.982       sparse_dot_topn/sparse_dot_topn.pyx:23:0: 'array_wrappers/ArrayWrapper_double.pxd' not found
8.982       
8.982       Error compiling Cython file:
8.982       ------------------------------------------------------------
8.982       ...
8.982       
8.982           if nnz_max_is_too_small:
8.982       
8.982                   # raise Exception("In sparse_dot_topn.pyx")
8.982       
8.982                   c_indices = np.asarray(ArrayWrapper_int(vCj)).squeeze(axis=0)
8.982                                ^
8.982       ------------------------------------------------------------
8.982       
8.982       sparse_dot_topn/sparse_dot_topn.pyx:211:25: 'ArrayWrapper_int' is not a constant, variable or function identifier
8.982       
8.982       Error compiling Cython file:
8.982       ------------------------------------------------------------
8.982       ...
8.982           if nnz_max_is_too_small:
8.982       
8.982                   # raise Exception("In sparse_dot_topn.pyx")
8.982       
8.982                   c_indices = np.asarray(ArrayWrapper_int(vCj)).squeeze(axis=0)
8.982                   c_data = np.asarray(ArrayWrapper_double(vCx)).squeeze(axis=0)
8.982                             ^
8.982       ------------------------------------------------------------
8.982       
8.982       sparse_dot_topn/sparse_dot_topn.pyx:212:22: 'ArrayWrapper_double' is not a constant, variable or function identifier
8.982       Compiling ./sparse_dot_topn/sparse_dot_topn.pyx because it changed.
8.982       [1/1] Cythonizing ./sparse_dot_topn/sparse_dot_topn.pyx
8.982       Traceback (most recent call last):
8.982         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
8.982           main()
8.982         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
8.982           json_out['return_val'] = hook(**hook_input['kwargs'])
8.982         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
8.982           return _build_backend().build_wheel(wheel_directory, config_settings,
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 416, in build_wheel
8.982           return self._build_with_temp_dir(['bdist_wheel'], '.whl',
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 401, in _build_with_temp_dir
8.982           self.run_setup()
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
8.982           exec(code, locals())
8.982         File "<string>", line 65, in <module>
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
8.982           return distutils.core.setup(**attrs)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
8.982           return run_commands(dist)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
8.982           dist.run_commands()
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
8.982           self.run_command(cmd)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
8.982           super().run_command(command)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
8.982           cmd_obj.run()
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 343, in run
8.982           self.run_command("build")
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
8.982           self.distribution.run_command(command)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
8.982           super().run_command(command)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
8.982           cmd_obj.run()
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
8.982           self.run_command(cmd_name)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
8.982           self.distribution.run_command(command)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
8.982           super().run_command(command)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
8.982           cmd_obj.run()
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 84, in run
8.982           _build_ext.run(self)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
8.982           self.build_extensions()
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
8.982           self._build_extensions_serial()
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
8.982           self.build_extension(ext)
8.982         File "/tmp/pip-build-env-qpr73hsx/overlay/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
8.982           _build_ext.build_extension(self, ext)
8.982         File "/tmp/pip-build-env-qpr73hsx/normal/lib/python3.9/site-packages/Cython/Distutils/build_ext.py", line 122, in build_extension
8.982           new_ext = cythonize(
8.982         File "/tmp/pip-build-env-qpr73hsx/normal/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
8.982           cythonize_one(*args)
8.982         File "/tmp/pip-build-env-qpr73hsx/normal/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
8.982           raise CompileError(None, pyx_file)
8.982       Cython.Compiler.Errors.CompileError: ./sparse_dot_topn/sparse_dot_topn.pyx
8.982       [end of output]
8.982   
8.982   note: This error originates from a subprocess, and is likely not a problem with pip.
8.982   ERROR: Failed building wheel for sparse_dot_topn
8.982 Failed to build sparse_dot_topn
8.983 ERROR: Could not build wheels for sparse_dot_topn, which is required to install pyproject.toml-based projects
------

[kykyi]

Looks related to #76, #62,

[kykyi]

I did build it a few months ago so hope it can be done again, but I guess a work around could be to specify a .whl path to install and pass to docker build ...

[kposborne]

It looks like this is related to the recent Cython-3.0.0 release: https://pypi.org/project/Cython/#history. It was able to work around the problem with this example Dockerfile. It uses the --no-build-isolation flag to make it use the currently installed cython for building instead of taking the latest.

FROM python:3.9.7-slim

RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y build-essential cmake git wget
RUN pip install pip --upgrade

RUN pip install cython==0.29.36 numpy
RUN pip install sparse_dot_topn==0.3.0 --no-cache-dir --no-build-isolation

[sixdimensionalarray]

Not sure if it helps, but I also noticed that if I changed the imports in the sparse_dot_topn.pyx and sparse_dot_topn_threaded.pyx files and added the module prefix sparse_dot_topn. that it seemed to be able to build the wheels.

e.g. I changed this:

from array_wrappers cimport ArrayWrapper_int, ArrayWrapper_float, ArrayWrapper_double

to this:

from sparse_dot_topn.array_wrappers cimport ArrayWrapper_int, ArrayWrapper_float, ArrayWrapper_double

So I'm guessing something in the build process isn't setting a path to the modules or something correctly?

[kykyi]

Thanks @kposborne and @sixdimensionalarray for your swift replies 🙏

I was able to get the Dockerfile to build using the following as suggested by @kposborne

RUN pip install cython==0.29.36 numpy
RUN pip install sparse_dot_topn==0.3.0 --no-cache-dir --no-build-isolation

I had to also add

RUN pip install sparse-dot-topn-for-blocks --no-cache-dir --no-build-isolation

As sparse-dot-topn-for-blocks is a dep of another package and is based on sparse_dot_topn so needed similar treatment!

Thanks again 🚀

[asiunov]

@kykyi this is a workaround, but the issue still exists, right? Would you mind reopening the ticket?