Can't install due to dependency on numpy

[earlzero]

Hi,

TL;DR I can't install tensorflow_transform with pip in new venv. More details below.

In new virtual environment (python3 -m venv tt/) I'm trying to install tensorflow_transform with the following command: pip install -U tensorflow-transform.

My expectation: the library gets installed,
Result: I'm getting the following error ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects, more details in more complete listing (I omitted large part of output in the middle as there are 3k lines there).
OS: Ubuntu 22.04.1 LTS
Python version: Python 3.10.6

Could you please help me resolve this issue?

Output

Collecting tensorflow-transform
  Using cached tensorflow_transform-1.10.1-py3-none-any.whl (439 kB)
Collecting pyarrow<7,>=6
  Using cached pyarrow-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25.6 MB)
Requirement already satisfied: numpy<2,>=1.16 in /home/ez/dev/ml/tt/lib/python3.10/site-packages (from tensorflow-transform) (1.23.4)
Collecting apache-beam[gcp]<3,>=2.40
  Using cached apache-beam-2.42.0.zip (2.9 MB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting absl-py<2.0.0,>=0.9
  Using cached absl_py-1.3.0-py3-none-any.whl (124 kB)
Collecting pydot<2,>=1.2
  Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Collecting tensorflow-transform
  Using cached tensorflow_transform-1.10.0-py3-none-any.whl (439 kB)
  Using cached tensorflow_transform-1.9.0-py3-none-any.whl (436 kB)
Collecting tensorflow-metadata<1.10.0,>=1.9.0
  Using cached tensorflow_metadata-1.9.0-py3-none-any.whl (51 kB)
Collecting tensorflow-transform
  Using cached tensorflow_transform-1.8.0-py3-none-any.whl (435 kB)
Collecting tensorflow-metadata<1.9.0,>=1.8.0
  Using cached tensorflow_metadata-1.8.0-py3-none-any.whl (50 kB)
Collecting tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<2.9,>=1.15.5
  Using cached tensorflow-2.8.3-cp310-cp310-manylinux2010_x86_64.whl (498.5 MB)
Collecting tensorflow-transform
  Downloading tensorflow_transform-1.7.0-py3-none-any.whl (433 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 433.6/433.6 kB 3.9 MB/s eta 0:00:00
  Downloading tensorflow_transform-1.6.1-py3-none-any.whl (427 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 427.5/427.5 kB 26.3 MB/s eta 0:00:00
  Downloading tensorflow_transform-1.6.0-py3-none-any.whl (427 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 427.5/427.5 kB 28.5 MB/s eta 0:00:00
  Downloading tensorflow_transform-1.5.0-py3-none-any.whl (422 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 422.5/422.5 kB 26.8 MB/s eta 0:00:00
Collecting numpy<1.20,>=1.16
  Downloading numpy-1.19.5.zip (7.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 29.2 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting tensorflow-transform
  Downloading tensorflow_transform-1.4.1-py3-none-any.whl (413 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 413.1/413.1 kB 64.5 MB/s eta 0:00:00
Collecting tensorflow-metadata<1.5.0,>=1.4.0
  Downloading tensorflow_metadata-1.4.0-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.5/48.5 kB 5.1 MB/s eta 0:00:00
Collecting protobuf<4,>=3.13
  Using cached protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting absl-py<0.13,>=0.9
  Downloading absl_py-0.12.0-py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.4/129.4 kB 8.2 MB/s eta 0:00:00
Collecting tensorflow-transform
  Downloading tensorflow_transform-1.4.0-py3-none-any.whl (413 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 413.1/413.1 kB 28.5 MB/s eta 0:00:00
  Downloading tensorflow_transform-1.3.0-py3-none-any.whl (407 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 407.7/407.7 kB 28.8 MB/s eta 0:00:00
Collecting tensorflow-metadata<1.3.0,>=1.2.0
  Downloading tensorflow_metadata-1.2.0-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.5/48.5 kB 8.2 MB/s eta 0:00:00
Collecting tensorflow-transform
  Downloading tensorflow_transform-1.2.0-py3-none-any.whl (406 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 406.4/406.4 kB 29.5 MB/s eta 0:00:00
Collecting pyarrow<3,>=1
  Downloading pyarrow-2.0.0.tar.gz (58.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.9/58.9 MB 24.3 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [3402 lines of output]
      Ignoring numpy: markers 'python_version < "3.7"' don't match your environment
      Collecting cython>=0.29
        Using cached Cython-0.29.32-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
      Collecting numpy==1.16.0
        Downloading numpy-1.16.0.zip (5.1 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 18.3 MB/s eta 0:00:00
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting setuptools
        Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB)
....
            In file included from /usr/include/python3.10/Python.h:77,                                                                                      [88/1879]
                             from numpy/core/src/multiarray/scalartypes.c.src:3:
            /usr/include/python3.10/pyhash.h:10:38: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘double’
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);                                                                                        
                  |                                      ^~~~~~~~~~                                                                                                  
            numpy/core/src/multiarray/scalartypes.c.src:2975:16: error: too few arguments to function ‘_Py_HashDouble’
             2975 |     hashreal = _Py_HashDouble((double)
                  |                ^~~~~~~~~~~~~~   
            In file included from /usr/include/python3.10/Python.h:77,
                             from numpy/core/src/multiarray/scalartypes.c.src:3:
            /usr/include/python3.10/pyhash.h:10:23: note: declared here                                                                                              
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
                  |                       ^~~~~~~~~~~~~~           
            numpy/core/src/multiarray/scalartypes.c.src:2981:31: error: incompatible type for argument 1 of ‘_Py_HashDouble’
             2981 |     hashimag = _Py_HashDouble((double)                                                                                                           
                  |                               ^~~~~~~~
                  |                               |                   
                  |                               double                        
             2982 |             PyArrayScalar_VAL(obj, C@name@).imag); 
                  |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~          
            In file included from /usr/include/python3.10/Python.h:77,
                             from numpy/core/src/multiarray/scalartypes.c.src:3:                                                                                     
            /usr/include/python3.10/pyhash.h:10:38: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘double’
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);     
                  |                                      ^~~~~~~~~~               
            numpy/core/src/multiarray/scalartypes.c.src:2981:16: error: too few arguments to function ‘_Py_HashDouble’
             2981 |     hashimag = _Py_HashDouble((double)                                                                                                           
                  |                ^~~~~~~~~~~~~~                                 
            In file included from /usr/include/python3.10/Python.h:77,            
                             from numpy/core/src/multiarray/scalartypes.c.src:3:                                                                                     
            /usr/include/python3.10/pyhash.h:10:23: note: declared here                                                                                              
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);     
                  |                       ^~~~~~~~~~~~~~                          
            numpy/core/src/multiarray/scalartypes.c.src: In function ‘longdouble_arrtype_hash’:                                                                      
            numpy/core/src/multiarray/scalartypes.c.src:2967:27: error: incompatible type for argument 1 of ‘_Py_HashDouble’
             2967 |     return _Py_HashDouble((double) PyArrayScalar_VAL(obj, @name@));                                                                              
            In file included from /usr/include/python3.10/Python.h:77,
                             from numpy/core/src/multiarray/scalartypes.c.src:3:
            /usr/include/python3.10/pyhash.h:10:38: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘double’                            
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);                                                                                        
                  |                                      ^~~~~~~~~~  
            numpy/core/src/multiarray/scalartypes.c.src:2967:12: error: too few arguments to function ‘_Py_HashDouble’                                               
             2967 |     return _Py_HashDouble((double) PyArrayScalar_VAL(obj, @name@));                                                                              
                  |            ^~~~~~~~~~~~~~                                 
            In file included from /usr/include/python3.10/Python.h:77,                                                                                               
                             from numpy/core/src/multiarray/scalartypes.c.src:3:                                                                                     
            /usr/include/python3.10/pyhash.h:10:23: note: declared here
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
                  |                       ^~~~~~~~~~~~~~         
            numpy/core/src/multiarray/scalartypes.c.src: In function ‘clongdouble_arrtype_hash’:                                                                     
            numpy/core/src/multiarray/scalartypes.c.src:2975:31: error: incompatible type for argument 1 of ‘_Py_HashDouble’                                         
             2975 |     hashreal = _Py_HashDouble((double)              
                  |                               ^~~~~~~~         
                  |                               |                                                                                                                  
                  |                               double           
             2976 |             PyArrayScalar_VAL(obj, C@name@).real);  
                  |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            In file included from /usr/include/python3.10/Python.h:77,
                             from numpy/core/src/multiarray/scalartypes.c.src:3:
            /usr/include/python3.10/pyhash.h:10:38: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘double’                            
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);                                                                                        
                  |                                      ^~~~~~~~~~  
            numpy/core/src/multiarray/scalartypes.c.src:2975:16: error: too few arguments to function ‘_Py_HashDouble’                                               
             2975 |     hashreal = _Py_HashDouble((double)                                                                                                           
                  |                ^~~~~~~~~~~~~~                                                                                                                    
            In file included from /usr/include/python3.10/Python.h:77,                                                                                                                            from numpy/core/src/multiarray/scalartypes.c.src:3:                                                                                                 /usr/include/python3.10/pyhash.h:10:23: note: declared here                                                                                                             10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);                                                                                                          |                       ^~~~~~~~~~~~~~                                                                                                                         numpy/core/src/multiarray/scalartypes.c.src:2981:31: error: incompatible type for argument 1 of ‘_Py_HashDouble’                                                      2981 |     hashimag = _Py_HashDouble((double)                                                                                                                             |                               ^~~~~~~~                                                                                                           
                  |                               |                               
                  |                               double                          
             2982 |             PyArrayScalar_VAL(obj, C@name@).imag);                                                                                               
                  |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            In file included from /usr/include/python3.10/Python.h:77,            
                             from numpy/core/src/multiarray/scalartypes.c.src:3:                                                                                     
            /usr/include/python3.10/pyhash.h:10:38: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘double’                            
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);     
                  |                                      ^~~~~~~~~~                                                                                                  
            numpy/core/src/multiarray/scalartypes.c.src:2981:16: error: too few arguments to function ‘_Py_HashDouble’                                               
             2981 |     hashimag = _Py_HashDouble((double)                        
                  |                ^~~~~~~~~~~~~~                       
            In file included from /usr/include/python3.10/Python.h:77,            
                             from numpy/core/src/multiarray/scalartypes.c.src:3:  
            /usr/include/python3.10/pyhash.h:10:23: note: declared here           
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);   
              10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
                  |                       ^~~~~~~~~~~~~~
            numpy/core/src/multiarray/scalartypes.c.src: In function ‘half_arrtype_hash’:
            numpy/core/src/multiarray/scalartypes.c.src:2997:27: error: incompatible type for argument 1 of ‘_Py_HashDouble’
             2997 |     return _Py_HashDouble(npy_half_to_double(PyArrayScalar_VAL(obj, Half)));
                  |                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                  |                           |
                  |                           double
            In file included from /usr/include/python3.10/Python.h:77,
                             from numpy/core/src/multiarray/scalartypes.c.src:3:
            /usr/include/python3.10/pyhash.h:10:38: note: expected ‘PyObject *’ {aka ‘struct _object *’} but argument is of type ‘double’
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
                  |                                      ^~~~~~~~~~
            numpy/core/src/multiarray/scalartypes.c.src:2997:12: error: too few arguments to function ‘_Py_HashDouble’
             2997 |     return _Py_HashDouble(npy_half_to_double(PyArrayScalar_VAL(obj, Half)));
                  |            ^~~~~~~~~~~~~~
            In file included from /usr/include/python3.10/Python.h:77,
                             from numpy/core/src/multiarray/scalartypes.c.src:3:
            /usr/include/python3.10/pyhash.h:10:23: note: declared here
               10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
                  |                       ^~~~~~~~~~~~~~
            numpy/core/src/multiarray/scalartypes.c.src: In function ‘longdouble_arrtype_hash’:
            numpy/core/src/multiarray/scalartypes.c.src:2968:1: warning: control reaches end of non-void function [-Wreturn-type]
             2968 | }
                  | ^
            numpy/core/src/multiarray/scalartypes.c.src: In function ‘float_arrtype_hash’:
            numpy/core/src/multiarray/scalartypes.c.src:2968:1: warning: control reaches end of non-void function [-Wreturn-type]
             2968 | }
                  | ^
            numpy/core/src/multiarray/scalartypes.c.src: In function ‘half_arrtype_hash’:
            numpy/core/src/multiarray/scalartypes.c.src:2998:1: warning: control reaches end of non-void function [-Wreturn-type]
             2998 | }
                  | ^
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/vdot.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/umath/loops.c
            x86_64-linux-gnu-gcc: numpy/core/src/umath/umathmodule.c
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/typeinfo.c
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/flagsobject.c
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/usertypes.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/multiarray/nditer_templ.c
            x86_64-linux-gnu-gcc: numpy/core/src/umath/ufunc_object.c
            x86_64-linux-gnu-gcc: numpy/core/src/umath/reduction.c
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/getset.c
            x86_64-linux-gnu-gcc: numpy/core/src/umath/ufunc_type_resolution.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/npymath/ieee754.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/npymath/npy_math_complex.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/array_assign.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/mem_overlap.c
            x86_64-linux-gnu-gcc: numpy/core/src/umath/override.c
            x86_64-linux-gnu-gcc: numpy/core/src/npymath/halffloat.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/npy_longdouble.c
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/nditer_api.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/ucsnarrow.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/common/npy_cpu_features.c
            x86_64-linux-gnu-gcc: numpy/core/src/npymath/npy_math.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/ufunc_override.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/cblasfuncs.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/numpyos.c
            x86_64-linux-gnu-gcc: numpy/core/src/common/python_xerbla.c
            x86_64-linux-gnu-gcc: numpy/core/src/umath/extobj.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/umath/scalarmath.c
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/mapping.c
            x86_64-linux-gnu-gcc: numpy/core/src/multiarray/methods.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/umath/matmul.c
            x86_64-linux-gnu-gcc: build/src.linux-x86_64-3.10/numpy/core/src/umath/clip.c
            error: Command "x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DHAVE_CBLAS -Ibuild/src.linux-x86_64-3.10/numpy/core/src/umath -Ibuild/src.linux-x86_64-3.10/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.10/numpy/core/src/common -Inumpy/core/include -Ibuild/src.linux-x86_64-3.10/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/ez/dev/ml/tt/include -I/usr/include/python3.10 -Ibuild/src.linux-x86_64-3.10/numpy/core/src/common -Ibuild/src.linux-x86_64-3.10/numpy/core/src/npymath -c build/src.linux-x86_64-3.10/numpy/core/src/multiarray/scalartypes.c -o build/temp.linux-x86_64-3.10/build/src.linux-x86_64-3.10/numpy/core/src/multiarray/scalartypes.o -MMD -MF build/temp.linux-x86_64-3.10/build/src.linux-x86_64-3.10/numpy/core/src/multiarray/scalartypes.o.d -std=c99" failed with exit status 1
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for numpy
      Failed to build numpy
      ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[gaikwadrahul8]

@earlzero

I have tried to install tensorflow-transform library on Google Colab and I'm able to install by using the same command pip install tensorflow-transform so could you please try to create new virtual environment and please use below versions specifications and see is it working for you or not and even I have added gist file here for your reference

pip install numpy == 1.21.6
pip install python == 3.7

If you still face the same error here is some additional resource I'm adding for you so you can try it once and let us know is it working or not for you ?

Thank you!

[earlzero]

I tried to create new virtual environment and run both commands. First of them succeeded, second failed (I googled and it seems I can't manage version of python with pip).

I guess it is not working due to my 3.10.6 version of Python. It is not listed as supported (3.7, 3.8, 3.9) in the badge on README.md. Colab uses Python 3.7.15. When I used anaconda with Python 3.9.13 it also worked.

I tried to follow recommendation from the link you provided, but with no luck (I didn't find pyproject.toml file in the virtual environment).

Should we rename this issue to highlight Python 3.10 compatibility issue?

[gaikwadrahul8]

@earlzero

Thank you for trying above commands and at the moment TFX and other libraries do not support python version 3.10 so I would suggest you to please go with python version less than 3.10, TFX and other libraries with python 3.10 support will be released very soon and here is reference which you can follow to install Tensorflow-Transform library and I hope it will help you to resolve your issue and let us know if the issue still persists with python version less than 3.10

Thanks!

[earlzero]

Thank you! I think the ticket can be closed now.

[gaikwadrahul8]

@earlzero

Thank you for the confirmation and now I'm closing this issue

Thanks!