[Python] file INSTALL cannot duplicate symlink because: A directory already exists at that location

[nbro10]

Describe the bug, including details regarding any error messages, version, and platform.

I was having this issue #34120, which I wasn't able to solve. So, for 1-2 weeks, I didn't do anything about it. Now, I'm back to it because I actually need to solve it. I tried to do a simple pip install pyarrow and I get this different error

Collecting pyarrow
  Using cached pyarrow-11.0.0.tar.gz (1.0 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.16.6 in /Users/ME/Library/Caches/pypoetry/virtualenvs/cornucopia-dyX6J5JG-py3.7/lib/python3.7/site-packages (from pyarrow) (1.21.6)
Building wheels for collected packages: pyarrow
  Building wheel for pyarrow (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pyarrow (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [550 lines of output]
      running bdist_wheel

...
      CMake Error at cmake_install.cmake:58 (file):
        file INSTALL cannot duplicate symlink
      
          /opt/homebrew/include/arrow
      
        at
      
          /private/var/folders/9l/5rlxs0hd68j70xl9_4q01shr0001cb/T/pip-install-dto72zm3/pyarrow_441ee5835a0b469c9333c69d970d8694/build/lib.macosx-12.5-arm64-cpython-37/pyarrow/include/arrow
      
        because: A directory already exists at that location
      
      
      make: *** [install] Error 1
      error: command '/Applications/CMake.app/Contents/bin/cmake' failed with exit code 2
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects

I don't know if this is due to some leftovers of my previous attempts to install pyarrow and arrow from source and with brew (as reported here #34120) or not. In any case, I'm not sure how to solve this problem and I think other people may face it.

I tried to do

• a brew reinstall apache-arrow and then pip install pyarrow
• a brew uninstall apache-arrow and then pip install pyarrow

But I still get the error above

So, now that I'm not specifying version 7.0.0 now. If I do, I get another completely different error

Collecting pyarrow==7.0.0
  Using cached pyarrow-7.0.0.tar.gz (816 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.16.6 in /Users/ME/Library/Caches/pypoetry/virtualenvs/cornucopia-dyX6J5JG-py3.7/lib/python3.7/site-packages (from pyarrow==7.0.0) (1.21.6)
Building wheels for collected packages: pyarrow
  Building wheel for pyarrow (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pyarrow (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [649 lines of output]

...


          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'pyarrow.vendored' as an importable package,
          but it is not listed in the `packages` configuration of setuptools.
      
          'pyarrow.vendored' has been automatically added to the distribution only
          because it may contain data files, but this behavior is likely to change
          in future versions of setuptools (and therefore is considered deprecated).
      
          Please make sure that 'pyarrow.vendored' is included as a package by using
          the `packages` configuration field or the proper discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
      
          You can read more about "package discovery" and "data files" on setuptools
          documentation page.
      
      
      !!
      
        check.warn(importable)
      error: command '/Applications/CMake.app/Contents/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects

Python version: 3.7.13

Component(s)

Python

[nbro10]

After having installed apache-arrow a second time with brew uninstall apache-arrow and having face this other issue #34396, if I do ls -l /opt/homebrew/include/arrow, now I get ls: /opt/homebrew/include/arrow: No such file or directory, so there should not be any symlink anymore. If I try to do pip install pyarrow, I get the error

Collecting pyarrow
  Using cached pyarrow-11.0.0.tar.gz (1.0 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.16.6 in /Users/ME/Library/Caches/pypoetry/virtualenvs/cornucopia-dyX6J5JG-py3.7/lib/python3.7/site-packages (from pyarrow) (1.21.6)
Building wheels for collected packages: pyarrow
  Building wheel for pyarrow (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pyarrow (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [334 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-12.5-arm64-cpython-37
      creating build/lib.macosx-12.5-arm64-cpython-37/pyarrow


...


      CMake Error at CMakeLists.txt:268 (find_package):
        By not providing "FindArrow.cmake" in CMAKE_MODULE_PATH this project has
        asked CMake to find a package configuration file provided by "Arrow", but
        CMake did not find one.
      
        Could not find a package configuration file provided by "Arrow" with any of
        the following names:
      
          ArrowConfig.cmake
          arrow-config.cmake
      
        Add the installation prefix of "Arrow" to CMAKE_PREFIX_PATH or set
        "Arrow_DIR" to a directory containing one of the above files.  If "Arrow"
        provides a separate development package or SDK, be sure it has been
        installed.
      
      
      -- Configuring incomplete, errors occurred!
      error: command '/Applications/CMake.app/Contents/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects

I suppose I get this error because I first need the backend, i.e. I need to do brew install apache-arrow, but the latter seems to create a symbolic link issue...

[phalverson]

I don't know if this is due to some leftovers of my previous attempts to install pyarrow and arrow from source and with brew or not. In any case, I'm not sure how to solve this problem and I think other people may face it.

I've just run into this same problem, first time.

      CMake Error at cmake_install.cmake:58 (file):
        file INSTALL cannot duplicate symlink
      
          /opt/homebrew/include/arrow
      
        at
      
          /private/var/folders/.../build/lib.macosx-12.6-arm64-cpython-37/pyarrow/include/arrow
      
        because: A directory already exists at that location

It might still be a config issue but more likely an issue with the pyarrow build. It definitely shouldn't be installing to the homebrew libdir, makes me think there's something awry with the install_prefix.

Time to start digging through source code 😞

[kou]

You can try #34674 by:

git clone https://github.com/kou/arrow.git
cd arrow
git switch python-include-arrow-symlink
pip install ./python