brew install apache-arrow --HEAD fails due to not finding Python3 and NumPy

[asfimport]

For me this fails both on Intel and Apple silicone Macs, with similar error messages. It appears that what is in the brew formula alone is not sufficient to get an environment setup and built (via brew). Do we need to declare the numpy dependency here so that brew knows to install it in the version of python that it will be using in the build process later? Or specify this elswhere? (note ARROW-8998 would remove this dependency I believe, but it's not yet done).

(base) dragos@Dragoss-MacBook-Pro r % brew install apache-arrow --HEAD
==> Cloning https://github.com/apache/arrow.git
Updating /Users/dragos/Library/Caches/Homebrew/apache-arrow--git
From https://github.com/apache/arrow
   29892ba5c..65c131321  master     -> origin/master
==> Checking out branch master
Already on 'master'
Your branch is behind 'origin/master' by 20 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
HEAD is now at 65c131321 ARROW-14417: [R] Joins ignore projection on left dataset
Entering 'cpp/submodules/parquet-testing'
Entering 'testing'
/Users/dragos/Library/Caches/Homebrew/apache-arrow--git/cpp/submodules/parquet-testing
/Users/dragos/Library/Caches/Homebrew/apache-arrow--git/testing
==> cmake ../cpp -DCMAKE_FIND_PACKAGE_PREFER_CONFIG=TRUE -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON -DARROW_JEMALLOC=ON -DARROW_ORC=ON -DARROW_PARQUET=ON -DARROW_PLASMA=ON -DARROW_PROTOBUF_US
Last 15 lines from /Users/dragos/Library/Logs/Homebrew/apache-arrow/01.cmake:
-- Found approximate gRPC version: 1.36 (ARROW_FLIGHT_REQUIRE_TLSCREDENTIALSOPTIONS=)
CMake Error at /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) (found
  version "3.9.7")
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPython/Support.cmake:3166 (find_package_handle_standard_args)
  /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPython3.cmake:485 (include)
  cmake_modules/FindPython3Alt.cmake:54 (find_package)
  src/arrow/python/CMakeLists.txt:22 (find_package)


-- Configuring incomplete, errors occurred!
See also "/tmp/apache-arrow-20211021-75909-1pqubs7/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/apache-arrow-20211021-75909-1pqubs7/build/CMakeFiles/CMakeError.log".

Reporter: Dragoș Moldovan-Grünfeld / @dragosmg

Original Issue Attachments:

• Screenshot 2021-10-29 at 11.57.40.png

_{Note: This issue was originally created as ARROW-14420. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
Could you try the following patch and send a pull request to Homebrew if this works?

diff --git a/Formula/apache-arrow.rb b/Formula/apache-arrow.rb
index 3c11899d12e..d5c46061831 100644
--- a/Formula/apache-arrow.rb
+++ b/Formula/apache-arrow.rb
@@ -58,7 +58,7 @@ class ApacheArrow < Formula
       -DARROW_WITH_BROTLI=ON
       -DARROW_WITH_UTF8PROC=ON
       -DARROW_INSTALL_NAME_RPATH=OFF
-      -DPYTHON_EXECUTABLE=#{Formula["python@3.9"].bin/"python3"}
+      -DPython3_EXECUTABLE=#{Formula["python@3.9"].bin/"python3"}
     ]
 
     args << "-DARROW_MIMALLOC=ON" unless Hardware::CPU.arm?

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
It doesn't seem to be working for me - I get a similar error message (below with traceback).

dragos@Dragoss-MacBook-Pro ~ % brew install apache-arrow --HEAD
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
Updated 1 formula.

==> Cloning https://github.com/apache/arrow.git
Updating /Users/dragos/Library/Caches/Homebrew/apache-arrow--git
From https://github.com/apache/arrow
   311a9531e..9c57eb3e8  master     -> origin/master
==> Checking out branch master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
HEAD is now at 9c57eb3e8 ARROW-14189: [Docs] Add version dropdown to the sphinx docs
Entering 'cpp/submodules/parquet-testing'
Entering 'testing'
/Users/dragos/Library/Caches/Homebrew/apache-arrow--git/cpp/submodules/parquet-testing
/Users/dragos/Library/Caches/Homebrew/apache-arrow--git/testing
==> cmake ../cpp -DCMAKE_FIND_PACKAGE_PREFER_CONFIG=TRUE -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON -DARROW_JEMALLOC=ON -DARROW_ORC=ON -DARROW_PARQUET=ON -DARROW_PLASMA=ON -D
Last 15 lines from /Users/dragos/Library/Logs/Homebrew/apache-arrow/01.cmake:
-- Found approximate gRPC version: 1.36 (ARROW_FLIGHT_REQUIRE_TLSCREDENTIALSOPTIONS=)
CMake Error at /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) (found
  version "3.9.7")
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPython/Support.cmake:3166 (find_package_handle_standard_args)
  /usr/local/Cellar/cmake/3.21.3_1/share/cmake/Modules/FindPython3.cmake:485 (include)
  cmake_modules/FindPython3Alt.cmake:54 (find_package)
  src/arrow/python/CMakeLists.txt:22 (find_package)


-- Configuring incomplete, errors occurred!
See also "/tmp/apache-arrow-20211027-81715-1wniuql/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/apache-arrow-20211027-81715-1wniuql/build/CMakeFiles/CMakeError.log".

READ THIS: https://docs.brew.sh/Troubleshooting

Please create pull requests instead of asking for help on Homebrew's GitHub,
Twitter or any other official channels.

 

[asfimport]

Jonathan Keane / @jonkeane:
I might be misreading ARROW-8998 but it seems until that's done, we look for numpy at build time. @dragosmg could you try checking to see if numpy has been installed with whatever Python 3.9 is being used in this install (you might need to find the executable in brew or use brew to set your python3 to point to that)?

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
I removed Homebrew and re-installed everything. I currently have only one (in addition to whatever comes bundled with the macOS) version of Python 3.9 and NumPy 1.21.3. Both Python3 and NumPy were installed as dependencies for apache-arrow so I think one was installed with the other.

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
Some update. It looks like there might be some sort of issue with gcc

Python 3.9.7 (default, Oct 13 2021, 06:45:31)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/numpy/core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/usr/local/lib/python3.9/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/local/lib/python3.9/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/usr/local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 2): Symbol not found: ___addtf3
  Referenced from: /usr/local/opt/gcc/lib/gcc/11/libquadmath.0.dylib
  Expected in: /usr/local/lib/libgcc_s.1.dylib
 in /usr/local/opt/gcc/lib/gcc/11/libquadmath.0.dylib

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/numpy/__init__.py", line 150, in <module>
    from . import core
  File "/usr/local/lib/python3.9/site-packages/numpy/core/__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/usr/local/opt/python@3.9/bin/python3.9"
  * The NumPy version is: "1.21.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/usr/local/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-darwin.so, 2): Symbol not found: ___addtf3
  Referenced from: /usr/local/opt/gcc/lib/gcc/11/libquadmath.0.dylib
  Expected in: /usr/local/lib/libgcc_s.1.dylib
 in /usr/local/opt/gcc/lib/gcc/11/libquadmath.0.dylib

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
@jorisvandenbossche @kszucs could I ask you for an opinion on this? I wasn't able to figure out why the brew installation is tripping up on a Python / NumPy dependency. I tried @kou's suggestion, but it didn't solve my issue. I was able to replicate this on both of my machines (M1 Mac & 2018 Intel Mac).

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Just to be clear: the numpy import error is from just installing numpy with homebrew? (not yet related with building arrow?)
I am not familiar with homebrew, and I don't know what the reason could be, but that's something that should be solved first before attempting to build arrow I think.

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
Both Python and NumPy are installed as part of the arrow installation with homebrew brew install apache-arrow --HEAD. There is no error message when installing either of them. The error message surfaces a bit later with:

Could NOT find Python3 (missing: Python3_NumPy_INCLUDE_DIRS NumPy) (found
 version "3.9.7")

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

There is no error message when installing either of them

There might be no error message installing them, but clearly the installation did go wrong if you can't even import numpy. Do you know how python / numpy gets installed when doing brew install apache-arrow? Are there some log outputs related to this?

Does it only fail in case of --HEAD?

[asfimport]

Dragoș Moldovan-Grünfeld / @dragosmg:
It seems to fail only for --HEAD. I wasn't able to retrieve more specific log messages (than the error message above (before the import numpy one).

[asfimport]

Alenka Frim / @AlenkaF:
Tried to build R Arrow with Brew formula on M1 also. Yesterday in native, today in Rosetta. No luck. In my case there is no info about not finding NumPy though. PrtSrc attached.

Will try to change cmake in apache-arrow.rb to see if anything works.

[asfimport]

Jonathan Keane / @jonkeane:
Could you post the contents of ~/Library/Lobs/Homebrew/apache-arrow/02.make? The last 15 lines look like they don't actually contain the error that you're running into there.

[asfimport]

Alenka Frim / @AlenkaF:
Sry @jonkeane, I lost the file when reinstalling Homebrew I guess.

[assignUser]

We now have M1 nightlies that are green so this issue is fixed.