[CI][Homebrew] Properly enable Gandiva and improve testing

[asfimport]

ARROW-9086 enabled Gandiva in our Homebrew formula, but when I tried to add that to the official Homebrew formula at release time, it failed. See some discussion at Homebrew/homebrew-core#58581, though unfortunately the build logs are gone.

It turns out that the testing that Homebrew does is more thorough than the install/audit we do in CI. See https://github.com/Homebrew/homebrew-core/pull/58581/checks?check_run_id=915732878 for example. They install, build the bottle, then remove all dependencies and reinstall the bottle. Since this failed, what I think it means is that llvm is not a build-only dependency for Gandiva--it built but couldn't run successfully because llvm had been removed.

cc @kou

Reporter: Neal Richardson / @nealrichardson

_{Note: This issue was originally created as ARROW-9572. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
How about forcing using the system libc++ like v8 does?
https://github.com/Homebrew/homebrew-core/blob/master/Formula/v8.rb#L101-L102

diff --git a/dev/tasks/homebrew-formulae/apache-arrow.rb b/dev/tasks/homebrew-formulae/apache-arrow.rb
index c38b8ce51..f078e49d4 100644
--- a/dev/tasks/homebrew-formulae/apache-arrow.rb
+++ b/dev/tasks/homebrew-formulae/apache-arrow.rb
@@ -24,6 +24,8 @@ class ApacheArrow < Formula
 
   def install
     ENV.cxx11
+    # link against system libc++ instead of llvm provided libc++
+    ENV.remove "HOMEBREW_LIBRARY_PATHS", Formula["llvm"].opt_lib
     args = %W[
       -DARROW_FLIGHT=ON
       -DARROW_GANDIVA=ON

For testing, it seems that we can use brew test-bot --only-formulae for checking correct build dependency.

[asfimport]

Kouhei Sutou / @kou:
FYI: Gandiva was enabled by Homebrew/homebrew-core#60189

[asfimport]

Neal Richardson / @nealrichardson:
Yeah I saw; I was going to pull that change into our CI and fix the testing (but if you want to take it, please feel free :)

[asfimport]

Krisztian Szucs / @kszucs:
Since it has been added upstream I'm postponing to 3.0.

[asfimport]

Neal Richardson / @nealrichardson:
@kou you took care of this in ARROW-10273 right?

[asfimport]

Kouhei Sutou / @kou:
Unfortunately, no.
ARROW-10273 just fixes the way how to run self test described in https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request#create-your-pull-request-from-a-new-branch . (It also backports the change in Homebrew/homebrew-core#60189 .)
But it doesn't check build dependency that is done by Homebrew's CI: https://github.com/Homebrew/homebrew-core/blob/master/.github/workflows/tests.yml#L64-L72
We can't detect a bug that LLVM is needed at run-time yet.

[asfimport]

Krisztian Szucs / @kszucs:
Postponing to 4.0