[R] R package is linking to the wrong OpenSSL on MacOS BigSur?

[paleolimbot]

Describe the bug, including details regarding any error messages, version, and platform.

From the r-binary-packages job failures: https://github.com/ursacomputing/crossbow/actions/runs/5442353558/jobs/9898961393

All of them look like this:

** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
ERROR: loading failed
 unable to load shared object '/Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so':
  dlopen(/Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so, 6): Symbol not found: _EVP_DigestSignUpdate
* removing ‘/Users/voltrondata/Library/R/arm64/4.1/library/arrow’
  Referenced from: /Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so
  Expected in: flat namespace
 in /Users/voltrondata/Library/R/arm64/4.1/library/00LOCK-arrow/00new/arrow/libs/arrow.so
Error: loading failed
Execution halted
Warning message:
In install.packages("arrow", type = "source", repos = sub("file://",  :
  installation of package ‘arrow’ had non-zero exit status
Error in library(arrow) : there is no package called ‘arrow’
Execution halted

Component(s)

R

[paleolimbot]

I mention OpenSSL here because the symbol that is not found is EVP_DigestSignUpdate...I suppose it could also be failing to link to any OpenSSL.

[paleolimbot]

I wonder if:

arrow/r/configure

Lines 111 to 120 in 7ecec6c

	# find openssl on macos. macOS ships with libressl. openssl is installable
	# with brew, but it is generally not linked. We can over-ride this and find
	# openssl but setting OPENSSL_ROOT_DIR (which cmake will pick up later in
	# the installation process). FWIW, arrow's cmake process uses this
	# same process to find openssl, but doing it now allows us to catch it in
	# nixlibs.R and throw a nicer error.
	if [ "${OPENSSL_ROOT_DIR}" = "" ] && brew --prefix openssl >/dev/null 2>&1; then
	export OPENSSL_ROOT_DIR="`brew --prefix openssl`"
	export PKG_CONFIG_PATH="${OPENSSL_ROOT_DIR}/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}"
	fi

is at odds with:

arrow/cpp/cmake_modules/FindOpenSSLAlt.cmake

Lines 22 to 39 in 7ecec6c

	if(APPLE AND NOT OPENSSL_ROOT_DIR)
	find_program(BREW brew)
	if(BREW)
	foreach(BREW_OPENSSL_VERSION "" "3" "3.0" "1.1")
	set(BREW_OPENSSL_PACKAGE "openssl")
	if(BREW_OPENSSL_VERSION)
	string(APPEND BREW_OPENSSL_PACKAGE "@${BREW_OPENSSL_VERSION}")
	endif()
	execute_process(COMMAND ${BREW} --prefix --installed ${BREW_OPENSSL_PACKAGE}
	OUTPUT_VARIABLE BREW_OPENSSL_PREFIX
	OUTPUT_STRIP_TRAILING_WHITESPACE)
	if(BREW_OPENSSL_PREFIX)
	set(OPENSSL_ROOT_DIR ${BREW_OPENSSL_PREFIX})
	break()
	endif()
	endforeach()
	endif()
	endif()

(that was added in #36336 )

[kou]

Ah, this is caused by mixing multiple OpenSSL versions.
It seems that openssl/*.h of OpenSSL 1.1 is used but libcrypto.dylib of OpenSSL 3 is used when building Apache Arrow C++. (Is autobrew used to build Apache Arrow C++?)
Could you check it?

[paleolimbot]

Yes, it's autobrew in that job ( https://github.com/ursacomputing/crossbow/actions/runs/5442353558/jobs/9897977884#step:13:60 ). I think that script is here: https://github.com/apache/arrow/blob/main/r/tools/autobrew and the formula is here: https://github.com/apache/arrow/blob/main/dev/tasks/homebrew-formulae/autobrew/apache-arrow-static.rb .

I think I'm remembering correctly that we always try to link to system openssl on MacOS and not the one provided by homebrew/autobrew... @assignUser may be more familiar with the details!

Also worth nothing that we're considering dropping autobrew to reduce the number of steps needed for the R package release; however, I don't know if doing a bundled build (i.e., running cmake on the bundled cpp source instead of autobrew) is more or less complicated with respect to openssl.

[assignUser]

may be more familiar with the details

eh, not really due to lack of macos locally ^^ but I can also have a look at this!

[raulcd]

homebrew-r-autobrew nightly job seems to fail with the same issue. Should this be considered a blocker for the release? I am marking it as a blocker at the moment.

[paleolimbot]

Thanks...yes, it's a blocker for release! CRAN does some builds on MacOS BigSur which is how most people will install the R package.

[assignUser]

As discussed in the PR this seems to be an issue with the gh runners explicitly running brew link openssl which causes the isssue. It would be nice if someone could try to repo locally with a default brew openssl install to make sure. I would assume that cran doesn't do this...

edit: or not as that solution does not work afterall

[paleolimbot]

Thank you all for working on this! I have a few MacOS environments locally to test with and will try a few things today. I do think that @kou is on the right track here (the GitHub MacOS environment has some modifications that are not typical for most users)...the build succeeds on MacOS 10.13 (where there is no brew).

[paleolimbot]

A few notes from my failed attempt today:

• Reverting GH-35710: [R] Followup improvements to new configure script #36435 didn't have any effect
• Recreating a local autobrew build is hard. The comment at https://github.com/apache/arrow/blob/main/r/configure#L241 suggests that you can do cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb tools/apache-arrow.rb. I've done this before successfully but currently it gives me an error (see details).

dewey@tidytuesday r % R CMD INSTALL . --preclean

• installing to library ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library’
• installing source package ‘arrow’ ...
  ** using staged installation
  *** Generating code with data-raw/codegen.R
  *** > 537 functions decorated with [[arrow|acero|dataset|substrait|parquet|s3|gcs|json::export]]
  *** > src/arrowExports.cpp not modified
  *** > R/arrowExports.R not modified
  *** Downloading apache-arrow
  **** Using local manifest for apache-arrow
  Fri 7 Jul 2023 15:38:24 ADT: Auto-brewing apache-arrow-static in /var/folders/4v/hvphchw50373drst9g5smz1w0000gn/T//build-apache-arrow-static...
  ==> Downloading https://ghcr.io/v2/homebrew/portable-ruby/portable-ruby/blobs/sha256:61029cec31c68a1fae1fa90fa876adf43d0becff777da793f9b5c5577f00567a
  Already downloaded: /var/folders/4v/hvphchw50373drst9g5smz1w0000gn/T//portable-ruby-2.6.10_1.el_capitan.bottle.tar.gz
  ==> Pouring portable-ruby-2.6.10_1.el_capitan.bottle.tar.gz
  ==> Tapping homebrew/core
  Cloning into '/private/var/folders/4v/hvphchw50373drst9g5smz1w0000gn/T/build-apache-arrow-static/Library/Taps/homebrew/homebrew-core'...
  Updating files: 100% (7101/7101), done.
  Tapped 3 commands and 6745 formulae (7,112 files, 453.7MB).
  ==> Tapping homebrew/cask
  Cloning into '/private/var/folders/4v/hvphchw50373drst9g5smz1w0000gn/T/build-apache-arrow-static/Library/Taps/homebrew/homebrew-cask'...
  Tapped 4233 casks (4,306 files, 365.7MB).
  Note: No available formula with the name "apache-arrow-static". Did you mean apache-arrow-glib?
  ==> Searching for similarly named formulae and casks...
  ==> Formulae
  apache-arrow-glib

To install apache-arrow-glib, run:
brew install apache-arrow-glib

==> Casks
apache-directory-studio

To install apache-directory-studio, run:
brew install --cask apache-directory-studio
cp: /var/folders/4v/hvphchw50373drst9g5smz1w0000gn/T//build-apache-arrow-static/Cellar///lib/*.a: No such file or directory
created /var/folders/4v/hvphchw50373drst9g5smz1w0000gn/T//build-apache-arrow-static/lib/libbrew.a
------------------------- NOTE ---------------------------
There was an issue preparing the Arrow C++ libraries.
See https://arrow.apache.org/docs/r/articles/install.html

ERROR: configuration failed for package ‘arrow’

• removing ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/arrow’
• restoring previous ‘/Library/Frameworks/R.framework/Versions/4.2/Resources/library/arrow’

[assignUser]

In the r-binary-packages job we modify the formula with the current hash and repo, maybe that is what's missing? There is some jinja in there but this is what we do, cwd is arrow/r:

 cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow*.rb tools/

      # Pin the git commit in the formula to match
      cd tools
      if [ "{{ is_fork }}" == "true" ]; then
        sed -i.bak -E -e 's/apache\/arrow.git"$/{{ arrow.github_repo.split("/") | join("\/") }}.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow*.rb
      else
        sed -i.bak -E -e 's/arrow.git"$/arrow.git", :revision => "'"{{ arrow.head }}"'"/' apache-arrow*.rb
      fi
      rm -f apache-arrow*.rb.bak

[paleolimbot]

Hmm...from the logs we also get ( https://github.com/ursacomputing/crossbow/actions/runs/5442353558/jobs/9897978025#step:13:5350 )

-- Check for working CXX compiler: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-apache-arrow-static/Library/Homebrew/shims/mac/super/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Note at CMakeLists.txt:66 (message):
  The Google Cloud C++ client libraries use the native OpenSSL library.  In
  most macOS systems, you need to set the OPENSSL_ROOT_DIR environment
  variable to find this dependency, for example:

      cmake -DOPENSSL_ROOT_DIR=/usr/local/opt/openssl ...

  You have not set this variable.  Most likely, this will result in a broken
  build with fairly obscure error messages.  If your environment does not
  require setting OPENSSL_ROOT_DIR, you can disable this check using:

      cmake -DGOOGLE_CLOUD_CPP_ENABLE_MACOS_OPENSSL_CHECK=OFF ...

I see set(OPENSSL_ROOT_DIR ${BREW_OPENSSL_PREFIX}) in the current version of "find openssl"...somehow that isn't propagating to the Google Cloud Platform sdk?

[nealrichardson]

• Recreating a local autobrew build is hard. The comment at https://github.com/apache/arrow/blob/main/r/configure#L241 suggests that you can do cp ../dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb tools/apache-arrow.rb. I've done this before successfully but currently it gives me an error (see details).

Now that there are two autobrew formulae, I think you have to copy them both, i.e. also apache-arrow-static.rb. That's the one that will get used on a contemporary version of macOS. See r/tools/autobrew

[raulcd]

This is expected to be on 13.0.0, right?

[paleolimbot]

We don't technically need this in 13.0.0 to release the R package but it will probably make your life easier because it fixes the binary packages and autobrew jobs.

[raulcd]

Thanks @paleolimbot , I'll cherry-pick it