[R] create_package_with_all_dependencies() fails on MacOS 11.2

[asfimport]

I installed the Arrow Package inside a fresh R-Install on Mac OS 11.2 (BigSur) and tried to use the function create_package_with_all_dependencies():

> library(arrow)
> create_package_with_all_dependencies('/home/max/test.tar.gz')
Downloading Arrow source file
versuche URL 'https://mirror.dogado.de/cran/src/contrib/arrow_6.0.1.tar.gz'
Content type 'application/octet-stream' length 4774511 bytes (4.6 MB)
===================================================
downloaded 4.6 MB

Downloading files to /var/folders/dn/v941w15111s7858wxz6xwzzh0000gn/T//RtmpBjlYTP/file78c52918a/4e/arrow/tools/thirdparty_dependencies
readlink: illegal option -- f
usage: readlink [-n] [file ...]
Fehler in create_package_with_all_dependencies("/home/max/test.tar.gz") :
  Failed to download thirdparty dependencies

Environment: R on MacOS 11.2 (Big Sur)
Reporter: Maximilian König
Assignee: Karl Dunkle Werner / @karldw

Related issues:

• [R] create_package_with_all_dependencies() fails on Windows 10 (is cloned by)
• [R] arrow::create_package_with_all_dependencies() fails to download third party dependencies (fixes)
• [R] create_package_with_all_dependencies() fails on Windows 10 (is duplicated by)
• [R] arrow::create_package_with_all_dependencies() fails to download third party dependencies (relates to)

PRs and other links:

• GitHub Pull Request #12849

_{Note: This issue was originally created as ARROW-15092. Please see the migration documentation for further details.}

[asfimport]

Dewey Dunnington / @paleolimbot:
Thank you for reporting this! I can replicate this on MacOS as well:

arrow::create_package_with_all_dependencies(tempfile())
#> Downloading Arrow source file
#> Downloading files to /var/folders/gt/l87wjg8s7312zs9s7c1fgs900000gn/T//RtmplKo3qJ/file19d4737d41d5/arrow/tools/thirdparty_dependencies
#> Error in arrow::create_package_with_all_dependencies(tempfile()): Failed to download thirdparty dependencies

trying URL 'https://cran.rstudio.com/src/contrib/arrow_6.0.1.tar.gz'
Content type 'application/x-gzip' length 4774511 bytes (4.6 MB)
==================================================
downloaded 4.6 MB

readlink: illegal option -- f
usage: readlink [-n] [file ...]

It looks like the error is coming from here:

https://github.com/apache/arrow/blob/master/cpp/thirdparty/download_dependencies.sh#L33

[asfimport]

Nicola Crane / @thisisnic:
Looking into this, it looks like this is a Linux utility that doesn't exist on MacOS. I think we could either update this script to manually perform the steps instead of using readlink or using greadlink (if installed?) when on MacOS - see the below links for more info:

• https://stackoverflow.com/questions/1055671/how-can-i-get-the-behavior-of-gnus-readlink-f-on-a-mac
• http://biercoff.com/fixing-readlink-illegal-option-f-error-on-a-mac/

[asfimport]

Jonathan Keane / @jonkeane:
I would be curious if you could add a bit more information about what you're using this for [~max_koe] If you're trying to install arrow on an airgapped mac, you might have better luck downloading binaries from cran and installing those directly (e.g. from https://cran.r-project.org/web/packages/arrow/index.html).

[asfimport]

Maximilian König:
I was trying to use a Mac Laptop simply to download the artifacts necessary to install ARROW on an offline Machine (internal protected linux server). (Cf. ARROW-15000)

I had both a Windows and a Mac machine at my disposal, both failed in different ways when trying to use this function (create_package_with_all_dependencies)

For myself I finally managed to achive my task by setting up a new linux machine in AWS and using that. I reported this in the hope that future some user will not run into this same hurdle.
(I should also open the issue that occured on Windows probably).

[asfimport]

Jonathan Keane / @jonkeane:
Ah, yes either a separate issue, or even the error output on this one for windows would be helpful to try and make sure this is possible.

We also might want to make sure that none of the dependencies that we're downloading are platform dependent (I don't think they are, but we should double check that) to ensure that this is true

[asfimport]

Maximilian König:
I now created the bug report for windows, it is ARROW-15608

[asfimport]

Karl Dunkle Werner / @karldw:
Thanks for flagging these! I helped with parts of the offline installation scripting, and I'd be happy to help here.

create_package_with_all_dependencies() is currently designed to be run on a linux machine to produce a dependency bundle for offline installation on another linux machine. As you all have found, it hits minor snags on a Mac, and major snags when it tries to run the download_dependencies.sh shell script on a Windows machine.

The underlying files that are downloaded are listed in versions.txt. The files that get downloaded do not depend on the system running the command, and cmake will check the files' hashes to be sure.
 
One way to make this function more platform-independent would be to continue using {}versions.txt{}, but run the downloads in R instead of using {}download_dependencies.sh{}. Does that seem reasonable? This change might be easier said than done, because we need the DEPENDENCIES array with all the variables substituted in.

[asfimport]

Karl Dunkle Werner / @karldw:
Okay, I wrote some code that parses versions.txt. This overall approach might be too fragile to changes in the structure of versions.txt, but if it's worth building on, let me know, and I can open a PR. (CC @nealrichardson)
https://gist.github.com/karldw/1038f9b8605b4ee97842f4b65ed6c09f
 

I assumed a few design constraints:

• Cannot edit the form of versions.txt
• Only base R functions

[asfimport]

Jacob Wujciak / @assignUser:
Hi @karldw are you still interested in doing a PR for this? If so feel free to open one! Otherwise let me know :)

[asfimport]

Karl Dunkle Werner / @karldw:
Started: #12849

Feedback is very welcome here.

[asfimport]

Jonathan Keane / @jonkeane:
Issue resolved by pull request 12849
#12849