error in noaa_stageX functions

[jmzobitz]

running
weather <- noaa_stage1()

I get the following error message:
Error in (function (anonymous, access_key, secret_key, session_token, : Cannot call fs___S3FileSystem__create(). See https://arrow.apache.org/docs/r/articles/install.html for help installing Arrow C++ libraries.

relevant R versions:

• R version 4.4
• arrow version 15.0.1

Searching the web for help wasn't useful - I don't know if this is related to my local install.

Thanks for any insight!

[cboettig]

@jmzobitz thanks for the report! Looks like your local install of arrow was built without support for S3. Did the instructions linked in the error message above (https://arrow.apache.org/docs/r/articles/install.html) provide any help?

Can you let us know the output of sessionInfo()?

[jmzobitz]

Unfortunately no. I am starting to suspect that this may be an issue with my computer (newer Macbook Air with Apple M2), so I understand that I might need to sort this out on my end. 1. Here is the sessionInfo: R version 4.4.0 (2024-04-24) Platform: aarch64-apple-darwin20 Running under: macOS Ventura 13.5.2 Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0 locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 time zone: America/Chicago tzcode source: internal attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] neon4cast_0.1.0 loaded via a namespace (and not attached): [1] assertthat_0.2.1 utf8_1.2.4 R6_2.5.1 bit_4.0.5 [5] tidyselect_1.2.1 magrittr_2.0.3 glue_1.7.0 tibble_3.2.1 [9] pkgconfig_2.0.3 bit64_4.0.5 dplyr_1.1.4 generics_0.1.3 [13] lifecycle_1.0.4 cli_3.6.2 fansi_1.0.6 arrow_15.0.1 [17] vctrs_0.6.5 compiler_4.4.0 purrr_1.0.2 rstudioapi_0.16.0 [21] tools_4.4.0 pillar_1.9.0 rlang_1.1.3

…

---- 2. Based on the instructions in https://arrow.apache.org/docs/r/articles/install.html I ran this: options( HTTPUserAgent = sprintf( "R/%s R (%s)", getRversion(), paste(getRversion(), R.version["platform"], R.version["arch"], R.version["os"]) ) ) install.packages("arrow", repos = " https://packagemanager.rstudio.com/all/__linux__/focal/latest") But received the following message: *** pkg-config not found. *** OpenSSL not found *** Proceeding without libarrow (build not authorized) ------------------------- NOTE --------------------------- There was an issue building the Arrow C++ libraries. See https://arrow.apache.org/docs/r/articles/install.html --------------------------------------------------------- 3. I realized that I needed to install homebrew and install pkg-config and OpenSSL, which were successful in homebrew. However I've still received the same message when installing again. As I said, this may be an issue on my end of the computer which I might need to figure out. I appreciate any insight. John

On Tue, May 14, 2024 at 11:14 PM Carl Boettiger ***@***.***> wrote: @jmzobitz <https://github.com/jmzobitz> thanks for the report! Looks like your local install of arrow was built without support for S3. Did the instructions linked in the error message above ( https://arrow.apache.org/docs/r/articles/install.html) provide any help? Can you let us know the output of sessionInfo()? — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPZNTRLNEOCT62GUWQ5MHTZCLOLJAVCNFSM6AAAAABHXCZH7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJRGU2DONJRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cboettig]

@jmzobitz interesting! Support for Mac's arm based chips has been tricky historically, but CRAN should have Mac arm binaries available that should have support for S3 built in. However, sometimes things still go pear-shaped, especially so soon after the annual release of the new version of R last month.

Did you install arrow using install.packages() ? Can you try reinstalling with that? Note that if asked if you want to install from source (it will sometimes say something about a newer version being available from source), make sure you choose the answer "no". (This message is meaningless to many users, who figure the safe choice is "yes" -- poor UI design! building from source requires you have all the libraries installed).

Can you also just double-check that in your current setup, S3 is not present by confirming that the following returns FALSE:

arrow::arrow_with_s3()

[jmzobitz]

Thanks! `arrow::arrow_with_s3()` was `FALSE` unfortunately, even after reinstalling from source. Unfortunately that didn't work. I'll keep trying - the maddening part was that last summer at the ESA workshop installation worked beautifully, then my computer was updated by the University. *********** John M. Zobitz, PhD Pronouns: He, him, his Professor of Mathematics and Data Science Data Science Program Director Environmental Studies Program Augsburg University 2211 Riverside Avenue, CB 93 Minneapolis, MN 55454 Hagfors Center 187 P: (612) 330-1068

…

On Wed, May 15, 2024 at 9:46 AM Carl Boettiger ***@***.***> wrote: @jmzobitz <https://github.com/jmzobitz> interesting! Support for Mac's arm based chips has been tricky historically, but CRAN should have Mac arm binaries available that should have support for S3 built in. However, sometimes things still go pear-shaped, especially so soon after the annual release of the new version of R last month. Did you install arrow using install.packages() ? Can you try reinstalling with that? Note that if asked if you want to install from source (it will sometimes say something about a newer version being available from source), make sure you choose the answer "no". (This message is meaningless to many users, who figure the safe choice is "yes" -- poor UI design! building from source requires you have all the libraries installed). Can you also just double-check that in your current setup, S3 is not present by confirming that the following returns FALSE: arrow::arrow_with_s3() — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPZNTTIWAKFKS3COEERTF3ZCNYNBAVCNFSM6AAAAABHXCZH7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJSG42DOMZSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cboettig]

@jmzobitz right, sorry I didn't word that well. By 'answer "no"' to install from source I meant that you do want to install the prebuilt binary directly from CRAN, which should have S3 support built in.

I think the install from source is the reason you don't have s3 support. building arrow from source with support for all the options is rather complex, as detailed in the docs. https://arrow.apache.org/docs/r/articles/install.html#dependencies-for-s3-and-gcs-support. much easier to install the prebuilt binary.

[jmzobitz]

Finally found success! 🎉

Thanks for helping me (1) understand the error message (especially since my error message was outside of the package) and (2) pointing me to helpful resources. For posterity, I am going to record the steps that worked in case others have similar issues (working on a Macbook M2 chip). Happy to make additional edits if these aren't correct or delete if way off topic.

Homerew

• Install homebrew
• In a terminal window, install curl, cmake, and openssl (brew install curl cmake openssl) LINK

Path correction

Homebrew installed some packages not in the main path, so I had to edit my ~/.zshrc file:

• export PATH="/opt/homebrew/bin:$PATH"
• export PATH="/opt/homebrew/opt/curl/bin:$PATH"
• export PATH="/opt/homebrew/opt/openssl/bin:$PATH"
• Determine where openssl is saved: echo $(brew --prefix openssl) yielded /opt/homebrew/opt/openssl@3
• From there, create and edit an .R file so R can locate where openssl and libcurl are located:
• In a terminal: mkdir -p ~/.R (to make the .R directory)
• create the file with touch ~/.R/Makevars
• In the terminal add the correct path with echo LDFLAGS=-L/usr/local/opt/openssl@3/lib > ~/.R/Makevars && echo CPPFLAGS=-I/usr/local/opt/openssl@3/include >> ~/.R/Makevars

Install arrow from source in R:

• Make sure when arrow is installed it builds support for AWS S3: Sys.setenv("ARROW_S3"="ON")
• Determine the location of the source package on CRAN:
• packageurl <- "https://cran.r-project.org/src/contrib/arrow_15.0.1.tar.gz"
• Install the package: install.packages(packageurl, repos=NULL, type="source")

Installation took a while, but was successful when done!

Thanks again,
John