[R] S3 credentials not being picked up on Centos7

[asfimport]

We use EC2 machines to read parquet datasets from S3. One user has a problem with getting this to work. The initial issue was trying to read parquet files from S3 failed, with what looked like an AWS credentials error:

Error: IOError: When getting information for key 'X' in bucket 'Y': AWS Error [code 15]: No response body.

The AWS CLI doesn't have a permissions problem, nor does the pyarrow package in Python, and neither do other packages in R (eg paws). I assumed it was to do with not having a full install of the Arrow package with S3 support. Trying a few different versions of reinstalling from source all failed at a stage building Thrift, which in turn seemed to be related to not finding Boost libraries – see the attached error log.

Usual approach we use, which worked on other machines

Sys.setenv("LIBARROW_MINIMAL" = FALSE)
install.packages("arrow")

Trying to use an Arrow binary, I think probably not the right way, and it's just falling back to building from source.

Sys.setenv("LIBARROW_BINARY" = TRUE) # I'm not sure this is actually the correct use
Sys.setenv("LIBARROW_MINIMAL" = FALSE)
install.packages("arrow")

Following some advice in another issue I now can't find, I tried specifying the compilers

Sys.setenv(CC="/usr/bin/gcc")
Sys.setenv(CXX="/usr/bin/g++")
Sys.setenv(LIBARROW_MINIMAL="FALSE")
Sys.setenv(LIBARROW_BINARY="FALSE")
Sys.setenv(ARROW_R_DEV="TRUE")
install.packages("arrow")

We've also tried getting the missing dependencies from yum:

yum install -y libcurl-devel
yum install -y openssl-devel
yum install -y thrift
yum install -y boost boost-thread boost-devel

But are still failing at the same point.

If we install from the RSPM repo for Centos7 (We're running Amazon Linux 2), the package installs seemingly happily but we still have the AWS Error message when trying to read from an S3 bucket.

Environment: Amazon Linux 2, R 4.1, Arrow 9.0.0
Reporter: Adam Giles

Original Issue Attachments:

• error.log

_{Note: This issue was originally created as ARROW-17652. Please see the migration documentation for further details.}

[asfimport]

Dewey Dunnington / @paleolimbot:
ARROW-17967 Also reported problems with Thrift. Would it be possible to check to see if this issue still exists with Arrow 10.0.0?

[asfimport]

Adam Giles:
I tried this again earlier, something is definitely different but it's
still not working for us. Following the guidance to set up for C++ 17 here:
https://arrow.apache.org/docs/r/articles/install.html#compilers and then
installing from source with LIBARROW_MINIMAL = FALSE, now does successfully

It's still not picking up the credentials correctly, similar to when we
install the pre-compiled binary for Centos7 from RSPM. Gives:

Error: IOError: When getting information for key 'XXX' in bucket 'YYY': AWS
Error ACCESS_DENIED during HeadObject operation: No response body.

Upgrading with the build from source on one of the machines where it was
already working, it continues to work.

On Mon, 12 Dec 2022 at 14:50, Dewey Dunnington (Jira) jira@apache.org
wrote:

[asfimport]

Dewey Dunnington / @paleolimbot:
I'm glad to hear the build is working better this time around. I will retitle this ticket to draw attention to the S3 credentials...unfortunately I don't have any experience with that part of the Arrow bindings!