Connecting to Athena throws error: Could not connect to the endpoint URL: "https://sts.c.amazonaws.com/"

[NanisTe]

Issue Description

dbConnect() returns Error: Could not connect to the endpoint URL: "https://sts.c.amazonaws.com/"

nslookup and ping on this URL do not work as well.

R runs on an instance in AWS. Everything worked two to five days ago.

nslookup and ping on https://sts.amazonaws.com work though.

Meaning my instance is connected.

Reproducible Example

Sorry not possible to support you with an reproducible example

Example:

library(RAthena)
library(DBI)
con <- RAthena::dbConnect(RAthena::athena(),s3_staging_dir = 's3://bucketnamepath',role_arn = 'role:arn')

Error: Could not connect to the endpoint URL: "https://sts.c.amazonaws.com/"

Session Info

devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────
 setting  value                       
 version  R version 3.6.3 (2020-02-29)
 os       Ubuntu 19.10                
 system   x86_64, linux-gnu           
 ui       RStudio                     
 language (EN)                        
 collate  C.UTF-8                     
 ctype    C.UTF-8                     
 tz       Etc/UTC                     
 date     2020-04-29                  

─ Packages ──────────────────────────────────────────────────────
 ! package     * version date       lib source        
 P assertthat    0.2.1   2019-03-21 [?] CRAN (R 3.6.3)
 P backports     1.1.6   2020-04-05 [?] CRAN (R 3.6.3)
 P callr         3.4.3   2020-03-28 [?] CRAN (R 3.6.3)
 P cli           2.0.2   2020-02-28 [?] CRAN (R 3.6.3)
 P crayon        1.3.4   2017-09-16 [?] CRAN (R 3.6.3)
 P data.table    1.12.8  2019-12-09 [?] CRAN (R 3.6.3)
 P DBI         * 1.1.0   2019-12-15 [?] CRAN (R 3.6.3)
 P desc          1.2.0   2018-05-01 [?] CRAN (R 3.6.3)
 P devtools    * 2.3.0   2020-04-10 [?] CRAN (R 3.6.3)
 P digest        0.6.25  2020-02-23 [?] CRAN (R 3.6.3)
 P ellipsis      0.3.0   2019-09-20 [?] CRAN (R 3.6.3)
 P fansi         0.4.1   2020-01-08 [?] CRAN (R 3.6.3)
 P fs            1.4.1   2020-04-04 [?] CRAN (R 3.6.3)
 P glue          1.4.0   2020-04-03 [?] CRAN (R 3.6.3)
 P jsonlite      1.6.1   2020-02-02 [?] CRAN (R 3.6.3)
 P lattice       0.20-41 2020-04-02 [?] CRAN (R 3.6.3)
 P magick        2.3     2020-01-24 [?] CRAN (R 3.6.3)
 P magrittr      1.5     2014-11-22 [?] CRAN (R 3.6.3)
 P Matrix        1.2-18  2019-11-27 [?] CRAN (R 3.6.1)
 P memoise       1.1.0   2017-04-21 [?] CRAN (R 3.6.3)
 P pkgbuild      1.0.7   2020-04-25 [?] CRAN (R 3.6.3)
 P pkgload       1.0.2   2018-10-29 [?] CRAN (R 3.6.3)
 P prettyunits   1.1.1   2020-01-24 [?] CRAN (R 3.6.3)
 P processx      3.4.2   2020-02-09 [?] CRAN (R 3.6.3)
 P ps            1.3.2   2020-02-13 [?] CRAN (R 3.6.3)
 P R6            2.4.1   2019-11-12 [?] CRAN (R 3.6.3)
 P RAthena     * 1.8.0   2020-03-17 [?] CRAN (R 3.6.3)
 P Rcpp          1.0.4.6 2020-04-09 [?] CRAN (R 3.6.3)
 P remotes       2.1.1   2020-02-15 [?] CRAN (R 3.6.3)
   renv          0.9.3   2020-02-10 [1] CRAN (R 3.6.3)
 P reticulate    1.15    2020-04-02 [?] CRAN (R 3.6.3)
 P rlang         0.4.5   2020-03-01 [?] CRAN (R 3.6.3)
 P rprojroot     1.3-2   2018-01-03 [?] CRAN (R 3.6.3)
 P rstudioapi    0.11    2020-02-07 [?] CRAN (R 3.6.3)
 P sessioninfo   1.1.1   2018-11-05 [?] CRAN (R 3.6.3)
 P testthat      2.3.2   2020-03-02 [?] CRAN (R 3.6.3)
 P usethis     * 1.6.1   2020-04-29 [?] CRAN (R 3.6.3)
 P uuid          0.1-4   2020-02-26 [?] CRAN (R 3.6.3)
 P withr         2.2.0   2020-04-20 [?] CRAN (R 3.6.3)

[1] /home/ubuntu/efs/project/usagebox-potential-tdm-buyers/renv/library/R-3.6/x86_64-pc-linux-gnu
[2] /tmp/RtmpmJZSnq/renv-system-library

 P ── Loaded and on-disk path mismatch.

[DyfanJones]

This is strange as no change has been pushed to the cran yet.

Are you able to make the connection to STS with python boto3?

import boto3

sts = boto3.Session().client("sts")

[NanisTe]

Yes with reticulate I am able to do:

boto <- import("boto3")
test <- boto$client(service_name="sts")

It returns an object. Is that what you asked for?
Maybe there are changes on AWS side, that sts.c.amazonaws.com is not available anymore?

[DyfanJones]

I am not a 100% sure, RAthena calls the above code in a simple wrapper when role_arn is not null. Can you trying out the wrapper function to see if it gives your credentials:

library(RAthena)

assume_role(role_arn = 'role:arn')

[DyfanJones]

Just ran my unit test for role_arn: https://github.com/DyfanJones/RAthena/blob/master/tests/testthat/test-arn.R

Everything seems fine. Has there been any changes to your aws role in the last few days that could explain why you are getting this error?

[NanisTe]

Hi

nothing changed in the role. Just checked that. assume_role(role_arn = 'role:arn') throws the same error massage. Even updated all pip packages in conda for reticulate and all R packages.
It is really strange. Was working fine before the weekend.

[DyfanJones]

That function is a basic wrapper of:

library(reticulate)

boto <- import("boto3")

sts <- boto$Session()$client("sts")
response <- sts$assume_role(RoleArn = "role_arn",
                              RoleSessionName = sprintf("RAthena-session-%s", as.integer(Sys.time())),
                              DurationSeconds = 3600L)

I am guessing sts$assume_role is causing your error. Can you please confirm this for me.

[NanisTe]

I think I solved it. I recently started to use your package and forgot that I read that we should use the regions to make assume role work well. Went to my history of terminal commands and saw the difference. Sorry for steeling your time but maybe it helps others in future.

Solution: use assume_role(role_arn = 'role:arn', region_name = 'eu-central-1') in my case because the service is in eu-central-1 AWS Region.

or

library(RAthena)
library(DBI)
con <- RAthena::dbConnect(RAthena::athena(),s3_staging_dir = 's3://bucketnamepath',role_arn = 'role:arn', region_name = 'eu-central-1')

unfortunately dbConnect does not suggest many parameter options in RStudio as suggestions or in terminal explanations.

Thanks for your help!

[DyfanJones]

Perfect i am glad it is working :) i have some documentation around the new parameters and methods.

i havent added region_name it as a requirement as region can be set in the config file. However i could add a note to make sure users are aware

[DyfanJones]

@NanisTe I will added a catch to ensure users are aware that region_name needs to be set up in the backend or hard coded in dbConnect()

if(is.null(ptr$region_name)) stop("`region_name` is required to be set. Please set region in .config file, environment variables or hard coded in `dbConnect()`.")