dbtools credentials refresh error

[davidread]

In R Studio, when running dbtools::get_athena_query_response you see this error:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  RuntimeError: Credentials were refreshed, but the refreshed credentials are still expired.

This appears to crop up randomly and stay.

Workaround

• The problem went away for one user after restarting R Studio a couple of times from the AP Control Panel.
• Another says that doesn't work. Instead use the little orange button in the top-right of RStudio to restart RStudio.

Details

The traceback is:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  RuntimeError: Credentials were refreshed, but the refreshed credentials are still expired.
 
18.
    
stop(structure(list(message = "RuntimeError: Credentials were refreshed, but the refreshed credentials are still expired.", 
    call = py_call_impl(callable, dots$args, dots$keywords), 
    cppstack = structure(list(file = "", line = -1L, stack = c("/home/<user>/R/library/reticulate/libs/reticulate.so(Rcpp::exception::exception(char const*, bool)+0x84) [0x7ff5c02c1f54]", 
    "/home/<user>/R/library/reticulate/libs/reticulate.so(Rcpp::stop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x27) [0x7ff5c02c20a7]",  ...
 
17.
    
_protected_refresh at
 credentials.py#456
16.
    
_refresh at
 credentials.py#413
15.
    
get_frozen_credentials at
 credentials.py#518
14.
    
get_auth_instance at
 signers.py#229
13.
    
sign at
 signers.py#149
12.
    
handler at
 signers.py#90
11.
    
_emit at
 hooks.py#211
10.
    
emit at
 hooks.py#228
9.
    
emit at
 hooks.py#356
8.
    
create_request at
 endpoint.py#116
7.
    
_send_request at
 endpoint.py#132
6.
    
make_request at
 endpoint.py#102
5.
    
_make_api_call at
 client.py#611
4.
    
_api_call at
 client.py#320
3.
    
get_athena_query_response at
 <string>#33
2.
    
get_athena_query_response(sql_query = sql_query, out_path = out_path, 
    return_athena_types = return_athena_types, timeout = timeout)
 
1.
    
dbtools::get_athena_query_response(q, out_path = "s3://<bucket>/<path>")

dbtools is using reticulate to run the python script, which uses boto3 to talk to Athena. boto3 is getting the credentials from the AWS instance API, but decides they have already expired, according to the clock and current timezone.

The clock and timezone seem ok:

> library(reticulate)
> py_run_string('import datetime; from dateutil.tz import tzlocal; print(datetime.datetime.now(tzlocal()))')
(...then run dbtools and it shows the result:)
2018-11-21 15:57:05.644476+00:00

which was the correct time.
And the creds work fine with S3 tools:

> s3tools::list_files_in_buckets('alpha-gluejobutils')

and on the command-line:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/alpha_user_<user> -L
{"AccessKeyId":"...","Code":"Success","Expiration":"2018-11-21T14:59:29Z","LastUpdated":"2018-11-21T14:29:29Z","SecretAccessKey":"...","Type":"AWS-HMAC"}

(which I ran at 14:29)
and all is fine with python in the container:

python
import boto3
client = boto3.client('s3')
client.list_buckets()
...(correct list for this user)...

So I'm not sure what the problem is.

[davidread]

Closing for now, as we don't have an environment exhibiting the trouble any more and we have a workaround