Support for paws.common credential mechanisms #122

davidski · 2020-11-24T23:45:33Z

Thanks for the very nifty package!

paws.common recently added support for credential_process in .aws/config files, allowing role assumption, nested profiles, and external credentials all to work seamlessly together. It looks like noctua constructs the config object itself rather than relying on paws, though I'm not clear on the full object model of paws. Is it practical for noctua to hand off all authentication work to paws, allowing a user to take advantage of this greatly improved credential process?

The text was updated successfully, but these errors were encountered:

DyfanJones · 2020-11-25T12:40:39Z

Hi @davidski,

The current implementation of noctua should be able to take full advantage of the recent updates that paws.common has to offer.

When building the config list for paws, noctua does the following rough steps:

Check for Hardcoded credentials
Check for environment variables
if all fails leaves the rest to paws.common

The function nocuta:::cred_set simply builds the config list and passes it to paws.common, if that list is empty (user using .aws/config files) then paws.common should be able to do the rest :)

noctua:::cred_set(aws_access_key_id = NULL,
                  aws_secret_access_key = NULL,
                  aws_session_token = NULL,
                  profile_name = NULL,
                  region_name = NULL)
# returns
list()

nocuta:::cred_set also lets you pass in profile_name (which are set in .aws/config files). For example:

noctua:::cred_set(aws_access_key_id = NULL,
                  aws_secret_access_key = NULL,
                  aws_session_token = NULL,
                  profile_name = "made-up",
                  region_name = NULL)

# returns
$credentials
$credentials$profile
[1] "made-up"

Changing the .aws/config example:

# .aws/config
[profile made_up]
region = eu-west-1
output = json

# R: noctua
library(DBI)
con <- dbConnect(noctua::athena(), profile_name = "made_up")
dbGetInfo(con)

$profile_name
[1] "made_up"

$s3_staging
[1] ########

$dbms.name
[1] "default"

$work_group
[1] "primary"

$poll_interval
NULL

$encryption_option
NULL

$kms_key
NULL

$expiration
NULL

$keyboard_interrupt
[1] TRUE

$region_name
[1] "eu-west-1"

$paws
[1] "0.1.9"

$noctua
[1] "1.9.1"

# .aws/config change:
[profile made_up]
region = eu-west-2
output = json

# R: noctua
library(DBI)
con <- dbConnect(noctua::athena(), profile_name = "made_up")
dbGetInfo(con)

$profile_name
[1] "made_up"

$s3_staging
[1] ######

$dbms.name
[1] "default"

$work_group
[1] "primary"

$poll_interval
NULL

$encryption_option
NULL

$kms_key
NULL

$expiration
NULL

$keyboard_interrupt
[1] TRUE

$region_name
[1] "eu-west-2"

$paws
[1] "0.1.9"

$noctua
[1] "1.9.1"

Please check #90 for any extra information.

Please let me know if I haven't been able to answer your question 😄

davidski · 2020-11-25T15:12:54Z

Thanks for the quick response! This is a complicated workflow to explain and I appreciate the patience.

I do get different behavior with noctua (CRAN v1.9.1 - MacOS) as compared with paws. Here's a redacted configuration to demonstrate what works with the boto-powered CLI and, most recently, with paws.common but not with noctua.

~/.aws/config

[profile base]
region = us-east-2
credential_process = aws-vault exec --json base --prompt=osascript

[profile child-profile]
source_profile = base
include_profile = base
role_arn = arn:aws:iam::XXXX/XXXX

The following works under paws. By working, I mean there is a prompt (via aws-vault) for a MFA token if required, and the call "just works" 😄 This was implemented in paws via paws-r/paws#328

athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "us-east-2"))
athena$list_work_groups()

The following fails under noctua with Error in f() : No credentials provided

library(DBI)
con <- dbConnect(noctua::athena(),
                 profile_name = "child-profile",
                 region = "us-east-2",
                 s3_staging_dir = 's3://XXX')

DyfanJones · 2020-11-25T16:31:35Z

Hi @davidski,

From my testing I get the following error:
Error in f() : No credentials provided

From paws and noctua

> library(DBI)
> 
> con <- dbConnect(noctua::athena(), profile_name ="child-profile", region = "eu-west-1")
Error in f() : No credentials provided
> 
> athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "eu-west-1"))
> athena$list_work_groups()
Error in f() : No credentials provided

Ignore above i had the wrong profile linked

DyfanJones · 2020-11-25T16:36:54Z

Update:

This is very confusing. When I updated my .aws/config to point to the correct profile noctua and paws worked perfectly.

# .aws/config
[default]
region = eu-west-1
output = json


[profile child-profile]
source_profile = default
include_profile = default
role_arn = arn:aws:iam::XXX/XXX

> library(DBI)
> 
> con <- dbConnect(noctua::athena(), profile_name ="child-profile", region = "eu-west-1")
> 
> athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "eu-west-1"))
> athena$list_work_groups()
$WorkGroups
$WorkGroups[[1]]
$WorkGroups[[1]]$Name
[1] "primary"

$WorkGroups[[1]]$State
[1] "ENABLED"

$WorkGroups[[1]]$Description
[1] ""

$WorkGroups[[1]]$CreationTime
[1] "2019-08-22 15:14:47 GMT"



$NextToken
character(0)

> dbGetQuery(con, "select * from iris")
Info: (Data scanned: 3.63 KB)
     sepal_length sepal_width petal_length petal_width   species
  1:          5.1         3.5          1.4         0.2    setosa
  2:          4.9         3.0          1.4         0.2    setosa
  3:          4.7         3.2          1.3         0.2    setosa
  4:          4.6         3.1          1.5         0.2    setosa
  5:          5.0         3.6          1.4         0.2    setosa
 ---                                                            
146:          6.7         3.0          5.2         2.3 virginica
147:          6.3         2.5          5.0         1.9 virginica
148:          6.5         3.0          5.2         2.0 virginica
149:          6.2         3.4          5.4         2.3 virginica
150:          5.9         3.0          5.1         1.8 virginica

I will try to see if it works when I point it to a staging profile

DyfanJones · 2020-11-25T16:42:53Z

When I created a staging profile (demo_profile) noctua and paws seem to be working fine:

# .aws/config
[default]
region = eu-west-1
output = json


[profile demo_profile]
region = eu-west-1
output = json


[profile child-profile]
source_profile = demo_profile
include_profile = demo_profile
role_arn = arn:aws:iam::XXX/XXX

> library(DBI)
> 
> con <- dbConnect(noctua::athena(), profile_name ="child-profile", region = "eu-west-1")
> athena <- paws::athena(config = list(credentials=list(profile="child-profile"), region = "eu-west-1"))
> athena$list_work_groups()
$WorkGroups
$WorkGroups[[1]]
$WorkGroups[[1]]$Name
[1] "primary"

$WorkGroups[[1]]$State
[1] "ENABLED"

$WorkGroups[[1]]$Description
[1] ""

$WorkGroups[[1]]$CreationTime
[1] "2019-08-22 15:14:47 GMT"



$NextToken
character(0)

> dbGetQuery(con, "select * from iris")
Info: (Data scanned: 3.63 KB)
     sepal_length sepal_width petal_length petal_width   species
  1:          5.1         3.5          1.4         0.2    setosa
  2:          4.9         3.0          1.4         0.2    setosa
  3:          4.7         3.2          1.3         0.2    setosa
  4:          4.6         3.1          1.5         0.2    setosa
  5:          5.0         3.6          1.4         0.2    setosa
 ---                                                            
146:          6.7         3.0          5.2         2.3 virginica
147:          6.3         2.5          5.0         1.9 virginica
148:          6.5         3.0          5.2         2.0 virginica
149:          6.2         3.4          5.4         2.3 virginica
150:          5.9         3.0          5.1         1.8 virginica

davidski · 2020-11-25T16:44:33Z

Do you perhaps have an ~/.aws/credentials file setup, with creds for the demo_profle profile? In my configuration, I do not use a credentials file at all. https://github.com/99designs/aws-vault stores credentials in my keychain and call STS for ephemeral credentials, which is what paws.common now supports.

EDIT: corrected the profile name to match yours

DyfanJones · 2020-11-25T17:38:13Z

Thanks @davidski for bring aws-vault to my attention it looks pretty awesome. From my testing, noctua should be able to support paws using aws-vault. Just created a video hope it helps to demonstrate noctua using paws new features :)

# .aws/config file
[default]
region = eu-west-1
output = json

[profile base]
region = eu-west-1
credential_process = aws-vault exec --json base --prompt=osascript

[profile child-profile]
source_profile = base
include_profile = base
role_arn = arn:aws:iam::XXX/XXX

# aws-vault listed sessions
> aws-vault list
Profile                  Credentials              Sessions                    
=======                  ===========              ========                    
default                  -                        -                           
base                     base                     sts.GetSessionToken:51m33s  
child-profile            -                        -

davidski · 2020-11-25T18:46:36Z

Awesome that it's working for you! Now I have to figure out what's different in my environment. ☹️ I'll see what I can mock up. You're not running any known non-CRAN builds on this stack are you?

davidski · 2020-11-25T18:57:42Z

Oh my giddy.... Somewhere in my testing I fat fingered a profile name in my live configuration. With that fixed, everything works seamlessly. While I'm delighted that works, I'm very sorry to take up so much of your time for user error. I'll blame the US Thanksgiving holiday. 😉 Thanks again for the package and all the support. Really looking forward to putting this through it's paces!

DyfanJones · 2020-11-25T19:00:22Z

@davidski no worries I am glad it's working for you. Plus I am happy you pointed me in the direction of aws-vault. Feel free to raise any future tickets.

davidski closed this as completed Nov 25, 2020

emmansh mentioned this issue Feb 23, 2023

Can I set various parameters in .aws/config file and have DBI::dbConnect() read those directly from that file? #193

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for paws.common credential mechanisms #122

Support for paws.common credential mechanisms #122

davidski commented Nov 24, 2020

DyfanJones commented Nov 25, 2020

davidski commented Nov 25, 2020

DyfanJones commented Nov 25, 2020 •

edited

Loading

DyfanJones commented Nov 25, 2020

DyfanJones commented Nov 25, 2020 •

edited

Loading

davidski commented Nov 25, 2020 •

edited

Loading

DyfanJones commented Nov 25, 2020

davidski commented Nov 25, 2020

davidski commented Nov 25, 2020

DyfanJones commented Nov 25, 2020

Support for paws.common credential mechanisms #122

Support for paws.common credential mechanisms #122

Comments

davidski commented Nov 24, 2020

DyfanJones commented Nov 25, 2020

davidski commented Nov 25, 2020

DyfanJones commented Nov 25, 2020 • edited Loading

DyfanJones commented Nov 25, 2020

DyfanJones commented Nov 25, 2020 • edited Loading

davidski commented Nov 25, 2020 • edited Loading

DyfanJones commented Nov 25, 2020

davidski commented Nov 25, 2020

davidski commented Nov 25, 2020

DyfanJones commented Nov 25, 2020

DyfanJones commented Nov 25, 2020 •

edited

Loading

DyfanJones commented Nov 25, 2020 •

edited

Loading

davidski commented Nov 25, 2020 •

edited

Loading