Service account authentification doesn't work

[tvroylandt]

Hi,

Thanks for this great package.

I tried to configure gmailr using a service account token like gm_auth(path = "my_token.json") but it send me this error :

Error: Must create an app and register it with `gm_auth_configure()`

I know my token is good because when I use it with googledrive or googlesheets4, it works.

I can create an app using OAuth, but I would like to use service account token here.

Thanks

[yogat3ch]

Hi @jennybc & @jimhester,
I am still unable to get a Service Account working with gmailr so I thought I'd ask for assistance again. I raised the issue on the wrong thread (#144), for which @jennybc responded and answered all of the related issues (Thank you!) though I was not able to figure out how to authorize gmailr with a service account from the advice provided there. I'm trying again and this is what I have thus far:

A Service account is created on a standard/non-workspace Google account and the json is downloaded.

I've attempted to authorize with the following code derived from the docs here:

token <- gargle::token_fetch(
  scopes = c("https://www.googleapis.com/auth/gmail.compose",
             "https://www.googleapis.com/auth/gmail.send"),
  path = "inst/auth/rminor@rminor-333915.iam.gserviceaccount.com.json")

config <- gmailr::gm_auth_configure(app = token)
gmailr::gm_auth(token = config,
                cache = FALSE,
                email = "rminor@rminor-333915.iam.gserviceaccount.com")

I get this error message from gmailr::gm_auth_configure:
Error in if (!have_app) { : missing value where TRUE/FALSE needed

and this from gm_auth:

Error: Must create an app and register it with `gm_auth_configure()`

I created an Oauth consent for testing as I thought that may be an issue:

but that did not resolve it.

I've tried passing the Service Account json directly to path as one would do with googlesheets4:

gmailr::gm_auth_configure(path = "inst/auth/rminor@rminor-333915.iam.gserviceaccount.com")

but this results in the following error:
Error: Can't find client_id and client_secret in the JSON.

If one of y'all has some time could you provide some guidance for myself and others on this thread (or whoever may encounter it) as to how to get a service account and gmailr working together? Or is it not possible because the Service account doesn't have an associated email inbox?

Thanks in advance,
Stephen

R Session Info

``` R version 4.1.0 (2021-05-18) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices datasets utils methods
[7] base

loaded via a namespace (and not attached):
[1] tidyselect_1.1.1 janitor_2.1.0
[3] remotes_2.4.2 purrr_0.3.4
[5] gargle_1.2.0 snakecase_0.11.0
[7] colorspace_2.0-2 archive_1.1.1.9000
[9] vctrs_0.3.8 generics_0.1.1
[11] testthat_3.0.4.9000 usethis_2.1.3
[13] base64enc_0.1-3 utf8_1.2.2
[15] rlang_0.4.12 pkgbuild_1.2.1
[17] pillar_1.6.4 glue_1.6.0
[19] withr_2.4.3 DBI_1.1.1
[21] rdrop2_0.8.2.1 sessioninfo_1.2.2
[23] audio_0.1-10 lifecycle_1.0.1
[25] stringr_1.4.0 munsell_0.5.0
[27] RmData_0.0.0.9003 UU_0.0.0.9001
[29] devtools_2.4.3.9000 memoise_2.0.1
[31] callr_3.7.0 fastmap_1.1.0
[33] ps_1.6.0 clarity.looker_0.0.0.9001
[35] curl_4.3.2 fansi_0.5.0
[37] openssl_1.4.6 renv_0.14.0-143
[39] scales_1.1.1 cachem_1.0.5
[41] desc_1.4.0 pkgload_1.2.4
[43] jsonlite_1.7.2 fs_1.5.2
[45] HMIS_0.0.0.9000 askpass_1.1
[47] stringi_1.7.6 gmailr_1.0.1.9000
[49] processx_3.5.2 dplyr_1.0.7
[51] rprojroot_2.0.2 cli_3.1.0
[53] tools_4.1.0 magrittr_2.0.1
[55] beepr_1.3 tibble_3.1.6
[57] crayon_1.4.2 pkgconfig_2.0.3
[59] ellipsis_0.3.2 prettyunits_1.1.1
[61] lubridate_1.8.0 assertthat_0.2.1
[63] httr_1.4.2 rstudioapi_0.13
[65] R6_2.5.1 compiler_4.1.0

</details>

[yogat3ch]

Hi gmailr devs,
After exploring the gm_auth_configure it looks like it can be circumvented and the authorization credential can be registered directly with the following:
assign("cred",gargle::token_fetch(path = "google_SA.json"), gmailr:::.auth)

This allowed me to start sending requests to the API. I tried to send a test email and received the following error message:

Error in gmailr_POST(c("messages", "send"), user_id, class = "gmail_message",  : 
  Gmail API error: 403
  Gmail API has not been used in project 392021992927 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/gmail.googleapis.com/overview?project=xxxxx then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

I went ahead and enabled the Gmail API and tried again.
The error message changed to:
Error in gmailr_POST(c("messages", "send"), user_id, class = "gmail_message", : Gmail API error: 400

I think we might need to enable gmail domain-wide and delegate permissions to the Service Account for the Gmail API to resolve this. We're going to try this. I am wondering if we're barking up the right tree here though?

[alexquant1993]

Hi @yogat3ch, thank you very much for this post.
I am building a Shiny App and I use the gmailr package to send emails in production. So, in my case, the service account method is the way to go.
As you initially pointed out, the service account credential worked for both googledrive and googlesheets4 using drive_auth() and gs4_auth() functions. However, for the gmailr package is not as straightforward, I've had the same error as you, given that in this case, it is expected to use gm_auth_configure() first and then gm_auth(), stating that the way to go is the OAuth method.
@jennybc & @jimhester this should be corrected, the main issue is with the function gm_oauth_app():

gm_oauth_app <- function () 
{
  if (!is.null(.auth$app)) {
    return(.auth$app)
  }
  stop("Must create an app and register it with `gm_auth_configure()`", 
    call. = FALSE)
}

Please also include in the gm_auth() function, the "subject" parameter as well to allow a service account to impersonate a given Gmail Workspace account.

Here is how I made it work, using @yogat3ch's hack:

    json_path <- gargle:::secret_read("your_package", "credential.json")
    # Get token given the service account credentials
    token <- gargle::credentials_service_account(
      scopes = "https://mail.google.com/", # Full scope
      path = rawToChar(json_path), # Encrypted credential
      subject = get_golem_config("gmail_account") # Allow service account to act on behalf of this subject
    )
    # Assign token to .auth internal object from gmailr package
    assign("cred", token, gmailr:::.auth)

In order to grant domain-wide authority, you need to have an administrator Google Workspace account:

1. Create a project on Google Cloud developers.
2. Enable APIs, for instance: Gmail, Googledrive, Googlesheets
3. Create a service account on Google Cloud developers and get the client ID.
4. Download the JSON credential. Encrypt it to manage your credential securely.
5. Go to the administrator dashboard from your Google Workspace account.
6. Go to Security, API controls, Domain-wide delegation.
7. Add new. Paste service account client ID and list the scopes to be granted. The scopes that are neccesary for the gmailr package are: "https://www.googleapis.com/auth/userinfo.email"(Required by credentials_service_account() function) and "https://mail.google.com/" (If you want full scope, place another one if you require it. See https://developers.google.com/gmail/api/auth/scopes).

[skgithub14]

@alexquant1993 thanks for the excellent post. I had been troubleshooting this issue for a while and your solution worked perfectly.

[jennybc]

Yeah @alexquant1993 I agree with your diagnosis. The early unqualified call to gm_oauth_app() is causing an early error for someone trying to use a service account. Creative solution! But yes we should fix it more properly in gmailr.

I'm curious: are you successfully sending email as someone other than the service account this way? This has been reported as "not working" in gargle and I've never truly dug into it because I am not allowed to grant a service account domain wide delegation at my workplace, which makes it really hard to investigate. I have formed a hypothesis is that the problem is specifying the subject's email address vs. an id. What does get_golem_config("gmail_account") return? Does it look like an email address or a user ID?

[jennybc]

A slightly less "exciting" adaptation of @alexquant1993's workaround is more like (warning: code untested):

# decoy oauth app that we won't actually use
gm_auth_configure(key = "PLACEHOLDER", secret = "PLACEHOLDER")

# Get token given the service account credentials
token <- gargle::credentials_service_account(
  scopes = "https://mail.google.com/",
  path = "path/to/the/json/for/the/service/account.json",
  subject = "??" # specifying the subject
)

gm_auth(token = token)

Update: I tried to test that and all of the above works. But I can't truly test it by, say, sending an email, since I don't have a service account with domain wide authority. And I'm almost certain that a service account must be acting on behalf of a subject with the Gmail API.

[jennybc]

Even simpler workaround (again: not completely tested since I can't grant a service account domain wide authority)

gm_auth_configure(key = "PLACEHOLDER", secret = "PLACEHOLDER")
gm_auth(
  path = "path/to/the/json/for/the/service/account.json",
  subject = "??" # specify the subject
)

Update: in hindsight, this can't work because gm_auth() won't accept subject.

[jennybc]

I'm still trying to figure out exactly what to do here. Jotting down some analysis and notes for myself.

Overall, the magical gargle::token_fetch() is great, because it heroically tries lots of methods of auth. But the flipside of that coin is that it silently swallows all sorts of errors, some of which would be useful for the user to see.

If we could reliably tell what sort of auth the user wants, it would be possible to determine whether an OAuth client is required. But that's pretty opposed to the design of gm_auth() and similar functions in other packages.

As it stands, the unconditional call to app <- gm_oauth_app() makes it extremely clunky (see various creative workarounds above) to auth via any method other than credentials_user_oauth2(). That is almost certainly the most common method of auth for gmailr, but by default all these other methods are tried first and should be possible to use (up to some other, bigger problems described next):

• credentials_byo_oauth2()
• credentials_service_account()
• credentials_external_account()
• credentials_app_default()
• credentials_gce()

I have the impression that it's not possible to send email as a service account. Quoting from (googleapis/google-api-nodejs-client#2322 (comment)):

The Gmail API isn't intended to be used with service accounts (other than domain-wide delegation use cases.) You need to be acting as a real user -- either using oauth credentials obtained with user consent, or in the case of a Gsuite domain, using a service account delegating/impersonating a real user.

So I think the only Gmail-using scenario where a service account is relevant is when one is also passing the sub specifying the regular user on whose behalf the service account is operating. Which gmailr::gm_auth() currently does not support, although I plan to add that.

Therefore, the title of thread is basically true ("Service account authentification doesn't work"), although the root cause isn't really this call to app <- gm_oauth_app() (although that's problematic). The real blocker is the lack of support for passing the subject parameter along.

[jennybc]

With dev gmailr, here is the current best pre-emptive workaround for (theoretically) using one of the auth methods other than credentials_user_oauth2() inside the gargle::token_fetch() call, inside gm_auth():

fake_client <- gargle::gargle_oauth_client(
  id = "PLACEHOLDER",
  secret = "PLACEHOLDER"
)
gm_auth_configure(fake_client)

This is necessary, for example, to use credentials_byo_oauth2(), which is viable today, and is what gmailr uses in its own tests.