Authentication fails with a service account token

[pofl]

This still worked yesterday. It might be that the error is new in 0.5.0
I have put a service account token in the location at $CLOUDSDK_CONFIG
I run this code in a container

> options(gargle_quiet = FALSE)
> library(bigrquery)
> library(httr)
> set_config(verbose(data_out = TRUE, data_in = TRUE, info = TRUE, ssl = FALSE))
> bq_auth()
trying token_fetch()
trying credentials_service_account()
Error: Argument 'txt' must be a JSON string, URL or file.
trying credentials_app_default()
file exists at ADC path: /root/gcp-gargle/application_default_credentials.json
ADC cred type: service_account
trying credentials_service_account()
adding 'userinfo.email' scope
*  Expire in 0 ms for 6 (transfer 0x55e812033c50)
*  Expire in 10000 ms for 2 (transfer 0x55e812033c50)
*  Connection 0 seems to be dead!
*  Closing connection 0
*  17 bytes stray data read before trying h2 connection
*  Found bundle for host oauth2.googleapis.com: 0x55e81230d350 [can multiplex]
*  Re-using existing connection! (#2) with host oauth2.googleapis.com
*  Connected to oauth2.googleapis.com (172.217.23.106) port 443 (#2)
*  Using Stream ID: 3 (easy handle 0x55e812033c50)
-> POST /token HTTP/2
-> Host: oauth2.googleapis.com
-> User-Agent: libcurl/7.64.0 r-curl/4.3 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: application/x-www-form-urlencoded
-> Content-Length: 839
->
>> grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=...

*  We are completely uploaded and fine
<- HTTP/2 400
<- content-type: application/json; charset=UTF-8
<- vary: Origin
<- vary: X-Origin
<- vary: Referer
<- content-encoding: gzip
<- date: Thu, 07 May 2020 10:43:28 GMT
<- server: scaffolding on HTTPServer2
<- cache-control: private
<- content-length: 162
<- x-xss-protection: 0
<- x-frame-options: SAMEORIGIN
<- x-content-type-options: nosniff
<- alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<-
<<  ��

*  Connection #2 to host oauth2.googleapis.com left intact
Error: Bad Request (HTTP 400).
trying credentials_gce()
*  Expire in 0 ms for 6 (transfer 0x55e813724890)
*  Expire in 800 ms for 8 (transfer 0x55e813724890)
*  Expire in 10000 ms for 2 (transfer 0x55e813724890)
*  Expire in 1 ms for 1 (transfer 0x55e813724890)
[... about 20 more lines of "Expires in" ...]
*  Could not resolve host: metadata.google.internal
*  Expire in 2 ms for 1 (transfer 0x55e813724890)
*  Closing connection 4
trying credentials_byo_oauth()
Error: inherits(token, "Token2.0") is not TRUE
trying credentials_user_oauth2()
Gargle2.0 initialize
attempt from: bigrquery
adding 'userinfo.email' scope
loading token from the cache
no matching token in the cache
initiating new token
Waiting for authentication in browser...
Press Esc/Ctrl + C to abort

[jennybc]

This does not immediately look like a gargle problem to me. It seems to be finding the token, recognizing it as a service account token, and sending that all off to httr as it should.

FWIW I'm using several service account tokens locally and in CI and all is well.

How persistent and reproducible is this? Do you have other service account tokens and are they working? Do you use service tokens in other context (so not this particular container)?

[pofl]

Hi Jenny, sorry for the somewhat false alarm. I finally figured out what the problem is and it's been a hell of a ride. So I was executing the code inside a docker container on Windows and I am using the WSL2 docker backend. The WSL2 has this stupid behavior where its system clock can fall out of sync microsoft/WSL#4677.

It looks like when this clock offset gets too big, authentication stops working. I assume it has to do with the expiry time that you submit with the request for a token or something. I don't know. Anyway restarting WSL2 fixes the problem for me. What a nasty nasty bug!

[jennybc]

Ah, we have seen that before then: #111

[jennybc]

Would you @pofl or @Rensa be interested in making a PR to some piece of gargle documentation where we record this gotcha? Or do you feel it is / wiill be fixed in WSL soon?

[jimjam-slam]

Docker has deployed a few fixes for this one (docker/for-win#4526), but unfortunately it keeps sticking 😩 It really is a difficult one! Looks like it affects both Hyper-V backend and WSL2 (probably because the WSL2 backend still uses parts of Hyper-V).

I'd normally be happy to put in the PR, but I probably can't promise anything until I get my own package back on its legs 😅

[jimjam-slam]

@pofl Other current workarounds include:

• Restarting Docker, or
• Opening up Hyper-V Manager, going to settings, and toggling the Time Synchronisation Service on and off (works for me with Hyper-V but haven't tested with WSL2!)

[jimjam-slam]

I did eventually get to submitting a PR (#154) to document this! Sorry it took so long! 😅

[jimjam-slam]

No need to reopen this, just wanted to note that WSL2 is exhibiting this behaviour again, so you might see more reports of this from Windows users!