Google Cloud Debugger Insufficient Authentication Scopes

[glozow]

A lot of our services experience an authentication error when trying to use Google Cloud Debugger, all along the lines of "insufficient authentication scopes." Usually, they retry every few seconds and we see logs like this:

In nodejs (payment and currency)

[paymentservice-6d984879c6-7kp7j server] @google-cloud/debug-agent Failed to re-register debuggee gz-sandbox: Error: Request had insufficient authentication scopes.
[currencyservice-77dfcc558b-q8mh5 server] @google-cloud/debug-agent Failed to re-register debuggee gz-sandbox: Error: Request had insufficient authentication scopes.

In python (recommendation):

[recommendationservice-6bf8d6f9f5-mf2xq server]   File "/usr/local/lib/python3.7/site-packages/googleclouddebugger/gcp_hub_client.py", line 346, in _RegisterDebuggee
     response = service.debuggees().register(body=request).execute()
   File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
     return wrapped(*args, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 907, in execute
     raise HttpError(resp, content, uri=self.uri)
 googleapiclient.errors.HttpError: <HttpError 403 when requesting https://clouddebugger.googleapis.com/v2/controller/debuggees/register?alt=json returned "Request had insufficient authentication scopes.">

[0xSage]

I wasn't able to repro this issue on current master.

I searched for the above error messages in the container logs and master logs and this issue didn't turn up. It seems correctly configured:

Debugger console seems enabled for the 3 services as well.

[simonz130]

Awesome!

We have a different bug - we should create a source mapping file so debugger for currency and payment service should show the right version of the code like recommendationserver in your picture. Also, we need to figure out how to upload the code so folks don't need to do it. I noticed these issues when I was upgrading runtimes for nodejs, so no need to open a new bug but you should close this bug as no repro - we might have fixed that by upgrading packages in all microservices.

[Shabirmean]

I know this is closed but for what's it's worth - I found this issue with Online Boutique whilst debugging another issue. This happens when the cluster is not created with the access scopes. Specifically the default gke scope does not have access to the Debugger API. Any cluster created with the --scopes=https://www.googleapis.com/auth/cloud_debugger,gke-default will work just fine. I've explained it in: GoogleCloudPlatform/microservices-demo#645