GCP Auth issues with ctpu

[ghost]

Hi I am unable to use ctpu through cloud shell, it worked first time then it stopped working.

akshayubhat@cloudshell:~ (dvatfrc)$ ctpu ls
2018/06/22 23:58:36 Error listing Cloud TPUs: googleapi: Error 403: Read access to project 'dvatfrc' was denied, forbidden
akshayubhat@cloudshell:~ (dvatfrc)$ ctpu --zone us-central-f ls
2018/06/22 23:58:50 Error listing Cloud TPUs: googleapi: Error 403: Read access to project 'dvatfrc' was denied, forbidden
akshayubhat@cloudshell:~ (dvatfrc)$ ctpu --zone us-central-f ls
2018/06/22 23:58:55 Error listing Cloud TPUs: googleapi: Error 403: Read access to project 'dvatfrc' was denied, forbidden
akshayubhat@cloudshell:~ (dvatfrc)$

It also wont use gcloud config e.g. I tried changing the region/zone but it does not changes when I view it using ctpu cfg. It's also not clear how to reset ctpu, restarting the VM did not fix this.

[ghost]

For reference it's ctpu version: 1.2 I am using the cloud shell

[ghost]

I deleted the entire home directory hoping that it would solve this bug but no luck.
Upon further investigation I found that if passed explicit project id for another project the ls command worked.

ctpu --zone us-central1-f  --project deepvideoanalytics-161408 ls

However for dvatfrc it failed, I ensured that there was no mistake in specifying project id etc.

[ghost]

Here is the output with -log-http

akshayubhat@cloudshell:~ (dvatfrc)$ ctpu --zone us-central1-f -log-http ls
=========================================
-------- Request ---------
GET /compute/v1/projects/dvatfrc/zones/us-central1-b/instances?alt=json HTTP/1.1
Host: www.googleapis.com
User-Agent: google-api-go-client/0.5 ctpu/1.2 env/devshell
Accept-Encoding: gzip


-------- Response --------
HTTP/2.0 200 OK
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Sat, 23 Jun 2018 07:41:02 GMT
Etag: "X4HBGRm7T7CAAwnHdJT78UfePCw=/tZhRpmpdnTHvqhEj37VGzC3Mrgo="
Expires: Sat, 23 Jun 2018 07:41:02 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

{
 "kind": "compute#instanceList",
 "id": "projects/dvatfrc/zones/us-central1-b/instances",
 "selfLink": "https://www.googleapis.com/compute/v1/projects/dvatfrc/zones/us-central1-b/instances"
}

=========================================
=========================================
-------- Request ---------
GET /v1alpha1/projects/dvatfrc/locations/us-central1-b/nodes?alt=json HTTP/1.1
Host: tpu.googleapis.com
User-Agent: google-api-go-client/0.5 ctpu/1.2 env/devshell
Accept-Encoding: gzip


-------- Response --------
HTTP/2.0 403 Forbidden
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Sat, 23 Jun 2018 07:41:02 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
{
  "error": {
    "code": 403,
    "message": "Read access to project 'dvatfrc' was denied",
    "errors": [
      {
        "message": "Read access to project 'dvatfrc' was denied",
        "domain": "global",
        "reason": "forbidden"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}
=========================================
2018/06/23 00:41:02 Error listing Cloud TPUs: googleapi: Error 403: Read access to project 'dvatfrc' was denied, forbidden

[saeta]

Hi @AKSHAYUBHAT !

Thank you for reporting this bug, and for providing such a complete bug report. The issue is that there seems to have been a regression in ctpu 1.1 and 1.2. I have a fix in review now, and will publish a new version of ctpu hopefully this week, and the cloud shell environment should pick it up by the end of next week.

In the mean time, as a work-around, move the --zone flag from before the command (e.g. ls in this case) to after the command. For example: ctpu -log-http ls --zone us-central1-f should work.

Sorry about the inconvenience!

All the best,
-Brennan

[saeta]

@AKSHAYUBHAT ctpu version 1.3 should now be available from https://dl.google.com/cloud_tpu/ctpu/latest/linux/ctpu (or your platform's equivalent). You can download it and use it locally as follows:

saeta@saeta0:/tmp/foo$ wget https://dl.google.com/cloud_tpu/ctpu/latest/linux/ctpu
--2018-06-27 10:46:21--  https://dl.google.com/cloud_tpu/ctpu/latest/linux/ctpu
Resolving dl.google.com (dl.google.com)... 2607:f8b0:4005:809::200e, 172.217.6.78
Connecting to dl.google.com (dl.google.com)|2607:f8b0:4005:809::200e|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7986558 (7.6M) [application/octet-stream]
Saving to: ‘ctpu’

ctpu                100%[===================>]   7.62M  15.0MB/s    in 0.5s    

2018-06-27 10:46:22 (15.0 MB/s) - ‘ctpu’ saved [7986558/7986558]

saeta@saeta0:/tmp/foo$ chmod a+x ctpu
saeta@saeta0:/tmp/foo$ ./ctpu version
ctpu version: 1.3
saeta@saeta0:/tmp/foo$

The Google Cloud Shell environment should pick up the new version over the next week and a half. Thanks!

[yangw502]

@saeta I updated the latest ctpu (1.3), but I still meet the same problem:

wxyang1503@cloudshell:~ (test)$ ./ctpu version
ctpu version: 1.3
wxyang1503@cloudshell:~ (test)$ ./ctpu up
2018/07/11 10:40:47 googleapi: Error 403: Read access to project 'test' was denied, forbidden

I am still unable to use ctpu through cloud shell.

[ultrons]

I too see the same problem..

[saeta]

@Mayer123 Can you please include the output of a run with the --log-http flag set that reproduces the issue? Thanks! (Note: please do open a new issue with this information.)

[Mayer123]

Thank you for your response, I restarted the cloud shell and the it magically worked. I have deleted my post. Sorry for your trouble.

…

On Wed, Oct 3, 2018 at 3:48 PM Brennan Saeta ***@***.***> wrote: @Mayer123 <https://github.com/Mayer123> Can you please include the output of a run with the --log-http flag set that reproduces the issue? Thanks! (Note: please do open a new issue with this information.) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#142 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQgCgEMQqMikt_zjIZ8XfwPGfrMoYQePks5uhRR_gaJpZM4U0rVD> .

[tpl2go]

is there a way to find out which is the correct zone to use?
I had to try a few zones before the suggested fix worked for me

[frankchn]

@tpl2go -- here are available zones for use with Cloud TPUs: https://cloud.google.com/tpu/docs/regions