Error on TPU Pod

[rd16395p]

I am trying to connect to my TPU Pod using the documentation found here and the example in the TPU github for MNIST.

https://cloud.google.com/tpu/docs/training-on-tpu-pods

However, when I try to connect, I receive the following error;
[I1107 14:24:47.276069 140521844594496 transport.py:157] Attempting refresh to obtain initial access_token
[W1107 14:24:47.301813 140521844594496 http.py:118] Invalid JSON content from response: b'{\n "error": {\n "code": 403,\n "message": "Request had insufficient authentication scopes.",\n "status": "PERMISSION_DENIED"\n }\n}\n'

Does anyone know how to fix this issue? I have ran gcloud init to ensure I am using my account.

[CyFeng16]

It is that your GCE instance does not have the right to access TPU domain.
Just Check the GCE instance's configuration and give it all necessary access right needed.

[CyFeng16]

@rd16395p You may want to read this.

[rd16395p]

Yes that was the issue, I changed it via editing the VM. Cheers!

[CyFeng16]

@rd16395p Congrats and enjoy the TPUs.

[manhlab]

please tell me how you fixed this. I tried it but not success !

[rd16395p]

please tell me how you fixed this. I tried it but not success !

@manhlab

You need to edit the VM to have permissions to do the task.

See "Changing the service account and access scopes for an instance" here:
https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances

[manhlab]

sorry but I already have this

[manhlab]

I try to create a new key. But I think all think is already but don't know Why not running TPU?

[rd16395p]

@manhlab Did you try fixing the VM?

Do you have a TPU instance running?

[manhlab]

what i did: run VM, run TPU
export TPU_NAME=****

[rd16395p]

@manhlab

1. Create a VM with those settings I link to
2. Create a TPU and get the IP to connect to. You can find this in the details of it.
3. Use the following code ; and follow along to " Get TPU Address";
   https://blog.tensorflow.org/2019/01/keras-on-tpus-in-colab.html

[manhlab]

thanks for your advince. but i am running code on colab. all working with my code. but don't know how to link TPU GC and VM instance. This is first time of me

[manhlab]

i still cant fix this. please help me

[rd16395p]

@manhlab Show me your connection string and I can help you edit it.

[rd16395p]

@manhlab https://github.com/rd16395p/TPU-Pod-Examples/tree/master/iris%20example
This code may help you. I have instructions on how to use it as well.