You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I0708 16:33:54.308651 139689314522944 core.py:324] Submitting request!
W0708 16:33:54.516765 139689314522944 http.py:171] Sleeping 1.11 seconds before retry 1 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:33:55.784710 139689314522944 http.py:171] Sleeping 2.46 seconds before retry 2 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:33:58.420140 139689314522944 http.py:171] Sleeping 5.17 seconds before retry 3 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:03.754786 139689314522944 http.py:171] Sleeping 1.37 seconds before retry 4 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:05.324873 139689314522944 http.py:171] Sleeping 5.63 seconds before retry 5 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:11.127667 139689314522944 http.py:171] Sleeping 13.70 seconds before retry 6 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:34:25.041211 139689314522944 http.py:171] Sleeping 124.27 seconds before retry 7 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:36:29.560709 139689314522944 http.py:171] Sleeping 79.67 seconds before retry 8 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:37:49.466338 139689314522944 http.py:171] Sleeping 351.15 seconds before retry 9 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
W0708 16:43:40.860112 139689314522944 http.py:171] Sleeping 656.62 seconds before retry 10 of 10 for request: POST https://ml.googleapis.com/v1/projects/taglia/jobs?alt=json, after 429
Submitting caliban_franciswi_google_com_1: 0%| | 0/1 [09:46<?, ?requests/s]
I'm not sure what to do to resolve this.
Thanks!
The text was updated successfully, but these errors were encountered:
Hey @fwilliams ! I suspect that this is a case of not having a service account key set up yet. I'll see what I can do replicate on my end and see if we can get a better error, vs passing the timeout 429 back directly.
We tracked this down internally at Google, and it looks like was a case of V100 GPUs being unavailable on this particular project. The 429 response is overloaded to mean "no quota" AND to mean that our submissions are being rate limited. I've submitted a bug internally to see if we can get these cases distinguished.
I'm running caliban to deploy a model to Google Cloud as follows:
I am getting stuck with the following error:
I'm not sure what to do to resolve this.
Thanks!
The text was updated successfully, but these errors were encountered: