Did anything change w/ cloud use case?

[elleobrien]

I'm rerunning the same cloud-deploy workflow as last week, but for some reason getting an error I've never seen before.

Run echo "Deploying..."
Deploying...
Creating CA: /github/home/.docker/machine/certs/ca.pem
Creating client certificate: /github/home/.docker/machine/certs/cert.pem
Running pre-create checks...
Creating machine...
(cml1593467392) Launching instance...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Error creating machine: Error running provisioning: Failed to obtain lock: Maximum number of retries (60) exceeded
The default lines below are for a sh/bash shell, you can specify the shell you're using, with the --shell flag.

Were there any changes to the cloud-runner image in the last four days that might be causing this?

[DavidGOrtega]

@andronovhopf can you share the repo? Nothing has changed so far.

[elleobrien]

Yep it's all in this repo: https://github.com/iterative/cml_cloud_case/actions (pointing you to actions dashboard to show which ones failed)

I'm experimenting with the AWS ami in case that has something tod o with it, not sure if they're prone to updates

[elleobrien]

see here for example: https://github.com/iterative/cml_cloud_case/runs/820181165?check_suite_focus=true

[elleobrien]

Update: I changed the AMI and it works. It's an AWS problem https://github.com/iterative/cml_cloud_case/runs/820305908?check_suite_focus=true

[DavidGOrtega]

AMI are not consistant, thats one of the reasons we have to build ours, also, gpus are not always available. Sometimes i have found 0 availability for gpu in a specific region

[DavidGOrtega]

We are thinking about that

[elleobrien]

very interesting. I'll definitely put a note in the README when we release that GPUs/AMIs are not always consistent. you're right that's a good reason to have a custom image