DigitalOcean: Creating custom fcos-39.20230916.1.2 image through doctl fails

[c4rt0]

Describe the bug

Issue identified while following the testcase and trying to launch Fedora CoreOS on DigitalOcean using the next image during FCOS test days.

Image creation ends within seconds, size on disk = 0.

Reproduction steps

1. In terminal install doctl CLI:
   sudo dnf install doctl
2. Create an ignition config
3. Create custom 'fcos-next' image with doctl:

doctl compute image create my-fcos-image --region sfo2 --image-url https://builds.coreos.fedoraproject.org/prod/streams/next/builds/39.20230916.1.2/x86_64/fedora-coreos-39.20230916.1.2-digitalocean.x86_64.qcow2.gz
while ! doctl compute image list-user | grep my-fcos-image; do sleep 5; done

Expected behavior

doctl compute image list-user returns a list with a successfully created custom image, which can be run as a droplet via:

image_id=$(doctl compute image list-user | grep my-fcos-image | cut -f1 -d ' ')
key_id=$(doctl compute ssh-key list | grep my-key | cut -f1 -d ' ')
doctl compute droplet create my-fcos-droplet --image "${image_id}" --region sfo2 --size s-2vcpu-2gb --user-data-file ./config.ign --ssh-keys "${key_id}" --wait

Actual behavior

image_id=$(doctl compute image list-user | grep my-fcos-image | cut -f1 -d ' ')
key_id=$(doctl compute ssh-key list | grep my-key | cut -f1 -d ' ')
doctl compute droplet create my-fcos-droplet --image "${image_id}" --region sfo2 --size s-2vcpu-2gb --user-data-file ./config.ign --ssh-keys "${key_id}" --wait

Returns :

Error: POST https://api.digitalocean.com/v2/droplets: 422 (request "f75107b4-d69d-44ab-8c15-17e5ab974b73") The image you selected is no longer available

System details

• DigitalOcean
• fedora-coreos-39.20230916.1.2-digitalocean.x86_64.qcow2.gz

Butane or Ignition config

{
  "ignition": {
    "version": "3.3.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPEo4WPF8SFbA/jv/7QqSK/PJpTVoZKcIWEZa3Eovr4TXXjbwOYQ4iWbJhuXNs5TAPmEw2Qj2KWznKyX+VYUtvQcezOBXgNilrJbg1WKCJx9kSbtpKqepC2pjXGXnYKZGdAl3DcIagO5p0Y1Cuv2s6557X+uVNsu7p7gBMy03h2G/Je+YAbIpuwrJ6+0QSFpR3dbd74dWMFkl3WpN+BCxRxe0kltCibDUbhAQiFUgJXtQsYkPt9xN5sQryJgVKZ2NexcRyRIKY5usxmCShFmgw+uIrTM7PN91xofyqDGDnAbKMWGoCluqegUQfYwqULLuDuJF7IF2SxdSJMYeU6Um8wg77xolZcMy/uJQfjiIA5DLNuOL8N4W+vDrGucyyIEu/8GZLIJKbsYJ/XQ2DBMrUkA30Y+zl8qi4Z60D19s0AOWpFsd+KUE0LHg73+23hRxFuinZx37+2s3hg7Oz9I234rubx2wkqx1I99yim4Dsp0XjLn8yYidpDm8cl1hH7GU="
        ]
      }
    ]
  }
}

Additional information

No response

[dustymabe]

@jmarrero did you see this same behavior?

[jmarrero]

I saw it but in digital ocean UI it showed as "pending", a few minutes later (maybe 10 mins) it showed it changed to being in a region and I could use it to create a VM successfully. Also the image ID changed between pending and actually being available in the region.

[dustymabe]

hmm @c4rt0 maybe try one more time? I wonder if it is specific to the region you were using.

[jmarrero]

I could be that we should change the while ! doctl compute image list-user | grep my-fcos-image; do sleep 5; done for something else. Because that let's you believe that once the image shows up it's ready to use.

[c4rt0]

hmm @c4rt0 maybe try one more time? I wonder if it is specific to the region you were using.

I tried various areas: fra1, sfo2, sfo3 always resulting with the very same results.

[jmarrero]

We can get together Monday and see if we can figure out what is different between our runs.

[c4rt0]

Awesome, I will send you a meeting invite @jmarrero, thanks!

[c4rt0]

I saw it but in digital ocean UI it showed as "pending", a few minutes later (maybe 10 mins) it showed it changed to being in a region and I could use it to create a VM successfully. Also the image ID changed between pending and actually being available in the region.

I can now confirm that droplet creation was eventually successful, after reading the above ("waiting 10 minutes" helped). I thought that the image generation process finished unsuccessfully.

I could be that we should change the while ! doctl compute image list-user | grep my-fcos-image; do sleep 5; done for something else. Because that let's you believe that once the image shows up it's ready to use.

I believe it's necessary.

[jmarrero]

I'll try to figure out what output from the cli we can use this week and do a doc PR.

[jmarrero]

It looks like we can use the json output to get status:

$ doctl compute image list-user --output json

[
  {
    "id": 141683405,
    "name": "my-fcos-image",
    "type": "custom",
    "distribution": "Unknown OS",
    "regions": [
      "sfo2"
    ],
    "min_disk_size": 10,
    "size_gigabytes": 0.84,
    "created_at": "2023-10-04T03:36:07Z",
    "status": "available"
  },
  {
    "id": 141683775,
    "name": "my-fcos-image-1",
    "type": "custom",
    "distribution": "Unknown OS",
    "created_at": "2023-10-04T04:01:47Z",
    "status": "NEW"
  }
]

and then filter by the name and status.

For example this works:

while ! doctl compute image list-user --output json | jq -c '.[] | select(.name=="my-fcos-image")' | grep available; do sleep 5; done

however, can we expect the user to have jq?

[jmarrero]

Looks like jq is all over the docs already:
PR: coreos/fedora-coreos-docs#592

[c4rt0]

Thanks for looking at this @jmarrero!
I can confirm this works indeed.

however, can we expect the user to have jq?

Well, to execute the above my toolbox container just got additional packages/dependencies installed:

{10:47}~/Work/testday/digitalocean ➭ sudo dnf install jq
Last metadata expiration check: 0:04:38 ago on Wed 04 Oct 2023 10:43:01.
Dependencies resolved.
===========================================================================
 Package     Arch     Version                              Repo       Size
===========================================================================
Installing:
 jq          x86_64   1.6-16.fc38                          updates   188 k
Installing dependencies:
 oniguruma   x86_64   6.9.8-2.D20220919gitb041f6d.fc38.1   fedora    218 k

Transaction Summary
===========================================================================
Install  2 Packages

Total download size: 407 k
Installed size: 1.2 M

I wonder if we should inform the user about the possible timeframe here - It took me over 15 minutes to see the status update.

[jmarrero]

Added coreos/fedora-coreos-docs#593