apt sources changes ignored while installing the packages

[Unb0rn]

Bug report

I'm trying to install docker (almost as per official instructions)
But after adding the docker repo, package_update seems to ignore the changes to repos config and installing packages like docker-ce-cli results in:

[ 147.431858] cloud-init[1634]: E: Package 'docker-ce' has no installation candidate
[ 147.432992] cloud-init[1634]: E: Package 'docker-ce-cli' has no installation candidate
[ 147.434112] cloud-init[1634]: E: Unable to locate package containerd.io
[ 147.435036] cloud-init[1634]: E: Couldn't find any package by glob 'containerd.io'
[ 147.435991] cloud-init[1634]: E: Couldn't find any package by regex 'containerd.io'
[ 147.437051] cloud-init[1634]: E: Unable to locate package docker-buildx-plugin
[ 147.437987] cloud-init[1634]: E: Unable to locate package docker-compose-plugin
Logging in to the machine and installing the packages works fine (the repo is added correctly)

I found almost the same issue on severfault that has never been resolved. Is this a bug? Is there any known workaround?

Steps to reproduce the problem

1. Set up the cloud-config so it will add the docker repo to sources list and install packages after
2. Pass it to Ubuntu image
3. Check the logs

Environment details

• Cloud-init version: 23.3.1-0ubuntu2
• Operating System Distribution: Ubuntu 24.04.1 LTS
• Cloud provider, platform or installer type: Yandex Cloud

cloud-init logs

cloud-init.tar.gz

[a-dubs]

Hello! Thank you for filing a bug. Would you be able to provide the cloud-config you used or at least the relevant portion of the cloud config?

[Unb0rn]

Hello! Thank you for filing a bug. Would you be able to provide the cloud-config you used or at least the relevant portion of the cloud config?

Thank you for a quick reply. Attaching the failing part
cloud-init-example.txt

[a-dubs]

Pasting the contents here for easier visibility:

#cloud-config
apt:
  sources:
    docker.list:
      source: deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
      
package_update: true
package_upgrade: true

packages:
 - curl
 - ca-certificates
 - docker-ce
 - docker-ce-cli
 - containerd.io
 - docker-buildx-plugin
 - docker-compose-plugin

[a-dubs]

Hey @Unb0rn,

I ran this locally and I was unable to reproduce the issue locally using your provided cloud-config on both 22.04 and 24.04 LXD containers. Would you mind explaining what you meant by "Logging in to the machine and installing the packages works fine (the repo is added correctly)". My understanding of this is that you launched the instance, waited for cloud-init to finish, then ssh'ed into the instance and ran sudo apt install docker-ce-cli ... and it succeeded without any other changes or actions?

It is very strange to see that your cloud-init version is 23.X when you are on noble (24.04), which should be running cloud-init 24.X. Do you have any insight into why this is the case? If not, no worries.

Due to conflicting dependencies, I am unable to install cloud-init 23.X on an ubuntu 24.04 container so I am unable to test that locally to see if that is the issue. Given that the cloud-init version installed is not compatible and not intended to be used on Noble, this falls outside the bounds of expected use cases for cloud-init. But with that being said, we would still like to know more about Yandex Cloud and do our best to support various cloud platforms where reasonable. So any context you can provided is incredibly valuable.

Also, would you mind providing the result of running cat /etc/cloud/build.info to get more info on the Ubuntu image that you are running on?

Thanks in advance! :D

[blackboxsw]

I also can confirm on latest ubuntu image I see the proper ordering which looks to write /etc/apt sources files, call apt-get update and apt-get dist-upgrade, and finally install the requested packages.

root@secure-ostrich:~# cloud-init --version
/usr/bin/cloud-init 24.2-0ubuntu1~24.04.2
root@secure-ostrich:~# lsb_release -sc
noble
root@secure-ostrich:~# cat /etc/cloud/build.info 
build_name: server
serial: 20240912
root@secure-ostrich:~# egrep 'apt-get|/etc/apt' /var/log/cloud-init.log  
2024-09-23 23:00:29,845 - util.py[DEBUG]: Writing to /etc/apt/sources.list.d/ubuntu.sources - wb: [644] 2988 bytes
2024-09-23 23:00:29,846 - util.py[DEBUG]: Reading from /etc/apt/sources.list (quiet=False)
2024-09-23 23:00:29,846 - util.py[DEBUG]: Read 270 bytes from /etc/apt/sources.list
2024-09-23 23:00:30,429 - util.py[DEBUG]: Writing to /etc/apt/trusted.gpg.d/docker.gpg - wb: [644] 2783 bytes
2024-09-23 23:00:30,430 - util.py[DEBUG]: Writing to /etc/apt/sources.list.d/docker.list - a: [644] 71 characters
2024-09-23 23:00:30,434 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update'] with allowed return codes [0] (shell=False, capture=False)
2024-09-23 23:00:40,807 - subp.py[DEBUG]: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update'] took 10.s to run
2024-09-23 23:00:40,807 - util.py[DEBUG]: apt-update [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet update] took 10.373 seconds
2024-09-23 23:00:41,088 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update'] with allowed return codes [0] (shell=False, capture=False)
2024-09-23 23:00:42,907 - subp.py[DEBUG]: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update'] took 1.8s to run
2024-09-23 23:00:42,907 - util.py[DEBUG]: apt-update [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet update] took 1.820 seconds
2024-09-23 23:00:42,907 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'dist-upgrade'] with allowed return codes [0] (shell=False, capture=False)
2024-09-23 23:00:58,572 - subp.py[DEBUG]: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'dist-upgrade'] took 15.s to run
2024-09-23 23:00:58,572 - util.py[DEBUG]: apt-dist-upgrade [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet dist-upgrade] took 15.665 seconds
2024-09-23 23:00:58,926 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'docker-compose-plugin', 'containerd.io', 'docker-buildx-plugin', 'curl', 'ca-certificates', 'docker-ce-cli', 'docker-ce'] with allowed return codes [0] (shell=False, capture=False)
2024-09-23 23:01:41,684 - subp.py[DEBUG]: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'docker-compose-plugin', 'containerd.io', 'docker-buildx-plugin', 'curl', 'ca-certificates', 'docker-ce-cli', 'docker-ce'] took 42.s to run
2024-09-23 23:01:41,684 - util.py[DEBUG]: apt-install [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install docker-compose-plugin containerd.io docker-buildx-plugin curl ca-certificates docker-ce-cli docker-ce] took 42.758 seconds
root@secure-ostrich:~# cloud-init status --long
status: done
extended_status: done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:01:14 +0000
detail: DataSourceLXD
errors: []
recoverable_errors: {}

[Unb0rn]

Hey @Unb0rn,

I ran this locally and I was unable to reproduce the issue locally using your provided cloud-config on both 22.04 and 24.04 LXD containers. Would you mind explaining what you meant by "Logging in to the machine and installing the packages works fine (the repo is added correctly)". My understanding of this is that you launched the instance, waited for cloud-init to finish, then ssh'ed into the instance and ran sudo apt install docker-ce-cli ... and it succeeded without any other changes or actions?

It is very strange to see that your cloud-init version is 23.X when you are on noble (24.04), which should be running cloud-init 24.X. Do you have any insight into why this is the case? If not, no worries.

Due to conflicting dependencies, I am unable to install cloud-init 23.X on an ubuntu 24.04 container so I am unable to test that locally to see if that is the issue. Given that the cloud-init version installed is not compatible and not intended to be used on Noble, this falls outside the bounds of expected use cases for cloud-init. But with that being said, we would still like to know more about Yandex Cloud and do our best to support various cloud platforms where reasonable. So any context you can provided is incredibly valuable.

Also, would you mind providing the result of running cat /etc/cloud/build.info to get more info on the Ubuntu image that you are running on?

Thanks in advance! :D

You're absolutely right!
While testing I used to wait while cloud-init finishes (with an error), ssh'd to the machine and just ran apt install docker-ce and it worked without re-adding the repo. I was surprised to find the issue on serverfault (the one I attached) with exactly the same version and symptoms.
I will try and get build.info when I can - I believe it's Yandex Cloud issue and the way they force old cloud-init into 24.04