Ubuntu Pro 18.04 on Azure does not configure azure.archive in sources.list

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1921429

Launchpad details

affected_projects = ['cloud-images']
assignee = None
assignee_name = None
date_closed = None
date_created = 2021-03-25T21:28:07.140585+00:00
date_fix_committed = 2021-03-31T13:52:31.983885+00:00
date_fix_released = 2021-03-31T13:52:31.983885+00:00
id = 1921429
importance = undecided
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1921429
milestone = None
owner = aaron-whitehouse
owner_name = Aaron Whitehouse
private = False
status = incomplete
submitter = aaron-whitehouse
submitter_name = Aaron Whitehouse
tags = []
duplicates = []

Launchpad user Aaron Whitehouse(aaron-whitehouse) wrote on 2021-03-25T21:28:07.140585+00:00

On "Ubuntu Server 18.04 LTS - Gen 1" or "Ubuntu Server 18.04 LTS - Gen 2" on Azure, the sources.list is correctly modified to use the Azure archive mirrors:

Note, this file is written by cloud-init on first boot of an instance

[...]
deb http://azure.archive.ubuntu.com/ubuntu/ bionic main restricted
[...]
deb http://azure.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
[...]

On "Ubuntu Pro 18.04 LTS - Gen 2" (Publisher canonical Offer 0001-com-ubuntu-pro-bionic Plan
pro-18_04-lts-gen2 VM generation V2), there is no Note at the top sources.list saying that it has been written by cloud-init and it has not been correctly updated:

deb http://archive.ubuntu.com/ubuntu/ bionic main restricted
[...]
deb http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
[...]

East US (Zone 1) in both cases.

[ubuntu-server-builder]

Launchpad user Aaron Whitehouse(aaron-whitehouse) wrote on 2021-03-25T21:28:07.140585+00:00

Launchpad attachments: Ubuntu Server 18.04 LTS - Gen 1

[ubuntu-server-builder]

Launchpad user Aaron Whitehouse(aaron-whitehouse) wrote on 2021-03-25T21:28:40.245865+00:00

Launchpad attachments: Ubuntu Pro 18.04 LTS - Gen 2

[ubuntu-server-builder]

Launchpad user Lucas Albuquerque Medeiros de Moura(lamoura) wrote on 2021-03-31T13:52:23.853720+00:00

Hello Aaron and thanks for this bug report. Could you please run on the affected PRO machine:

cloud-init collect-logs

and attach the generated tarball to this bug? I think this is a valid bug report, but additional logs/context will help us working on it. Thanks!

[ubuntu-server-builder]

Launchpad user Aaron Whitehouse(aaron-whitehouse) wrote on 2021-03-31T20:14:36.227553+00:00

It looks like Focal (gen 1) is working correctly:
Launchpad attachments: 20.04 gen 1 Pro sources.list

[ubuntu-server-builder]

Launchpad user Aaron Whitehouse(aaron-whitehouse) wrote on 2021-03-31T20:27:20.088633+00:00

Ubuntu Pro 18.04 gen 1 is also not correct. Output of cloud-init collect-logs attached.
Launchpad attachments: output of cloud-init collect-logs

[ubuntu-server-builder]

Launchpad user Aaron Whitehouse(aaron-whitehouse) wrote on 2021-03-31T20:27:48.292707+00:00

Launchpad attachments: sources.list from 18.04 Pro gen 1

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-03-31T20:45:33.335971+00:00

@aaron, I think this may be a race condition where people are able to ssh into the instance before cloud-init completes. Launching Azure Ubuntu PRO bionic gen1 instances, I was able to easily reproduce this same "invalid" stock apt source by sshing into the vm before cloud-init completed.

On first ssh into the machine I can see the example stock /etc/apt/sources.list without azure-specific mirrors.
I can also block until cloud-init finishes it's setup work with
cloud-init status --wait --long which will print "." until cloud-init is done as well as output any errors.

Once cloud-init finished the content in apt sources.list reflected azure mirrors

On PRO images something to note is that ubuntu-advantage-tools is involved in initial system boot too, and the setup/attach of those services takes a while and adds some cost to initial boot time.

ua auto-attach blocks later stages of cloud-init (like modules:config) from starting.

On Azure Bionic, I can see that this auto-attach setup cost an additional 2 mins

ubuntu@testpro-bgen1:~$ systemd-analyze blame
1min 51.944s ua-auto-attach.service
17.921s cloud-init.service

So checking before that completes would mean that you won't yet see cloud-init apt sources written.

Looking into /var/log/ubuntu-advantage.log on this system we can see the majority of it's setup time is spent waiting on snaps being seeded

2021-03-31 20:27:14,328 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap wait system seed.loaded, rc: 0 stderr: b''
2021-03-31 20:28:37,799 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap install canonical-livepatch, rc: 0 stderr: b''

Note the 1 min 23 second blocking call to /usr/bin/snap wait system seed.loaded in logs.

I'm guessting this is probably why on some PRO image flavors we can see delayed processing/writing of apt config writes.

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2021-04-01T17:05:24.932960+00:00

In further triage with Dan, this boot timing cost is mostly due to the download of the 'core' snap on which canonical-livepatch depends.

this issue has been taken against the ubuntu-advantage-tools project and we are sorting improving that initial boot performance separately from this issue.

canonical/ubuntu-pro-client#1502

[holmanb]

Cloud-init no longer waits on snapd by default.

It doesn't look like there is anything actionable to do on cloud-init's side. Please re-open if I missed something.