cloud-init 19.2.36 fails with python exception "Not all expected physical devices present ..." during bionic image deployment from MAAS #3459
Comments
|
Launchpad user Michał Ajduk(majduk) wrote on 2019-10-03T17:55:46.025567+00:00 Issue was introduced in Cloud-init v 19.2-36-g059d049c-0ubuntu1 Symptoms: It seems that following change causes that:
|
|
Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:00:56.483600+00:00 The deployments with bonds seems to be affected. I think since slaves MAC address change after a bond is created cloud-init fails to find the interfaces with original MAC address (they're present in netplan though). |
|
Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:05:17.152526+00:00 The cloud-init commit that introduced the change: b3a87fc It affects branches : 19.2-36 and 19.2-25 |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-03T18:19:59.511335+00:00 Confirmed I can successfully install bionic on baremetal with 19.2-36 without seeing this issue directly in MAAS. I'll try now using Juju to deploy the bare metal machine now to see if I can reproduce the failure. Nikolay do you have access to cloud-init collect-logs (or minimally /var/log/cloud-init.log) on the failed system? |
|
Launchpad user David Coronel(davecore) wrote on 2019-10-03T18:31:10.178956+00:00 subscribed ~field-high Issue seems to be affecting deploys with network bonds. Workaround is to use an older cloud image with an older cloud-init version. |
|
Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:40:15.387335+00:00 cloud-init-output.log attached |
|
Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:41:29.250470+00:00 Note mac addresses of bond slaves |
|
Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T18:42:14.384169+00:00 Note mac addresses again. See also the error in attached cloud-init-output.log. |
|
Launchpad user Jason Hobbs(jason-hobbs) wrote on 2019-10-03T18:44:49.587069+00:00 sub'd to field critical. this is blocking all of our tests. |
|
Launchpad user Dan Watkins(oddbloke) wrote on 2019-10-03T22:14:17.068098+00:00 EOD update from the cloud-init team: We've identified what the problem is: in the problematic code path, we have started filtering out network devices that have a "master", which accidentally also includes physical interfaces that are members of a bridge or bond. As suggested in comment #1, it's the "net: add is_master check for filtering device list" commit that introduced the issue. Unfortunately, that commit was a fix for a critical issue (bug 1844191) from another source, so we cannot simply revert it and take our time to land a different fix. We are currently pursuing three potential options for fixes: (a) completely circumvent the now-incorrect function in the code path that is raising the exception, (b) specifically avoid excluding bridge and bond member interfaces from the list, and (c) match more closely on the type of interface the 'master' check was intended to exclude. In order of preference, we would prefer (c) over (b) over (a). (Because (a) does not obviate the need for something along the lines of (b) or (c) for other code paths, and because we may have to play whack-a-mole with other cases that (b) needs to expand to include.) We have initial implementations of (a) and (b) which we are testing (though they are not yet ready to land in trunk in their current form). We are investigating (c), but it's likely that we won't reach a conclusion on that investigation fast enough to warrant waiting for its completion before addressing this issue. |
|
Launchpad user Nivedita Singhvi(niveditasinghvi) wrote on 2019-10-04T09:16:18.960810+00:00 Bumped up the importance due to more reports in production env |
|
Launchpad user Launchpad Janitor(janitor) wrote on 2019-10-04T13:03:00.541354+00:00 Status changed to 'Confirmed' because the bug affects multiple users. |
|
Launchpad user Server Team CI bot(server-team-bot) wrote on 2019-10-04T15:34:44.351035+00:00 This bug is fixed with commit a7d8d03 to cloud-init on branch master. |
|
Launchpad user Steve Langasek(vorlon) wrote on 2019-10-04T17:36:19.478055+00:00 Hello Nikolay, or anyone else affected, Accepted cloud-init into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/19.2-36-g059d049c-0ubuntu2~19.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. |
|
Launchpad user Steve Langasek(vorlon) wrote on 2019-10-04T17:37:10.059953+00:00 Hello Nikolay, or anyone else affected, Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/19.2-36-g059d049c-0ubuntu2~18.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. |
|
Launchpad user David Coronel(davecore) wrote on 2019-10-04T17:37:38.687955+00:00 As an FYI, my workaround for this is to grab the squasfh file from https://images.maas.io/ephemeral-v3/daily/bionic/amd64/20190930/ and copy it over the file /var/lib/maas/boot-resources/current/ubuntu/amd64/ga-18.04/bionic/daily/squashfs on my MAAS nodes (assuming you deploy Ubuntu 18.04 with the GA kernel). |
|
Launchpad user Steve Langasek(vorlon) wrote on 2019-10-04T17:46:44.834194+00:00 Hello Nikolay, or anyone else affected, Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/19.2-36-g059d049c-0ubuntu2~16.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. |
|
Launchpad user Launchpad Janitor(janitor) wrote on 2019-10-04T18:42:07.865309+00:00 This bug was fixed in the package cloud-init - 19.2-36-g059d049c-0ubuntu2 cloud-init (19.2-36-g059d049c-0ubuntu2) eoan; urgency=medium
-- Daniel Watkins oddbloke@ubuntu.com Fri, 04 Oct 2019 11:42:12 -0400 |
|
Launchpad user Jason Hobbs(jason-hobbs) wrote on 2019-10-04T21:34:45.489950+00:00 I successfully verified the bionic fix on MAAS. Here's what I did:
|
|
Launchpad user Mark Darmadi(darmadoo) wrote on 2019-10-04T22:07:34.763182+00:00 How would I make sure that MAAS pulls the latest package of cloud-init? running 'apt policy cloud-init' shows that I do not have cloud-init installed |
|
Launchpad user David Coronel(davecore) wrote on 2019-10-04T22:25:38.811530+00:00 @darmadoo: The cloud-init package is baked into the cloud images that MAAS uses to deploy instances. You have to wait for the next cloud image that will contain this fixed cloud-init, or you can use the workaround in my comment #16 to use a previous image in the meantime. |
|
Launchpad user Mark Darmadi(darmadoo) wrote on 2019-10-04T22:44:48.523702+00:00 I have also successfully verified the bionic fix on MAAS. Enabling the *-proposed repo via curtin_userdata pulled the latest cloud-init package fixed the issue. |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-04T23:03:56.832673+00:00 Oracle SRU verification logs |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-04T23:04:25.775081+00:00 gce sru verification logs Launchpad attachments: gce-sru-19.2.36.ubuntu2.txt |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-04T23:06:46.741104+00:00 Attach file openstack-sru-19.2.36.ubuntu2.txt. |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-04T23:07:10.642614+00:00 Attach file ec2-sru-19.2.36.ubuntu2.txt. |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-04T23:13:13.477614+00:00 Attach file nocloud-lxd-sru-19.2.36.ubuntu2.txt. |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-04T23:13:53.407205+00:00 Attach file nocloud-kvm-sru-19.2.36.ubuntu2.txt. |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-10-04T23:18:28.061838+00:00 Attach file azure-sru-19.2.36.ubuntu2.txt. |
|
Launchpad user Lee Trager(ltrager) wrote on 2019-10-05T02:02:15.636785+00:00 I manually tested the new cloud-init using the MAAS CI as we don't have automated tests for bonds or bridges. Xenial, Bionic, and Disco can all commissioning and deploy fine using static IPs, bonds, and bridges. |
|
Launchpad user Nobuto Murata(nobuto) wrote on 2019-10-06T15:37:39.278619+00:00 It looks like images.maas.io has an older image with cloud-init 19.2-24-ge7881d5c-0ubuntu1~18.04.1 (the version before the regression was introduced) as 20191004 (the latest as of right now). $ curl -s https://images.maas.io/ephemeral-v3/daily/bionic/amd64/20191003/squashfs.manifest | grep -w cloud-init $ curl -s https://images.maas.io/ephemeral-v3/daily/bionic/amd64/20191004/squashfs.manifest | grep -w cloud-init diff --git a/squashfs.manifest.20191003 b/squashfs.manifest.20191004 |
|
Launchpad user Dan Watkins(oddbloke) wrote on 2019-10-07T13:32:15.991978+00:00 That's correct, that was done to work around the cloud-init issue while we land the fix in the archive. (That specific change was tracked in bug 1846845, but that's in a private project so odds are people won't be able to view it.) |
|
Launchpad user Launchpad Janitor(janitor) wrote on 2019-10-07T20:20:51.792837+00:00 This bug was fixed in the package cloud-init - 19.2-36-g059d049c-0ubuntu2~19.04.1 cloud-init (19.2-36-g059d049c-0ubuntu2~19.04.1) disco; urgency=medium
-- Daniel Watkins oddbloke@ubuntu.com Fri, 04 Oct 2019 11:46:15 -0400 |
|
Launchpad user Brian Murray(brian-murray) wrote on 2019-10-07T20:20:57.819914+00:00 The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions. |
|
Launchpad user Launchpad Janitor(janitor) wrote on 2019-10-07T20:22:19.560873+00:00 This bug was fixed in the package cloud-init - 19.2-36-g059d049c-0ubuntu2~18.04.1 cloud-init (19.2-36-g059d049c-0ubuntu2~18.04.1) bionic; urgency=medium
-- Daniel Watkins oddbloke@ubuntu.com Fri, 04 Oct 2019 11:35:54 -0400 |
|
Launchpad user Launchpad Janitor(janitor) wrote on 2019-10-07T20:22:35.519514+00:00 This bug was fixed in the package cloud-init - 19.2-36-g059d049c-0ubuntu2~16.04.1 cloud-init (19.2-36-g059d049c-0ubuntu2~16.04.1) xenial; urgency=medium
-- Daniel Watkins oddbloke@ubuntu.com Fri, 04 Oct 2019 12:01:19 -0400 |
|
Launchpad user Richard Maynard(richard-maynard) wrote on 2019-10-09T05:53:09.649636+00:00 Using 19.2-36-g059d049c-0ubuntu1~18.04.1 any of our new AWS instances can not get networking. A standard package update AMI build resulted in moving from 19.2.24 to 19.2.36 and now our instances come up inaccessible. Our cloudconfig does not make any adjustments to the networking. The specific environment where we have the issue is AWS, US-West-2, m5.Xlarge instances, on a private subnet, within a VPC. Please let me know what information I can provide that may help to troubleshoot. I'm unable to access any instances running the new cloud init so information I can retrieve from them is limited.[ 30.990869] cloud-init[729]: Cloud-init v. 19.2-36-g059d049c-0ubuntu1~18.04.1 running 'init' at Wed, 09 Oct 2019 05:09:30 +0000. Up 30.80 seconds. |
|
Launchpad user Alexander Weber(deshke) wrote on 2019-10-09T12:15:11.722809+00:00 the issue Richard describes also applies to GCP and Azure instances. The netplan configuration is not re-generated if the mac address changes and is hardcoded with a the only work around i've found is |
|
Launchpad user Christian Ehrhardt (paelzer) wrote on 2019-10-09T12:36:31.904833+00:00 Hi Richard and Alexander, Until then if you both could try to get to a failing instance (or its disk) somehow and get the log 1 which usually is at /var/log/cloud-init-output.log that would be great. |
|
Launchpad user Dan Watkins(oddbloke) wrote on 2019-10-09T12:58:27.189811+00:00 Hi Richard, Alexander, Richard: We've published new daily images to AWS containing the newer version of cloud-init. Can you try to reproduce using the most recent daily image (which is ami-0802dbc378772aca8 in us-west-2), please? If you can still reproduce, then please file a new bug (because we'll need to triage and track the fix separately). Alexander: That sounds like a distinct issue, as that has been the behaviour of cloud-init for quite some time now. Could you file a separate bug so we can follow up on it? Thanks! Dan |
|
Launchpad user Richard Maynard(richard-maynard) wrote on 2019-10-09T15:12:15.597350+00:00 Thanks! I'll work to capture logs and reproduce. We worked around for now by using apt to hold the package back (we use an older base image from july that then does an apt get update all while building our image). |
|
Launchpad user Richard Maynard(richard-maynard) wrote on 2019-10-09T17:18:27.212778+00:00 I was able to capture the logs by making a snapshot of the volume and mounting it to a different host, however with the previous version of cloud-init it had the same problem, sorry for the false note but it looks like this issue is not related to the upgrade. |
|
Launchpad user Dan Watkins(oddbloke) wrote on 2019-10-09T17:52:29+00:00 On Wed, Oct 09, 2019 at 05:18:27PM -0000, Richard Maynard wrote:
That's a relief! Thanks for digging into this to get confirmation. If |
|
Launchpad user Alexander Weber(deshke) wrote on 2019-10-10T09:41:12.492476+00:00 Hey Dan, see https://bugs.launchpad.net/cloud-init/+bug/1847583 |
|
Launchpad user Joshua Powers(powersj) wrote on 2019-10-29T19:27:27.400444+00:00 This is fixed in Ubuntu and as such I am unsubscribing field-critical |
|
Launchpad user Chad Smith(chad.smith) wrote on 2019-12-19T23:00:10.153476+00:00 This bug is believed to be fixed in cloud-init in version 19.2-55. If this is still a problem for you, please make a comment and set the state back to New Thank you. |
This bug was originally filed in Launchpad as LP: #1846535
Launchpad details
Launchpad user Nikolay Vinogradov(nikolay.vinogradov) wrote on 2019-10-03T17:19:42.795903+00:00
[Impact]
Any instances launched with bridges or bonds in their network configuration will fail to bring up networking.
[Test Case]
Juju bootstrap on maas of a machine sets up a network bridge that triggers a failure in cloud-init init stage.
This results in a maas machine deployment failure and the machine gets released
Procedure:
Alternate steps on a maas machine with a bridge already created
A1. confirm a bridge interface is configured for the target machine on interface eno1, name it broam, attach it to a subnet and select auto-assign for IP
A2. click deploy -> bionic
A3. Once manual deployment fails go to step 2 below
Alternative 2 juju bootstrap failure on maas
B1: juju bootrap mymaas --no-gui
B2: Once bootstrap fails go to step 2
Expect failure message
Click Exit Rescue Mode on the node in MAAS UI.
ssh to the maas server add the following lines to /etc/maas/preseeds/curtin_userdata to test official *-proposed packages:
system_upgrade: {enabled: True}
apt:
sources:
proposed.list:
source: deb $MIRROR $RELEASE-proposed main universe # upstream -proposed
expect to see MAASDatasource from bootstrapped machine and no errors
Additional verification checks to avoid regression
- DONE oracle
- DONE ec2
- DONE openstack
- DONE gce
- DONE azure
[Regression Potential]
The change being SRU'd adds more conditions to an existing conditional. There is potential to regress the cases that the existing conditional was introduced to cover, so we will be testing those specifically. Other than that, there was some minor refactoring of the existing conditional statement (which did not change the logic it checks), which could cause issues for Oracle netfailover interfaces. We will also specifically test on Oracle.
[Original Report]
Symptoms
After deployment of Ubuntu Bionic image on MAAS provider (deploying to a bare metal server) juju cannot access any deployed machine due to missing SSH keys and machines are stuck in pending state:
$ juju ssh 0
ERROR retrieving SSH host keys for "0": keys not found
$ juju machines
Machine State DNS Inst id Series AZ Message
0 pending 172.20.10.125 block-3 bionic AZ3 Deployed
1 pending 172.20.10.124 block-2 bionic AZ2 Deployed
2 pending 172.20.10.126 block-1 bionic AZ1 Deployed
3 pending 172.20.10.127 object-2 bionic AZ1 Deployed
4 pending 172.20.10.128 object-1 bionic AZ2 Deployed
5 pending 172.20.10.129 object-3 bionic AZ3 Deployed
It worth mentioning that pods can be successfully deployed with MAAS, only bare metal deployment fails.
We checked different bionic images: cloud-init 19.2.24 works, and cloud-init 19.2.36 doesn't.
The text was updated successfully, but these errors were encountered: