ConfigDrive: cloud-init fails to configure network from network_data.json

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1577982

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)']
assignee = smoser
assignee_name = Scott Moser
date_closed = 2016-08-10T14:42:27.746858+00:00
date_created = 2016-05-03T23:26:49.816769+00:00
date_fix_committed = 2016-06-10T16:45:45.216761+00:00
date_fix_released = 2016-08-10T14:42:27.746858+00:00
id = 1577982
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1577982
milestone = None
owner = mgagne
owner_name = Mathieu Gagné
private = False
status = fix_released
submitter = mgagne
submitter_name = Mathieu Gagné
tags = []
duplicates = []

Launchpad user Mathieu Gagné(mgagne) wrote on 2016-05-03T23:26:49.816769+00:00

When running Ubuntu 16.04 on OpenStack, cloud-init fails to properly configure the network from network_data.json found in ConfigDrive.

When instance boots, network is configured fine until next reboot where it falls back to dhcp.

The /etc/network/interfaces.d/50-cloud-init.cfg file has the following content when instance is initially booted, this could explain why dhcp is used on second boot:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

When debugging, if this line in stages.py [1] is commented, we can see that cloud-init initially copy the /etc/network/interfaces file found in the configdrive (the network template injected by Nova) and isn't using the network config found in network_data.json. But later it falls back to "dhcp" and rewrites yet again the network config.

I also found that within self._find_networking_config(), it looks like no datasource is found at this point. This could be because cloud-init is still in "local" dsmode and then refuses to use the network config found in the ConfigDrive. (triggering the "dhcp" fallback logic)

Manually forcing "net" dsmode makes cloud-init configure /etc/network/interfaces.d/50-cloud-init.cfg properly with network config found in the ConfigDrive. However no gateway is configured and so, instance doesn't respond to ping or SSH.

At that point, I'm not sure what's going on and how I can debug further.

Notes:

• The image used for testing uses "net.ifnames=0". Removing this config makes things much worst. (no ping at all on first boot)
• Logs, configs and configdrive can be found attached to this bug report.

[1] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/stages.py#L604

[ubuntu-server-builder]

Launchpad user Mathieu Gagné(mgagne) wrote on 2016-05-03T23:26:49.816769+00:00

Launchpad attachments: cloud-init configs, logs and configdrive

[ubuntu-server-builder]

Launchpad user Michal Adamczyk(vanditboy) wrote on 2016-05-18T08:41:46.701806+00:00

Hi,

My cloud platform is OpenStack Mitaka on RDO (CentOS 7.2) with XEN hypervisor and config_drive option.

On my guest system is CentOS 6.7.

Cloud-init agent can pickup meta_data.json (as I can login via ssh when bring up the network manually) but it cannot setup network:

Starting cloud-init: Cloud-init v. 0.7.5 running 'init-local' at Wed, 18 May 2016 08:19:18 +0000. Up 22.67 seconds.
Kdump is not supported on this kernel [FAILED]
Starting cloud-init: Cloud-init v. 0.7.5 running 'init' at Wed, 18 May 2016 08:19:18 +0000. Up 23.39 seconds.
ci-info: +++++++++++++++++++++++Net device info++++++++++++++++++++++++
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | Device | Up | Address | Mask | Hw-Address |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . |
ci-info: | eth0 | False | . | . | fa:16:3e:7e:8f:f5 |
ci-info: +--------+-------+-----------+-----------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2016-05-18 08:19:19,566 - util.py[WARNING]: Failed forking and calling callback NoneType
Starting cloud-init: Cloud-init v. 0.7.5 running 'modules:config' at Wed, 18 May 2016 08:19:20 +0000. Up 24.60 seconds.
2016-05-18 08:19:20,623 - util.py[WARNING]: Failed to run debconf-set-selections for grub-dpkg
2016-05-18 08:19:20,651 - util.py[WARNING]: Running apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python2.6/site-packages/cloudinit/config/cc_apt_configure.pyc'>) failed
Starting cloud-init: Cloud-init v. 0.7.5 running 'modules:final' at Wed, 18 May 2016 08:19:20 +0000. Up 25.41 seconds.
ci-info: ++++++++++++++++Authorized keys from /home/centos/.ssh/authorized_keys for user centos+++++++++++++++++
ci-info: +---------+-------------------------------------------------+---------+-------------------------------+
ci-info: | Keytype | Fingerprint (md5) | Options | Comment |
ci-info: +---------+-------------------------------------------------+---------+-------------------------------+
ci-info: | ssh-rsa | f3:f2:08:9f:cd:fd:1b:67:4c:9b:57:b7:5f:c6:54:e5 | - | - |
ci-info: | ssh-rsa | 75:3e:12:1e:09:51:1e:b9:6d:79:fd:4d:84:31:5d:51 | - | root@controller.openstack.dev |
ci-info: +---------+-------------------------------------------------+---------+-------------------------------+
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 5a:70:be:fb:bf:67:1b:ce:12:a6:7e:ba:ef:04:4f:ef /etc/ssh/ssh_host_dsa_key.pub (DSA)
ec2: 2048 50:22:cd:6d:1c:56:eb:10:42:54:97:b6:7a:81:41:f1 /etc/ssh/ssh_host_key.pub (RSA1)
ec2: 2048 15:55:5d:e6:91:af:46:3d:3a:bb:6f:f6:0a:e6:03:6f /etc/ssh/ssh_host_rsa_key.pub (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
2048 35 29586527545742027782438580921178232663437101143928703426912025054688176358288179690392668089004663466445702410732614605397783202624646139295502758398814167103819934034196746059236193699430655169196387579999363830849934023234378572470588905292275597549504565746765062482140473229038530226088359713215205775911747584015062512682996836980078230340684375809843010738745842550136067314468156851646942636770260038080954627830216946849892671468270824657347859346874780152848989039875644793621094494068028083537779233426499102119273566073302996706152597952328064853521441175800415933531654197800194000963846122530363176449477
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzsNNwGQFsq6PEYhVMz90R61xzLOc8CIiK1A4qbBra9GgGHyNOtIqQibj2QtF9FfuU8C8uRU/39TKXxQS1mt1TEzIa77LLkeLeigBtI0rueQ4ZDBrHYUwYOONCPjOqPoUN+cMvE5ZqI2rICVDpnldBnsTtuxasoGUBuTZ5zH5G5Rxx2T+rJT8+dlTRQ1QIJ23ByIBWt4ZeYQOW3sUA8ssTayIl9KOEs++9T2SjRp5kIvdJeFDRtdxvOzqxzuY6fUcW5leom294+k+W+zOtDwf7vrrcepdmRVVTc/PJBx8gMr7dxwGO9PuFhJuI427+TrTCqbnY7SdBYz3J8tcv9ND3Q==
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.5 finished at Wed, 18 May 2016 08:19:21 +0000. Datasource DataSourceConfigDriveNet [net,ver=2][source=/dev/xvdz]. Up 25.64 seconds
Starting mysqld: [ OK ]
...
...

cloud-init configuration file (beside original) /etc/cloud/cloud.cfg.d/10_file.cfg:

datasource_list: [ ConfigDrive, None ]
disable_root: False
ssh_pwauth: True
ssh_deletekeys: False
resize_rootfs: noblock
growpart:
mode: auto
devices: ['/']
system_info:
distro: rhel
cloud_config_modules:

• emit_upstart
• disk_setup
• mounts
• ssh-import-id
• locale
• set-passwords
• grub-dpkg
• apt-pipelining
• apt-configure
• package-update-upgrade-install
• landscape
• timezone
• puppet
• chef
• salt-minion
• mcollective
• disable-ec2-metadata
• runcmd
• byobu

Config drive content:

/media/ec2/2009-04-04/meta-data.json
/media/ec2/latest/meta-data.json
/media/openstack/2012-08-10/meta_data.json
/media/openstack/2013-04-04/meta_data.json
/media/openstack/2013-10-17/meta_data.json
/media/openstack/2013-10-17/vendor_data.json
/media/openstack/2015-10-15/meta_data.json
/media/openstack/2015-10-15/network_data.json
/media/openstack/2015-10-15/vendor_data.json
/media/openstack/latest/meta_data.json
/media/openstack/latest/network_data.json
/media/openstack/latest/vendor_data.json

Here is the network_data.json content:

host:root:/media> cat openstack/latest/network_data.json | python -m json.tool
{
"links": [
{
"ethernet_mac_address": "fa:16:3e:7e:8f:f5",
"id": "tapdee9dfc4-b0",
"mtu": null,
"type": "bridge",
"vif_id": "dee9dfc4-b07f-4b2e-aa6e-ae7cafacb502"
}
],
"networks": [
{
"id": "network0",
"ip_address": "172.20.0.37",
"link": "tapdee9dfc4-b0",
"netmask": "255.255.255.0",
"network_id": "7e22475a-fe2e-43b5-b65c-3730ebd2c343",
"routes": [
{
"gateway": "172.20.0.1",
"netmask": "0.0.0.0",
"network": "0.0.0.0"
}
],
"type": "ipv4"
}
],
"services": []
}

My cloud-init version:

Installed Packages
Name : cloud-init
Arch : x86_64
Version : 0.7.5
Release : 10.el6.centos.2
Size : 1.4 M
Repo : installed
From repo : extras

Anyone has this issue?

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2016-05-18T09:43:19.639792+00:00

Hi Michal,

Consumption of network_data.json was introduced in trunk at some point before revision 1212; you'll need to update to use a more recent version of cloud-init to get it working.

Thanks,

Dan

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-06-03T03:19:06.308702+00:00

Hi, anyone listening I'd like test of my ppa at https://launchpad.net/~smoser/+archive/ubuntu/cloud-init-dev .

You should be able to test by:

• clean xenial (or yakkety or wily) instance
• you can add a local user with a password interactively here to allow you back in if it fails to get neworking.
• sudo sh -c 'apt-add-repository -y ppa:smoser/cloud-init-dev && sudo apt-get update && sudo apt-get install cloud-init -qy'
• sudo reboot # [check that it works fine]
  look to see that there are no 'WARN' values in /var/log/cloud-init.log
• sudo rm -Rf /var/lib/cloud /var/log/cloud-init*
• sudo reboot
  look to see WARN values

If this does not work for you, please attach the config drive that was used for me to look at (you can just 'dd if=/dev/sr0 | gzip -c > my.iso.gz'

And if you see any WARN in logs, please attach them.
thank you.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-06-10T16:44:50.151050+00:00

This was fixed in yakkety at 0.7.7~bzr1227-0ubuntu1 . It will be sru'd sometime soon.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-06-10T16:45:44.327649+00:00

This was fixed in trunk at revno 1225

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-07-26T16:43:20.469949+00:00

Hi,
This bug was fixed in cloud-init at revision 1225 and was sru'd to xenial under bug 1595302, but unfortunately not even marked in that bug.

So anything newer than 0.7.7bzr1245-0ubuntu116.04.1 in xenial should have the fix.

[ubuntu-server-builder]

Launchpad user long(long-cheung) wrote on 2016-08-08T06:03:56.374651+00:00

Hello scott,

i using cloud-init Nocloud have same problem,
Hostname and password can setup,
but network can not,i using same config in ubuntu 14.04 is work,
but using 16.04 can not setup the network.Can you help me to check?

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-08-10T14:42:26.650812+00:00

This is fixed in cloud-init 0.7.7

[ubuntu-server-builder]

Launchpad user long(long-cheung) wrote on 2016-08-11T02:17:23.962832+00:00

Dear scott,

Thank for your reply,
But i already using 0.7.7bzr1256-0ubuntu16.04.1
Maybe i have something wrong,But i already check 1week,
Can not fix,....Can you please help to try?

[ubuntu-server-builder]

Launchpad user long(long-cheung) wrote on 2016-08-11T08:12:29.288944+00:00

Dear scott,

Follow is my meta-data and user-date,i using nocloud

instance-id: 1470899540
local-hostname: soscared
network-interfaces: |
auto eth0
iface eth0 inet static
address 103.225.10.12
netmask 255.255.255.0
gateway 103.225.10.1
dns-servers 8.8.8.8

#cloud-config
ssh_pwauth: True
chpasswd:
list: |
root:sosadc123456
expire: False

packages:

• vim
• qemu-guest-agent
• zsh

package_upgrade: true

runcmd:

• reboot

vim: syntax=yaml list

Then i run the cloud-init,Hostname and password can setup,But network is fail.

i check in obj.pkl ip format is wrong,
Is some in put setting wrong or cloud-init gen the format is wrong?

Launchpad attachments: obj.pkl

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-08-11T12:30:37.473660+00:00

Hi Long,
could you please open a different bug? and link to it from here?

This bug specifically handled 'config-drive' (the openstack disk format rather than 'NoCloud').

I have reproduced your issue.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-08-11T12:43:01.733569+00:00

Hi Long,
I went ahead and opened a bug at
https://bugs.launchpad.net/cloud-init/+bug/1612231

[ubuntu-server-builder]

Launchpad user long(long-cheung) wrote on 2016-08-11T14:16:49+00:00

Dear Scott,

I update some comment in bugs,

Can you please help,

Thank for your help.

Long

-----Original Message-----
From: bounces@canonical.com [mailto:bounces@canonical.com] On Behalf Of Scott Moser
Sent: Thursday, August 11, 2016 8:43 PM
To: Long Cheung
Subject: [Bug 1577982] Re: ConfigDrive: cloud-init fails to configure network from network_data.json

Hi Long,
I went ahead and opened a bug at
https://bugs.launchpad.net/cloud-init/+bug/1612231

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1577982

Title:
ConfigDrive: cloud-init fails to configure network from
network_data.json

Status in cloud-init:
Fix Released
Status in cloud-init package in Ubuntu:
Fix Released
Status in cloud-init source package in Xenial:
Fix Released

Bug description:
When running Ubuntu 16.04 on OpenStack, cloud-init fails to properly
configure the network from network_data.json found in ConfigDrive.

When instance boots, network is configured fine until next reboot
where it falls back to dhcp.

The /etc/network/interfaces.d/50-cloud-init.cfg file has the following
content when instance is initially booted, this could explain why dhcp
is used on second boot:

  auto lo
  iface lo inet loopback
  
  auto eth0
  iface eth0 inet dhcp

When debugging, if this line in stages.py [1] is commented, we can see
that cloud-init initially copy the /etc/network/interfaces file found
in the configdrive (the network template injected by Nova) and isn't
using the network config found in network_data.json. But later it
falls back to "dhcp" and rewrites yet again the network config.

I also found that within self._find_networking_config(), it looks like
no datasource is found at this point. This could be because cloud-init
is still in "local" dsmode and then refuses to use the network config
found in the ConfigDrive. (triggering the "dhcp" fallback logic)

Manually forcing "net" dsmode makes cloud-init configure
/etc/network/interfaces.d/50-cloud-init.cfg properly with network
config found in the ConfigDrive. However no gateway is configured and
so, instance doesn't respond to ping or SSH.

At that point, I'm not sure what's going on and how I can debug
further.

Notes:

• The image used for testing uses "net.ifnames=0". Removing this config makes things much worst. (no ping at all on first boot)
• Logs, configs and configdrive can be found attached to this bug report.

[1] http://bazaar.launchpad.net/~cloud-init-dev/cloud-
init/trunk/view/head:/cloudinit/stages.py#L604

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1577982/+subscriptions

[ubuntu-server-builder]

Launchpad user Peter(pry) wrote on 2016-11-22T14:08:08.374310+00:00

Hello, Scott!

Looks like the problem that Long stated in the post #8 persists and it wasn't fixed in 0.7.7.

I perform cloud image configuration using the 'network-interfaces' key in meta-data file. First boot goes smooth, I detach the iso file and reboot and the VM falls back to DHCP. I already have net.ifnames=0 in my kernel parameters. I'm using version 0.7.8 and it doesn't seem to be fixed there.

P.S. You've opened the bug in #13, but it looks like it's a different bug so i decided to write here. I'll open a separate bug if it is needed.

Some info:

$ cloud-init -v
cloud-init 0.7.8

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.8.0-27-generic root=UUID=abe2efde-1980-4e3f-bb55-b7bf90872ef9 ro console=tty1 console=ttyS0 net.ifnames=0 biosdevname=0 ds=nocloud

Running Ubuntu 16.10 x86_64

[ubuntu-server-builder]

Launchpad user Deepa(dpaclt) wrote on 2017-06-14T13:41:23.192498+00:00

I do have the same issue .

#cloud-init -v
cloud-init 0.7.9

Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial

[ubuntu-server-builder]

Launchpad user Marc Siegrist(dev-marc) wrote on 2017-08-17T11:13:54.589381+00:00

This still affects me with cloud-init version 0.7.9 and Ubuntu 16.04.03 Lts.

[ubuntu-server-builder]

Launchpad user David Britton(dpb) wrote on 2017-09-01T04:40:07.652525+00:00

Hi Marc, Deepa, Peter --

It's probably best to open a new bug on this. If you wouldn't mind doing so with a clear reproduction in the form of "what happened, what I expected, observations made" that would be very helpful!

Thanks.