ENI format - configuring metadata network-interfaces with iface defined twice but different family breaks all networking on cloud-init instance

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1745671

Launchpad details

affected_projects = ['cloud-init (CentOS)']
assignee = None
assignee_name = None
date_closed = None
date_created = 2018-01-26T19:45:38.021665+00:00
date_fix_committed = None
date_fix_released = None
id = 1745671
importance = medium
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1745671
milestone = None
owner = akaris
owner_name = Andreas Karis
private = False
status = triaged
submitter = akaris
submitter_name = Andreas Karis
tags = []
duplicates = []

Launchpad user Andreas Karis(akaris) wrote on 2018-01-26T19:45:38.021665+00:00

Description of problem:
Configuring metadata network-interfaces with "iface eth2 inet6 static" breaks all networking on cloud-init instance. This only works when configuring a sub interface.

The following breaks networking:

auto eth2
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
iface eth2 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
+++
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

auto eth0
iface eth0 inet static
    address 192.0.2.7
    netmask 255.255.255.0
    gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

iface eth0 inet6 static
    address 2001:db8::c0ca:1eaf
    netmask 64
    gateway 2001:db8::1ead:ed:beef

+++

And https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html
+++
Options are usually indented for clarity (as in the example above) but are not required to be.
+++

The following works correctly:

iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
auto eth2:0
iface eth2:0 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb

Additional info:
result working:

Working instance: ./sosreport-20180125-075718/svc-1-lvsrouter/var/log/cloud-init-output.log

Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 07:48:44 +0000. Up 12.41 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 07:48:47 +0000. Up 16.00 seconds.
ci-info: +++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: | Device |  Up  |    Address     |      Mask     | Scope |     Hw-Address    |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: |  lo:   | True |   127.0.0.1    |   255.0.0.0   |   .   |         .         |
ci-info: |  lo:   | True |       .        |       .       |   d   |         .         |
ci-info: | eth1:  | True | 10.250.246.171 | 255.255.252.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth1:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True | 192.168.0.166  |  255.255.0.0  |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True | 10.247.246.174 | 255.255.252.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True | 172.16.30.226  | 255.255.255.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination  |   Gateway   |    Genmask    | Interface | Flags |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: |   0   |   0.0.0.0    | 192.168.0.1 |    0.0.0.0    |    eth2   |   UG  |
ci-info: |   1   | 10.247.244.0 |   0.0.0.0   | 255.255.252.0 |    eth0   |   U   |
ci-info: |   2   | 10.250.244.0 |   0.0.0.0   | 255.255.252.0 |    eth1   |   U   |
ci-info: |   3   | 172.16.30.0  |   0.0.0.0   | 255.255.255.0 |    eth3   |   U   |
ci-info: |   4   | 192.168.0.0  |   0.0.0.0   |  255.255.0.0  |    eth2   |   U   |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 07:48:50 +0000. Up 19.03 seconds.

result not working:

Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 11:50:02 +0000. Up 7.98 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 11:50:10 +0000. Up 15.71 seconds.
2018-01-25 06:50:10,348 - util.py[WARNING]: Route info failed: Unexpected error while running command.
Command: ['netstat', '-rn']
Exit code: 1
Reason: -
Stdout: Kernel IP routing table
        Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
Stderr: -
ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
ci-info: |  lo:   | True |     .     |     .     |   d   |         .         |
ci-info: | eth1:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |     .     |     .     |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 11:50:12 +0000. Up 17.90 seconds.

[ubuntu-server-builder]

Launchpad user Andreas(andreas-redhat-bugs) wrote on 2018-01-26T16:04:26+00:00

Description of problem:
Configuring metadata network-interfaces with "iface eth2 inet6 static" breaks all networking on cloud-init instance. This only works when configuring a sub interface.

The following breaks networking:

auto eth2
iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
iface eth2 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:
+++
If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

auto eth0
iface eth0 inet static
    address 192.0.2.7
    netmask 255.255.255.0
    gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

iface eth0 inet6 static
    address 2001:db8::c0ca:1eaf
    netmask 64
    gateway 2001:db8::1ead:ed:beef

+++

And https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html
+++
Options are usually indented for clarity (as in the example above) but are not required to be.
+++

The following works correctly:

iface eth2 inet static
address 192.168.0.166
network services
netmask 255.255.0.0
broadcast 192.168.255.255
gateway 192.168.0.1
auto eth2:0
iface eth2:0 inet6 static
address 2001::166
gateway 2001::1
hwaddress aa:aa:aa:bb:bb:bb

Additional info:
result working:

Working instance: ./sosreport-20180125-075718/svc-1-lvsrouter/var/log/cloud-init-output.log

Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 07:48:44 +0000. Up 12.41 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 07:48:47 +0000. Up 16.00 seconds.
ci-info: +++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: | Device |  Up  |    Address     |      Mask     | Scope |     Hw-Address    |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: |  lo:   | True |   127.0.0.1    |   255.0.0.0   |   .   |         .         |
ci-info: |  lo:   | True |       .        |       .       |   d   |         .         |
ci-info: | eth1:  | True | 10.250.246.171 | 255.255.252.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth1:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True | 192.168.0.166  |  255.255.0.0  |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True | 10.247.246.174 | 255.255.252.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True | 172.16.30.226  | 255.255.255.0 |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True |       .        |       .       |   d   | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+----------------+---------------+-------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination  |   Gateway   |    Genmask    | Interface | Flags |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
ci-info: |   0   |   0.0.0.0    | 192.168.0.1 |    0.0.0.0    |    eth2   |   UG  |
ci-info: |   1   | 10.247.244.0 |   0.0.0.0   | 255.255.252.0 |    eth0   |   U   |
ci-info: |   2   | 10.250.244.0 |   0.0.0.0   | 255.255.252.0 |    eth1   |   U   |
ci-info: |   3   | 172.16.30.0  |   0.0.0.0   | 255.255.255.0 |    eth3   |   U   |
ci-info: |   4   | 192.168.0.0  |   0.0.0.0   |  255.255.0.0  |    eth2   |   U   |
ci-info: +-------+--------------+-------------+---------------+-----------+-------+
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 07:48:50 +0000. Up 19.03 seconds.

result not working:

Cloud-init v. 0.7.9 running 'init-local' at Thu, 25 Jan 2018 11:50:02 +0000. Up 7.98 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 25 Jan 2018 11:50:10 +0000. Up 15.71 seconds.
2018-01-25 06:50:10,348 - util.py[WARNING]: Route info failed: Unexpected error while running command.
Command: ['netstat', '-rn']
Exit code: 1
Reason: -
Stdout: Kernel IP routing table
        Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
Stderr: -
ci-info: +++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |    Mask   | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
ci-info: |  lo:   | True |     .     |     .     |   d   |         .         |
ci-info: | eth1:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth2:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: | eth0:  | True |     .     |     .     |   d   | xx:xx:xx:xx:xx:xx |
ci-info: | eth3:  | True |     .     |     .     |   .   | xx:xx:xx:xx:xx:xx |
ci-info: +--------+------+-----------+-----------+-------+-------------------+
ci-info: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Route info failed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 25 Jan 2018 11:50:12 +0000. Up 17.90 seconds.

[ubuntu-server-builder]

Launchpad user Andreas(andreas-redhat-bugs) wrote on 2018-01-26T16:21:13+00:00

Tested with cloud-init-0.7.9-20.el7.x86_64

[ubuntu-server-builder]

Launchpad user Ryan(ryan-redhat-bugs) wrote on 2018-01-26T18:30:53+00:00

I don't believe this is intended to work. Did defining the interface twice previously work for the customer?

The current upstream release explicitly errors out when you do this with a parse error.

[ubuntu-server-builder]

Launchpad user Andreas(andreas-redhat-bugs) wrote on 2018-01-26T19:38:14+00:00

Hi,

cloud-init upstrea in cloudinit/net/eni.py
https://github.com/number5/cloud-init/blob/master/cloudinit/net/eni.py#L193

 elif option == "iface":
            iface, family, method = split[1:4]
            if iface not in ifaces:
                ifaces[iface] = {
                    # Include the source path this interface was found in.
                    "_source_path": src_path
                }
            elif 'family' in ifaces[iface]:
                raise ParserError(
                    "Interface %s can only be defined once. "
                    "Re-defined in '%s'." % (iface, src_path))
            ifaces[iface]['family'] = family
            ifaces[iface]['method'] = method
            currif = iface

a) iface eth2 inet static
b) iface eth2 inet6 static

a)
iface = eth2
family = inet
method = static

b)
iface = eth2
family = inet6
method = static

inet will define ifaces['eth2']['family'] as 'inet' and then on the next pass with inet6, elif 'family' in ifaces[iface] will evaluate to true and this throw an error.

The actual error message doesn't seem to figure in any of the collected customer data, but from the above, it seems pretty clear that the ENI format accepts only one interface and not different tuples of iface/family.

[ubuntu-server-builder]

Launchpad user Andreas(andreas-redhat-bugs) wrote on 2018-01-26T19:44:20+00:00

I'm bringing this upstream ...

http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-eni.html#network-config-eni

states that:

Please reference existing documentation for the /etc/network/interfaces(5) format.

From https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html - the man page doesn't seem to say anything else about duplicate iface declarations with different families:

Stanzas defining logical interfaces start with a line consisting of the word "iface" followed by the name of the logical interface. In simple configurations without mapping stanzas this name should simply be the name of the physical interface to which it is to be applied. (The default mapping script is, in effect, the echo command.) The interface name is followed by the name of the address family that the interface uses. This will be "inet" for TCP/IP networking, but there is also some support for IPX networking ("ipx"), and IPv6 networking ("inet6"). Following that is the name of the method used to configure the interface. 
(...)

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:

If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef

[ubuntu-server-builder]

Launchpad user Andreas Karis(akaris) wrote on 2018-01-26T19:46:12.538341+00:00

I'm bringing this upstream ...

http://cloudinit.readthedocs.io/en/latest/topics/network-config-format-eni.html#network-config-eni

states that:

Please reference existing documentation for the /etc/network/interfaces(5) format.

From https://manpages.debian.org/jessie/ifupdown/interfaces.5.en.html - the man page doesn't seem to say anything else about duplicate iface declarations with different families:

Stanzas defining logical interfaces start with a line consisting of the word "iface" followed by the name of the logical interface. In simple configurations without mapping stanzas this name should simply be the name of the physical interface to which it is to be applied. (The default mapping script is, in effect, the echo command.) The interface name is followed by the name of the address family that the interface uses. This will be "inet" for TCP/IP networking, but there is also some support for IPX networking ("ipx"), and IPv6 networking ("inet6"). Following that is the name of the method used to configure the interface. 
(...)

According to https://wiki.debian.org/NetworkConfiguration , this should work, though:

If you're configuring it manually then something like this will set the default gateway (network, broadcast and gateway are optional):

    auto eth0
    iface eth0 inet static
        address 192.0.2.7
        netmask 255.255.255.0
        gateway 192.0.2.254

If you want to add an IPv6 address, too, append something like:

    iface eth0 inet6 static
        address 2001:db8::c0ca:1eaf
        netmask 64
        gateway 2001:db8::1ead:ed:beef

[ubuntu-server-builder]

Launchpad user Andreas Karis(akaris) wrote on 2018-01-26T19:49:00.018223+00:00

The implementation doesn't seem to like this, though:

cloud-init upstrea in cloudinit/net/eni.py
https://github.com/number5/cloud-init/blob/master/cloudinit/net/eni.py#L193

 elif option == "iface":
            iface, family, method = split[1:4]
            if iface not in ifaces:
                ifaces[iface] = {
                    # Include the source path this interface was found in.
                    "_source_path": src_path
                }
            elif 'family' in ifaces[iface]:
                raise ParserError(
                    "Interface %s can only be defined once. "
                    "Re-defined in '%s'." % (iface, src_path))
            ifaces[iface]['family'] = family
            ifaces[iface]['method'] = method
            currif = iface

a) iface eth2 inet static
b) iface eth2 inet6 static

a)
iface = eth2
family = inet
method = static

b)
iface = eth2
family = inet6
method = static

inet will define ifaces['eth2']['family'] as 'inet' and then on the next pass with inet6, elif 'family' in ifaces[iface] will evaluate to true and this throw an error.

The actual error message doesn't seem to figure in any of the collected customer data, but from the above, it seems pretty clear that the ENI format accepts only one interface and not different tuples of iface/family.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2019-07-19T20:49:32.545135+00:00

Thanks for filing the bug. The ifupdown eni parser in eni.py does not handle reading multiple stanzas of the same interface; however it is support in eni; so this is a bug in the eni format parser.