vlan configuration/unconfigured interfaces creates slow boot time

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1565711

Launchpad details

affected_projects = ['curtin', 'ifupdown', 'maas']
assignee = None
assignee_name = None
date_closed = None
date_created = 2016-04-04T10:37:11.909344+00:00
date_fix_committed = 2019-02-19T16:45:21.606808+00:00
date_fix_released = 2019-02-19T16:45:21.606808+00:00
id = 1565711
importance = undecided
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1565711
milestone = None
owner = blake-rouse
owner_name = Blake Rouse
private = False
status = incomplete
submitter = james-page
submitter_name = James Page
tags = ['networking']
duplicates = []

Launchpad user James Page(james-page) wrote on 2016-04-04T10:37:11.909344+00:00

maas: 1.9.1+bzr4543-0ubuntu1~trusty1 (from proposed PPA)

Deploying juju bootstrap node on Ubuntu 14.04 with the following network configuration:

eth0
static assigned IP address, default VLAN (no trunking)

eth1
static assigned IP address, secondary VLAN

eth1.2667
static assigned IP address, VLAN 2667

eth1.2668
static assigned IP address, VLAN 2668

eth1.2669
static assigned IP address, VLAN 2669

eth1.2670
static assigned IP address, VLAN 2670

eth2
unconfigured

eth3
unconfigured

MAAS generates a /e/n/i which auto stanzas for the VLAN devices and the unconfigured network interfaces; the upstart process which checks that network configuration is complete waits for /var/run/ifup.XXXX to exists for all auto interfaces; these will never appear for either the VLAN interfaces or the unconfigured network interfaces.

As a result, boot time if very long as cloud-init and networking both take 2 minutes to timeout waiting for network interfaces that will never appear to be configured.

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-04T10:44:51.178380+00:00

Correction; auto is required on the VLAN interfaces.

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-04T10:45:16.716873+00:00

That said, the update if-up.d script still waits for an if-up.XXX file that will never exist for a VLAN interface.

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-04T10:50:18.361800+00:00

/etc/network/if-up.d/upstart is the script that waits for all interfaces to be configured, and then emits the static-network-up event.

[ubuntu-server-builder]

Launchpad user Blake Rouse(blake-rouse) wrote on 2016-04-04T19:07:20.477373+00:00

Can you provide the created '/etc/network/interface' along with "maas [session] node get-curtin-config [system-id]
" and "maas [session] interfaces read [system-id]".

[ubuntu-server-builder]

Launchpad user Blake Rouse(blake-rouse) wrote on 2016-04-04T19:07:46.357430+00:00

This might also be related to curtin and cloud-init. Targeting them as well.

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-05T08:28:51.973700+00:00

Launchpad attachments: interfaces

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-05T08:29:03.273716+00:00

Launchpad attachments: interfaces-before-add-juju-bridge

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-05T08:29:17.157072+00:00

Launchpad attachments: maas-interfaces

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-05T08:29:28.817583+00:00

Launchpad attachments: curtin-config

[ubuntu-server-builder]

Launchpad user James Page(james-page) wrote on 2016-04-05T08:30:04.977242+00:00

Blake

requested information added to bug - setting back to New.

[ubuntu-server-builder]

Launchpad user Blake Rouse(blake-rouse) wrote on 2016-04-05T15:55:43.689851+00:00

So the configuration that MAAS emits and curtin generates looks correct. Cloud-init just waits for the signal so I actually think the issue is with ifupdown. I have targeted that package as well, will leave the others for now just to track.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2016-04-05T18:12:26.522769+00:00

I can recreate this issue with multiple vlan ifaces over eth2 in a guest.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2016-04-05T18:28:43.310955+00:00

I concur with the ifupdown issue.

I copied out /run/network from the VM that had very slow networking start (timeout on base vlan device, eth2, and both unconfigured "manual" ifaces).

drwxr-xr-x 3 rharper rharper 4096 Apr 5 13:17 ../
-rw-r--r-- 1 rharper rharper 88 Apr 5 13:13 dynamic-interfaces
-rw-r--r-- 1 rharper rharper 96 Apr 5 13:15 ifstate
-rw-r--r-- 1 rharper rharper 5 Apr 5 13:13 ifstate.eth0
-rw-r--r-- 1 rharper rharper 1 Apr 5 13:15 ifstate.eth1
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifstate.eth2
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2667
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2668
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2669
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2670
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifstate.eth3
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifstate.eth4
-rw-r--r-- 1 rharper rharper 3 Apr 5 13:13 ifstate.lo
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 .ifstate.lock
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth0
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2667
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2668
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2669
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2670
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.lo

Note, this same configuration (eni with vlans) passes under Xenial.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2016-04-05T19:29:58.165334+00:00

It also passes on Vivid and Wily, so looking at changes between ifupdown-0.7.47.2ubuntu4.4 (trusty-updates) and ifupdown-0.7.48.1ubuntu10 (vivid). These looks particularly relevant:

+ifupdown (0.7.48.1ubuntu1) utopic; urgency=medium
+

• [ Stéphane Graber ]

• • Merge from Debian. Remaining changes:
• • Allow setting the MTU and HWADDR on manual interfaces. (LP: #1294807)
• • The above change also means that manual interfaces will now be

•  brought up and down (ias many users expected until now).
  

• • Disable link.defn as it's not covering all the cases supported by

•  the vlan and bridge hooks and so causes more harm than good at this
  

•  point. (LP: #1295304)
  

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2016-04-05T20:03:52.753193+00:00

Just as a test, using 0.7.48-1ubuntu10 from vivid works fine. Now to narrow down changes.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2016-04-06T01:04:24.264002+00:00

Unfortunately it's racy; I've had the exact config attached succeed and fail on trusty.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2016-04-06T12:29:16.396739+00:00

And now, I can't get it to trigger at all. 300 runs last night with out triggering the fault.

James, is it reproducible for you on physical hardware?

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2016-04-06T14:28:20.487366+00:00

I don't think there is a curtin issue here but we may figure out a way to mitigate this issue if we can't figure out if there's a clear bug in ifupdown.

[ubuntu-server-builder]

Launchpad user Andres Rodriguez(andreserl) wrote on 2018-03-16T17:18:40.798072+00:00

We believe that this is not longer an issue in the latest releases of MAAS. If you believe this is still an issue, please re-open this bug report and target it accordingly.

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2019-02-19T16:45:16.529940+00:00

Similarly, please re-open the cloud-init task if you think anything is required from us.

[holmanb]

This was previously closed in launchpad, and relates to software stacks which have since been replaced (ifconfig/upstart/eni). Feel free to open a new report if you believe this to still be an issue.