tests: workaround for NetworkManager veths tests on Jammy

[daniloegea]

NM will not change the link state to UP after creating the interfaces and emit an error saying the second interface already exists. It looks like a bug in NM. NM will change the link state to UP after trying to create the connections again. Running netplan apply twice will workaround the issue.

It seems to be related to this commit https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/4655b7c308461ae1f86d592ea6d45e00a2820423

The problem can be reproduced on Jammy with the latest netplan code.

Create a file with this content:

network:
  renderer: NetworkManager
  virtual-ethernets:
    veth0:
      peer: veth1
    veth1:
      peer: veth0

Run netplan apply. Note that both connections AND interfaces will be created.

The following error can be seen in the systemd's journal for NetworkManager:

NetworkManager[12555]: <error> [1692787489.8705] manager: (netplan-veth1) couldn't create the device: Failed to create veth interface 'veth1' for 'netplan-veth1': exists

None of the veths will be brought UP

# nmcli con show
NAME            UUID                                  TYPE      DEVICE
netplan-enp5s0  e1419a5e-5e1d-3fff-a17c-b2fc6d2cef99  ethernet  enp5s0
netplan-veth0   8a7133c8-3a94-3f7f-9ac5-e0141c96d702  veth      --
netplan-veth1   eecdccba-9c5d-3e3c-bc5e-a8ca86d733e5  veth      --

Run netplan apply again. Both veth will be UP now:

# nmcli con show
NAME            UUID                                  TYPE      DEVICE
netplan-enp5s0  e1419a5e-5e1d-3fff-a17c-b2fc6d2cef99  ethernet  enp5s0
netplan-veth0   8a7133c8-3a94-3f7f-9ac5-e0141c96d702  veth      veth0
netplan-veth1   eecdccba-9c5d-3e3c-bc5e-a8ca86d733e5  veth      veth1

Description

Checklist

• Runs make check successfully.
• Retains 100% code coverage (make check-coverage).
• New/changed keys in YAML format are documented.
• (Optional) Adds example YAML for new feature.
• (Optional) Closes an open bug in Launchpad.

[daniloegea]

🥲 well...

[daniloegea]

My workaround fixed the first test but the second one is still failing. Both tests work when testing locally...

[daniloegea]

test_create_veth_pair (__main__.TestNetworkManager) ... veth0 ........veth1 ok
test_create_veth_pair_with_ip_address (__main__.TestNetworkManager) ... veth0 ............................................................GENERAL.DEVICE:                         veth0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         3E:04:9E:71:28:B1
GENERAL.MTU:                            1500
GENERAL.STATE:                          20 (unavailable)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
WIRED-PROPERTIES.CARRIER:               off
IP4.GATEWAY:                            --
IP6.GATEWAY:                            --
FAIL
test_create_veth_pair (__main__.TestNetworkd) ... veth0 veth1 ok
test_create_veth_pair_with_ip_address (__main__.TestNetworkd) ... veth0 ..veth1 ok

======================================================================
FAIL: test_create_veth_pair_with_ip_address (__main__.TestNetworkManager)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.vyej5y/build.MM7/real-tree/tests/integration/veths.py", line 80, in test_create_veth_pair_with_ip_address
    self.generate_and_settle(['veth0', 'veth1'])
  File "/tmp/autopkgtest.vyej5y/build.MM7/real-tree/tests/integration/base.py", line 363, in generate_and_settle
    self.nm_wait_connected(iface, 60)
  File "/tmp/autopkgtest.vyej5y/build.MM7/real-tree/tests/integration/base.py", line 419, in nm_wait_connected
    self.wait_output(['nmcli', 'dev', 'show', iface], '(connected', timeout)
  File "/tmp/autopkgtest.vyej5y/build.MM7/real-tree/tests/integration/base.py", line 416, in wait_output
    self.fail('timed out waiting for "{}" to appear in {}'.format(expected_output, cmd))
AssertionError: timed out waiting for "(connected" to appear in ['nmcli', 'dev', 'show', 'veth0']

[daniloegea]

I applied the upstream fix to network-manager 1.36.6 (used in Jammy) and it fixes the activation issue we are observing here. It might be worth to SRU it to Jammy.

https://git.launchpad.net/~danilogondolfo/ubuntu/+source/network-manager/diff/debian/patches/veth-fix-veth-activation-on-booting.patch?h=ubuntu/jammy-devel&id=7bdcc13ed0ca5b26bb9e55d1cedb562c8c2bbde2

I also prepared a PPA with the fix for testing https://launchpad.net/~danilogondolfo/+archive/ubuntu/network-manager

[daniloegea]

For reference https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/2032824

[daniloegea]

Closing. Fixed here #405