Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YAML consolidation: final patch set for configmanager.py (FR-702) #255

Merged
merged 15 commits into from
Jun 30, 2022

Conversation

schopin-pro
Copy link
Contributor

@schopin-pro schopin-pro commented Jan 11, 2022

Description

This patch set, the last one in the YAML consolidation series, ports ConfigManager to use libnetplan's YAML parser. In order to do so, we had to expose a lot of the internal data of the Netplan state to the Python bindings, as the netplan apply logic needs to inspect the definitions to know what to do.

Checklist

  • Runs make check successfully.
  • Retains 100% code coverage (make check-coverage).

@schopin-pro
Copy link
Contributor Author

This PR builds upon #250, #251, #252 and #254 hence the Draft status.

@codecov-commenter
Copy link

codecov-commenter commented Apr 13, 2022

Codecov Report

Merging #255 (e905416) into main (6a47c70) will decrease coverage by 0.26%.
The diff coverage is 99.41%.

❗ Current head e905416 differs from pull request most recent head f939394. Consider uploading reports for the commit f939394 to get more accurate results

@@            Coverage Diff             @@
##             main     #255      +/-   ##
==========================================
- Coverage   99.19%   98.93%   -0.27%     
==========================================
  Files          61       61              
  Lines       11206    11131      -75     
==========================================
- Hits        11116    11012     -104     
- Misses         90      119      +29     
Impacted Files Coverage Δ
netplan/cli/commands/try_command.py 100.00% <ø> (ø)
src/abi_compat.c 52.63% <ø> (-47.37%) ⬇️
src/parse.c 100.00% <ø> (+0.15%) ⬆️
netplan/configmanager.py 97.80% <93.33%> (-2.20%) ⬇️
netplan/cli/commands/apply.py 100.00% <100.00%> (ø)
netplan/cli/ovs.py 100.00% <100.00%> (ø)
netplan/cli/sriov.py 100.00% <100.00%> (ø)
netplan/cli/utils.py 100.00% <100.00%> (ø)
netplan/libnetplan.py 99.66% <100.00%> (-0.34%) ⬇️
src/sriov.c 100.00% <100.00%> (ø)
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 161ccfd...f939394. Read the comment docs.

Copy link
Collaborator

@slyon slyon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big kudos to @schopin-pro !
Thank you very much for this final PR in a long series of changes.
This YAML parser consolidation makes the netplan internals so much cleaner and easier to maintain!

I did an initial round of review and added several comments inline that we should address. I need to have another go on this big change, though, especially the "netplan: migration configmanager away from PyYAML" refactoring commit (0993272) – The testsuite looks all good, but I'd like to double check that we don't accidentally change the behavior here.

src/types.c Outdated Show resolved Hide resolved
src/types.c Outdated Show resolved Hide resolved
src/types.c Show resolved Hide resolved
src/util.c Show resolved Hide resolved
src/util.c Show resolved Hide resolved
netplan/cli/commands/try_command.py Outdated Show resolved Hide resolved
netplan/configmanager.py Show resolved Hide resolved
src/util.c Show resolved Hide resolved
tests/test_configmanager.py Show resolved Hide resolved
src/abi_compat.c Show resolved Hide resolved
Copy link
Collaborator

@slyon slyon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding some more inline comments, after having a 2nd look.

netplan/cli/commands/apply.py Outdated Show resolved Hide resolved
netplan/cli/ovs.py Show resolved Hide resolved
tests/test_configmanager.py Outdated Show resolved Hide resolved
tests/test_configmanager.py Show resolved Hide resolved
tests/test_configmanager.py Outdated Show resolved Hide resolved
tests/test_configmanager.py Show resolved Hide resolved
tests/test_configmanager.py Outdated Show resolved Hide resolved
This was an internal API so this shouldn't create too much problem.
Leaving the symbol as internal for now, pending publication in a
subsequent PR.
Those attributes are generic enough to be useful in a number of
situations, but will mostly be used when reimplementing the
configmanager.py features.
V2: Rewrite the matching algorithm to take the new driver list feature
    into account.
The whole sriov.py file needs detailed informations from the net
definitions. It used to get them from configmanager, but those are going
away really soon!

V2:
* Use correct uint return type and explicitly check against UINT_MAX
  for invalid VLAN ID.
* Prefix all new C symbols with an underscore to adhere to the new
  API/ABI conventions.
* Remove superfluous trivial_compound_definition property.
There are a couple of inconsistencies in the test data. The test.yaml
netplan config tries to configure a vlan interface on eth99, which
hasn't been defined, and the merged config tries to add two new links on
patcha and patchb, while they're already in a link together. While the
current code doesn't catch this, parser consolidation will result in
stricter validation, making the merge test fail.
…open

I suspect that the fallback is needed because configmanager opens the
netplan conf file in Python, but that won't necessarily be the case when
the parsing is done via libnetplan.
@schopin-pro schopin-pro force-pushed the yaml-consolidation-full branch 2 times, most recently from c96012b to f939394 Compare June 15, 2022 17:01
This change has far reaching impacts, as it changes the fundamental data
types of the class.

V2:
* Split off the matching algorithm evolution which does NOT belong to
  this patch.
* Move some small test data changes into their own patch to give it more
  context.
* Add a test covering the error path on invalid input, since we now have
  some validation.
* Some whitespace fixes in the tests.
* apply.py: rework the whole matching loop to better fit the new data
  model
* try.py: fix the missing extra-config application when checking for
  revertability.
Those old functions have been written out in favor of the
user-friendlier State and Parser classes, so we now clean them up.

This means that we now have more code in abi_compat.c that isn't covered
by the test suite. I think that's OK, as they are simple wrappers around
functions that *are* covered.

V2: remove superfluous LCOV_EXCL_START statement
@schopin-pro
Copy link
Contributor Author

I finally took the time to go through the review comments. I left a couple of discussions open as there are still unanswered questions there. Sorry, I forgot to keep a global log of the new version, although each commit should have a local changelog.

Copy link
Collaborator

@slyon slyon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for all the effort you've put into this @schopin-pro!

All my remarks have been addressed, with some comments still open/unresolved. But those are about exporting the symbols according to the new libnetplan API spec and we agreed upon that this adoption will be done separately in an additional PR.

All unit-tests, ABI checks and integration-/autopkgtests PASS, so this is ready for merging! \o/

@slyon
Copy link
Collaborator

slyon commented Jun 29, 2022

I've created a PPA build including this PR + #254 + #275: https://launchpad.net/~slyon/+archive/ubuntu/netplan-yaml-parser-pr255/+packages

In addition to running some official autopkgtests against this build, I'd like to ask @mvo5 to maybe run the snapd test-suite against it, too, as this is the biggest consumer of the netplan get/set functionality.

@slyon slyon requested a review from mvo5 June 29, 2022 09:17
@mvo5
Copy link
Contributor

mvo5 commented Jun 29, 2022

Fwiw, I build a core22 snap that includes this netplan PPA:

$ git diff
diff --git a/hooks/001-extra-packages.chroot b/hooks/001-extra-packages.chroot
index 0a8ed86..5faafbb 100755
--- a/hooks/001-extra-packages.chroot
+++ b/hooks/001-extra-packages.chroot
@@ -51,6 +51,11 @@ Pin: origin ""
 Pin-Priority: 1001
 EOF
 
+# add netplan test repo
+apt update
+apt install -y software-properties-common
+add-apt-repository -y ppa:slyon/netplan-yaml-parser-pr255
+
 # install some packages we need
 apt update
 apt dist-upgrade -y

When I install that in a core22 image as a local install and run our tests (https://github.com/snapcore/snapd/blob/master/tests/core/netplan-cfg/task.yaml#L31) I get:

mvo@ubuntu:~$ grep netplan /usr/share/snappy/dpkg.list 
ii  libnetplan0:amd64             0.104-0ubuntu2.1+yaml0~ppa0             amd64        YAML network configuration abstraction runtime library
ii  netplan.io                    0.104-0ubuntu2.1+yaml0~ppa0             amd64        YAML network configuration abstraction for various backends
$ sudo -s
root@ubuntu:/home/mvo#     snap set system system.network.netplan.network.bridges.br54.dhcp4=false
error: cannot perform the following tasks:
- Run configure hook of "core" snap (run hook "configure": cannot try netplan config: no specific reason returned from netplan and cannot cancel netplan config: Unknown object '/io/netplan/Netplan/config/RV6MO1'.)
root@ubuntu:/home/mvo# journalctl -u snapd
Jun 29 11:25:25 ubuntu systemd[1]: Starting Snap Daemon...
Jun 29 11:25:28 ubuntu snapd[480]: AppArmor status: apparmor is enabled and all 
features are available
Jun 29 11:25:28 ubuntu snapd[480]: overlord.go:263: Acquiring state lock file
Jun 29 11:25:28 ubuntu snapd[480]: overlord.go:268: Acquired state lock file
Jun 29 11:25:28 ubuntu snapd[480]: daemon.go:247: started snapd/2.56 (series 16)
 ubuntu-core/22 (amd64) linux/5.15.0-35-generic.
Jun 29 11:25:28 ubuntu snapd[480]: daemon.go:340: adjusting startup timeout by 5
0s (pessimistic estimate of 30s plus 5s per snap)
Jun 29 11:25:28 ubuntu snapd[480]: devicemgr.go:326: bind-mounting ubuntu-save u
nder /var/lib/snapd/save
Jun 29 11:25:29 ubuntu systemd[1]: Started Snap Daemon.
Jun 29 11:26:28 ubuntu snapd[480]: api_snaps.go:317: Installing snap "jq" revisi
on unset
Jun 29 11:27:40 ubuntu snapd[480]: taskrunner.go:271: [change 5 "Run configure h
ook of \"core\" snap" task] failed: run hook "configure": cannot try netplan con
fig: no specific reason returned from netplan and cannot cancel netplan config: 
Unknown object '/io/netplan/Netplan/config/RV6MO1'.

root@ubuntu:/home/mvo# journalctl
...
Jun 29 11:32:31 ubuntu snapd[942]: netplan.go:228: DEBUG: store reachable before
 netplan changes: true (tried 0 times)
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]: Traceback (most recent call last
):
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:   File "/usr/sbin/netplan", line
 23, in <module>
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:     netplan.main()
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:   File "/usr/share/netplan/netpl
an/cli/core.py", line 50, in main
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:     self.run_command()
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:   File "/usr/share/netplan/netpl
an/cli/utils.py", line 225, in run_command
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:     self.func()
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:   File "/usr/share/netplan/netpl
an/cli/commands/try_command.py", line 80, in run
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:     self.run_command()
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:   File "/usr/share/netplan/netpl
an/cli/utils.py", line 225, in run_command
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:     self.func()
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:   File "/usr/share/netplan/netpl
an/cli/commands/try_command.py", line 83, in command_try
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:     if not self.is_revertable():
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:   File "/usr/share/netplan/netpl
an/cli/commands/try_command.py", line 177, in is_revertable
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]:     if settings and 'parameters'
 in settings:
Jun 29 11:32:31 ubuntu io.netplan.Netplan[995]: TypeError: argument of type 'Net
Definition' is not iterable
Jun 29 11:32:36 ubuntu io.netplan.Netplan[709]: 'netplan try' exited with status
: 0

@slyon
Copy link
Collaborator

slyon commented Jun 29, 2022

Thanks for testing, this is a good catch! I've built a unit-test case around it. I think we need to add a new private/internal _netplan_netdef_get_parameters() function to check for this. @schopin-pro WDYT?

diff --git a/netplan/cli/commands/try_command.py b/netplan/cli/commands/try_command.py
index 2e0b1af..20b8cbf 100644
--- a/netplan/cli/commands/try_command.py
+++ b/netplan/cli/commands/try_command.py
@@ -43,6 +43,7 @@ class NetplanTry(utils.NetplanCommand):
                          leaf=True)
         self.configuration_changed = False
         self.new_interfaces = None
+        self.config_file = None
         self._config_manager = None
         self.t_settings = None
         self.t = None
@@ -52,7 +53,7 @@ class NetplanTry(utils.NetplanCommand):
     @property
     def config_manager(self):  # pragma: nocover (called by later commands)
         if not self._config_manager:
-            self._config_manager = ConfigManager()
+            self._config_manager = ConfigManager(prefix=self._rootdir)
         return self._config_manager
 
     def clear_ready_stamp(self):
@@ -146,7 +147,7 @@ class NetplanTry(utils.NetplanCommand):
     def cleanup(self):  # pragma: nocover (requires user input)
         self.config_manager.cleanup()
 
-    def is_revertable(self):  # pragma: nocover (requires user input)
+    def is_revertable(self):
         '''
         Check if the configuration is revertable, if it doesn't contain bits
         that we know are likely to render the system unstable if we apply it,
@@ -161,7 +162,7 @@ class NetplanTry(utils.NetplanCommand):
         '''
 
         extra_config = []
-        if self.config_file:
+        if self.config_file:  # pargma: nocover
             extra_config.append(self.config.file)
         np_state = self.config_manager.parse(extra_config=extra_config)
         revert_unsupported = []
diff --git a/tests/test_cli_units.py b/tests/test_cli_units.py
index 48fe2d9..b235aeb 100644
--- a/tests/test_cli_units.py
+++ b/tests/test_cli_units.py
@@ -34,6 +34,7 @@ class TestCLI(unittest.TestCase):
     def setUp(self):
         self.tmproot = tempfile.mkdtemp()
         os.mkdir(os.path.join(self.tmproot, 'run'))
+        os.makedirs(os.path.join(self.tmproot, 'etc/netplan'))
         os.environ['DBUS_TEST_NETPLAN_ROOT'] = self.tmproot
 
     def tearDown(self):
@@ -103,3 +104,28 @@ class TestCLI(unittest.TestCase):
         self.assertTrue(os.path.isfile(stamp_file))
         self.assertTrue(cmd.clear_ready_stamp())
         self.assertFalse(os.path.isfile(stamp_file))
+
+    def test_netplan_try_is_revertable(self):
+        with open(os.path.join(self.tmproot, 'etc/netplan/test.yaml'), 'w') as f:
+            f.write('''network:
+  bridges:
+    br54:
+      dhcp4: false
+''')
+        cmd = NetplanTry()
+        self.assertTrue(cmd.is_revertable())
+
+    def test_netplan_try_is_not_revertable(self):
+        with open(os.path.join(self.tmproot, 'etc/netplan/test.yaml'), 'w') as f:
+            f.write('''network:
+  ethernets:
+    eth0:
+      dhcp4: true
+  bonds:
+    bn0:
+      interfaces: [eth0]
+      parameters:
+        mode: balance-rr
+''')
+        cmd = NetplanTry()
+        self.assertFalse(cmd.is_revertable())

@schopin-pro
Copy link
Contributor Author

Eh, I guess we finally found out why there was this trivial_compound_definition method after all :). Granted, the name wasn't very obvious.

I guess this code must have regressed during one of the many rebases and reworks of the patchset. I'll fix this in a jiffy, along with the tests you've nicely written.

schopin-pro and others added 4 commits June 29, 2022 16:42
Instead of forcing users to cast (which was done through a macro in
netplan.c anyway), we now use the proper type to say "give us a pointer
to whatever, we promise we won't touch its data".

This function will be used to expose a bit of the dirtiness of the bonds
and bridges properties to Python, which in turn will tell us how
revertable a given config is.
It marked the wrong pointer, instead marking the address at the end of
the input string as dirty, which doesn't make much sense (and would be
prone to conflicts down the line as allocations are reused).
The underlying implementation uses the dirtiness infrastructure for
this, as this allows for a rather generic implementation, and shows us
that there are actual parameters touched by the user config, whereas the
explicit boolean `custom_bridging` only tells us that there's a
parameters block, which might very well be empty.
The code hadn't been fully migrated to the new libnetplan-backed
configmanager.

This commit also adds some unit tests for this function, which were
sorely lacking.

Co-authored-by: Lukas Märdian <lukas.maerdian@canonical.com>
@schopin-pro
Copy link
Contributor Author

The issue should be fixed. Thanks for the tests, @mvo5! Not only was this particular CLI codepath buggy, but fixing it led me to uncover another issue deep in the parsing code that's now fixed in the branch as well.

@slyon
Copy link
Collaborator

slyon commented Jun 30, 2022

I've updated the PPA and built a core22 base snap & qemu image, as described by mvo above and ran the tests manually (from https://github.com/snapcore/snapd/blob/master/tests/core/netplan-cfg/task.yaml). All passed, looking good to me!
@mvo5 is there anything to test besides this? Otherwise, I think we're ready to merge this beast.

root@ubuntu:/home/slyon# grep netplan /usr/share/snappy/dpkg.list
ii  libnetplan0:amd64             0.104-0ubuntu2.1+yaml0~ppa1             amd64        YAML network configuration abstraction runtime library
ii  netplan.io                    0.104-0ubuntu2.1+yaml0~ppa1             amd64        YAML network configuration abstraction for various backends
root@ubuntu:/home/slyon# snap get system system.network.netplan.network.version
2
root@ubuntu:/home/slyon# snap get -d system system.network.netplan | jq .
{
  "system.network.netplan": {
    "network": {
      "ethernets": {
        "ens3": {
          "dhcp4": true
        }
      },
      "version": 2
    }
  }
}
root@ubuntu:/home/slyon# snap set system system.network.netplan.network.bridges.br54.dhcp4=false
root@ubuntu:/home/slyon# netplan get
network:
  version: 2
  ethernets:
    ens3:
      dhcp4: true
  bridges:
    br54:
      dhcp4: false
root@ubuntu:/home/slyon# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
4: br54: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 0a:95:e6:55:72:c9 brd ff:ff:ff:ff:ff:ff
root@ubuntu:/home/slyon# res=$(yq e '.network.bridges.br54.dhcp4' - < /etc/netplan/90-snapd-config.yaml)
root@ubuntu:/home/slyon# echo $res
false
root@ubuntu:/home/slyon# snap set system system.network.netplan.network.bridges.br54.dhcp6=false
root@ubuntu:/home/slyon# res=$(yq e '.network.bridges.br54.dhcp6' - < /etc/netplan/90-snapd-config.yaml)
root@ubuntu:/home/slyon# echo $res
false
root@ubuntu:/home/slyon# res=$(yq e '.network.bridges.br54.dhcp4' - < /etc/netplan/90-snapd-config.yaml)
root@ubuntu:/home/slyon# echo $res
false
root@ubuntu:/home/slyon# snap unset system system.network.netplan.network.bridges.br54.dhcp6
root@ubuntu:/home/slyon# res=$(yq e '.network.bridges.br54.dhcp6' - < /etc/netplan/90-snapd-config.yaml)
root@ubuntu:/home/slyon# echo $res
null
root@ubuntu:/home/slyon# snap unset system system.network.netplan.network.bridges.br54
root@ubuntu:/home/slyon# snap get -d system system.network.netplan
{
	"system.network.netplan": {
		"network": {
			"ethernets": {
				"ens3": {
					"dhcp4": true
				}
			},
			"version": 2
		}
	}
}
root@ubuntu:/home/slyon# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s3

@slyon
Copy link
Collaborator

slyon commented Jun 30, 2022

mvo replied out-of-band that there's nothing else to be tested from the snapd side. And Simon's latest changes LGTM. Merging.

@slyon slyon merged commit 474951a into canonical:main Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants