Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First boot OVS/WPA workaround for cloud-init #157

Closed

Conversation

slyon
Copy link
Collaborator

@slyon slyon commented Aug 4, 2020

Description

Cloud-init makes use of the netplan generator, but calls netplan generate manually at runtime (during the systemd boot transaction), instead of running it as intended at systemd generator stage, due to restrictions it has regarding fetching of its data source (e.g. netplan YAML config).

This leads to problems at first boot, as the systemd unit dependencies are calculated after the generator stage, but ahead of the boot transaction (e.g. via systemctl daemon-reload), therefore the new service units and its dependencies, which are generated by manually calling netplan generate are ignored during the first-boot transaction. In subsequent boots (where the cloud-init data source, netplan YAML config and unit files are already in place), everything works as expected.

Systemd v246 introduced a new feature, where target units can be lazy-loaded at runtime, if they are known ahead of a transaction (but still unloaded) and changed on disk in the meantime. Therefore, we always place a new netplan.target systemd unit in /lib/systemd/system/netplan.target, which does almost nothing and isn't loaded by default, but is available at all times on disk. When executing netplan generate we create new symlinks/dependencies as /run/systemd/system/netplan.target.wants/netplan-[ovs|wpa]-*.service, so that cloud-init (or any other service) can call systemctl start netplan.target after having called netplan generate to have the target unit incl. its new dependencies lazy loaded and started – even during a systemd (boot-) transaction.

We cannot statically link the netplan.target e.g. via /lib/systemd/system/network.target.wants/netplan.target, as my experiments showed that it would then already be loaded (read from disk) when the initial dependencies are calculated (ahead of the boot transaction) and therefor it does not know about the netplan service units, which are generated at a later stage by cloud-init calling netplan generate after it put the YAML config in place.

So in order to get this beast working, we need:

Reproducer

lxc profile copy default myovs
lxc profile edit myovs
# Add this config:
config:
  user.network-config: |
    # cloud-config
    version: 2
    bridges:
      ovs0:
        addresses: [10.10.10.20/24]
        interfaces: [eth0.21]
        parameters:
          stp: false
        openvswitch: {}
    ethernets:
      eth0:
        dhcp4: true
        addresses: [10.10.10.30/24]
    vlans:
      eth0.21:
        id: 21
        link: eth0
description: My OVS debugging profile

lxc remote add --protocol simplestreams ubuntu-minimal-daily https://cloud-images.ubuntu.com/minimal/daily/
lxc launch ubuntu-minimal-daily:groovy ovs-init
lxc file push clean.sh *netplan*_0.99-0ubuntu5_amd64.deb ovs-init/root/
lxc exec ovs-init bash
$ apt update
$ apt install software-properties-common
$ add-apt-repository ppa:ci-train-ppa-service/4177 # backported systemd features
$ apt install systemd openvswitch-switch
$ dpkg -i *netplan*_0.99-0ubuntu5_amd64.deb
$ ./clean.sh && poweroff
lxc snapshot ovs-init # prepare snapshot image of the state we prepared

lxc copy ovs-init/snap0 ovs-debug --profile myovs # start new instance with OVS netplan datasource
lxc start ovs-debug
lxc exec ovs-debug bash
$ ip a
$ ovs-vsctl show
$ ovsdb-tool show-log
$ systemctl list-dependencies netplan.target
$ systemctl status netplan-ovs-cleanup.service
$ systemctl status netplan-ovs-ovs0.service
$ cat /etc/netplan/*.yaml
$ netplan apply
$ reboot
$ ./clean && reboot # to simulate a first-boot

Checklist

  • Runs make check successfully.
  • Retains 100% code coverage (make check-coverage).
  • New/changed keys in YAML format are documented.
  • (Optional) Closes an open bug in Launchpad: LP#1870346 (partly)

… via systemd to start all netpla-*.service units at runtime
This way it can be started on first boot by cloud-init via 'systemctl start netplan.target'

This partly fixes (LP: #1870346), but needs fixed to cloud-init and
systemd v246 (systemd/systemd#16371) as well.
…get'"

This reverts commit d3dab6e4590f4e7d6006996d0b2c3e985337d356.
@slyon slyon marked this pull request as ready for review August 4, 2020 14:24
@xnox
Copy link
Contributor

xnox commented Aug 4, 2020

Changes in cloud-init to call systemctl start netplan.target after calling netplan generate in cloudinit/net/netplan.py:243

Hm, why can't we instead change netplan generate to do that, in the python code, after it executes the C-generate binary?

@@ -967,6 +967,7 @@ write_networkd_conf(const NetplanNetDefinition* def, const char* rootdir)

if (def->type == NETPLAN_DEF_TYPE_WIFI || def->has_auth) {
g_autofree char* link = g_strjoin(NULL, rootdir ?: "", "/run/systemd/system/systemd-networkd.service.wants/netplan-wpa-", def->id, ".service", NULL);
g_autofree char* link_target = g_strjoin(NULL, rootdir ?: "", "/run/systemd/system/netplan.target.wants/netplan-wpa-", def->id, ".service", NULL);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this facility is available, we should possibly also add netplan.target.wants/systemd-networked.service and netplan.target.wants/systemd-wait-online.server. This would enable us to have neither networkd nor NM enabled in the image, and cloud-init able to activate one or the other on boot.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should work. We would need some extra code to activate NetworkManager, as we're not setting up systemd units/dependencies from our NM renderer currently, but its certainly possible.

This would only work for cloud-init based images, though. As in a "normal" boot sequence netplan.target would never be called/activated.

@slyon
Copy link
Collaborator Author

slyon commented Aug 4, 2020

Changes in cloud-init to call systemctl start netplan.target after calling netplan generate in cloudinit/net/netplan.py:243

Hm, why can't we instead change netplan generate to do that, in the python code, after it executes the C-generate binary?

That is what I did here for testing. But in the real world people expect netplan generate to just generate the configs but not to apply/activate them. People call netplan try/apply instead if they want to have it applied right away.

So it would change the semantics of netplan's CLI generate command and possibly break lots of stuff because of this.
What we could do is to add a netplan generate --activate flag (or similar)... But I'm not a big fan of this, because people would probably confuse netplan apply VS netplan generate --activate. Also, this netplan.target is just a workaround for cloud-init and not used anywhere else, so I'd like to avoid changing public CLI for this workaround.

@raharper
Copy link
Collaborator

raharper commented Aug 4, 2020

Also, this netplan.target is just a workaround for cloud-init and not used anywhere else, so I'd like to avoid changing public CLI for this workaround.

I'm not terribly happy about the idea of this being a workaround. The primary case for OVS is in deploying cloud-init base images.
Should we not be designing with cloud-init netplan rendering in mind?

therefore the new service units and its dependencies, which are generated by manually calling netplan generate are ignored during the first-boot transaction.

What service units and deps are generated from netplan generate?
Is this an OVS specific implementation detail of netplan?
Could netplan generate render hooks for network-dispatcher which could invoke the newly generated systemd units at the right time?

netplan-[ovs|wpa]-*.service

Instead of generating custom service files (tell me if I'm incorrect about the content being dynamic); could those services read configuration files that netplan generate writes? In the case that netplan generate has no ovs or wpa content, those services are inactive; ConditionPath=/run/somepath/netplan-generate/writes/{wpa,ovs}?

@slyon
Copy link
Collaborator Author

slyon commented Aug 5, 2020

I'm not terribly happy about the idea of this being a workaround. The primary case for OVS is in deploying cloud-init base images. Should we not be designing with cloud-init netplan rendering in mind?

therefore the new service units and its dependencies, which are generated by manually calling netplan generate are ignored during the first-boot transaction.

What service units and deps are generated from netplan generate?
Is this an OVS specific implementation detail of netplan?
Could netplan generate render hooks for network-dispatcher which could invoke the newly generated systemd units at the right time?

Indeed it isn't great that it needs a workaround to fix cloud-init first boot. But the current design (using systemd service units) was spec'ed and signed-off quite some time ago. This is not a OVS specific implementation detail, as the same approach has been used in netplan to configure wpa_supplicant for many years.

I guess we would need @xnox's and/or @vorlonofportland's opinion if we wanted to change the overall architecture here.

Generally, netplan creates the following systemd units, using its networkd/OVS backends:

  • `/run/systemd/network/10-netplan-*.[netdev|network]
    • Those are triggered by changes to the (pyhsical) network interfaces and work as expected.
  • /run/systemd/system/netplan-[wpa|ovs]-*.service
    • Those are Type=oneshot units using ExecStart= commands to setup wpa_supplicant or OVS
    • The service units are usually triggered by systemd-networkd.service (see below)
    • The service units define certain inline Before=/After=/Requires= dependencies on each other
  • /run/systemd/system/systemd-networkd.service.wants/netplan-[wpa|ovs]-*.service
    • Symlink dependencies to start the netplan-[wpa|ovs]-*.service units
  • /run/systemd/system/netplan.target.wants/netplan-[wpa|ovs]-*.service – NEW
    • Symlink dependencies to start netplan-[wpa|ovs]-*.service after lazy-loading during runtime
    • Needed if used in an unintended way (non systemd generator)

Some examples are netplan-ovs-global.service, netplan-ovs-cleanup.service, netplan-ovs-ovsbr0.service, netplan-wpa-wlan0.service, ...

netplan-[ovs|wpa]-*.service

Instead of generating custom service files (tell me if I'm incorrect about the content being dynamic); could those services read configuration files that netplan generate writes? In the case that netplan generate has no ovs or wpa content, those services are inactive; ConditionPath=/run/somepath/netplan-generate/writes/{wpa,ovs}?

Correct, those service units contain dynamic content. The ExecStart= part could probably be moved to generated shell scripts which would then be called by a single, static service unit. But some of those units also depend on given network interfaces, e.g.:

"Requires=sys-subsystem-net-devices-%s.device"
"After=sys-subsystem-net-devices-%s.device"

%s being the name of a network interface like "wlan0", "ens3", which is not known ahead of time, but dynamically defined by the YAML config at runtime. I am not sure if/how this could be handled, using a static service unit + shell scripts.


@raharper A totally different approach I discussed with @xnox, which would not require any changes to netplan or systemd at all, would be to split up the cloud-init boot sequence into multiple stages, e.g.:

  • Start "Stage 0" systemd transaction: systemctl isolate cloud-stage0.target
    • execute the init local modules
    • setup basic networking (DHCP on eth0/ens3)
    • fetch data source & place netplan YAML in /etc/netplan/
  • Finish "Stage 0" transaction
  • Call systemctl daemon-reload
    • This will trigger all systemd generator (incl. netplan generate) and re-calculate all dependencies
  • Start "Stage 1" systemd transaction systemctl isolate cloud-final.target
    • execute all the normal cloud-init modules and start all the normal services
  • Finish "Stage 1" transaction
  • System is now fully booted

The idea here is to split up the boot sequence into two (or more?) systemd transactions, so we can call "daemon-reload" in between to re-run all the generators and re-calculate all the dependencies. This way all generators would be used in their intended way and would not need any workarounds. I am not sure if/how this is feasible to be implemented in cloud-init.

@xnox
Copy link
Contributor

xnox commented Aug 5, 2020

Also, this netplan.target is just a workaround for cloud-init and not used anywhere else, so I'd like to avoid changing public CLI for this workaround.

I'm not terribly happy about the idea of this being a workaround. The primary case for OVS is in deploying cloud-init base images.
Should we not be designing with cloud-init netplan rendering in mind?

I disagree with the usage of word "workaround". Netplan generate creates many units and dependencies which when created at systemd-genetator time are correctly loaded as part of the boot.

However, that precludes at the moment adding new units to the initial boot transaction when netplan yaml is not known ahead of time. For example when cloud-init injects it, just in time.

Hence today, cloud-init cannot dynamically enable/disable networkd or NetworkManager renderers via netplan. And we have to bake networkd/wait-online units as enabled, hoping that cloud-init will inject netplan and generate will be run before those daemons start.

therefore the new service units and its dependencies, which are generated by manually calling netplan generate are ignored during the first-boot transaction.

What service units and deps are generated from netplan generate?

network.target.wants systemd-networkd.service|socket, systemd-networkd-online, NetworkManager.service & its wait-online, wpa-supplicant "WiFi" units, OVS service units to establish OVS and configure bridges & fakevlans, or cleanup old OVS state (as itself is persistent across reboots).

Is this an OVS specific implementation detail of netplan?

No. Any and all renderings of netplan require units to be enabled always. So far we have worked around that by baking in networkd or NM as enabled on server/desktop images.

Could netplan generate render hooks for network-dispatcher which could invoke the newly generated systemd units at the right time?

That would be too late for some of them. And not guaranteed to complete before we start executing cloud-config / cloud-final stages of cloud-init at which point WiFi/OVS must already be up, running and configured.

netplan-[ovs|wpa]-*.service

Instead of generating custom service files (tell me if I'm incorrect about the content being dynamic); could those services read configuration files that netplan generate writes? In the case that netplan generate has no ovs or wpa content, those services are inactive; ConditionPath=/run/somepath/netplan-generate/writes/{wpa,ovs}?

That would loose granularity though. OVS rejecting setting up one bridge, will mean the rest of bridges are not attempted to be configured. For WPA, we do need a long running Daemon per WiFi network, as it has to continuously renew beacons & roam between APs. And it wouldn't help with starting NetworkManager or networkd.

@raharper alternative to this, is to re-engineer how cloud-init targets work a bit. Instead of booting to default.target, divert the boot to cloud-local.target, fully complete booting to that (no jobs left), then call systemctl daemon-reload, then start default.target with new dependencies calculated. Doing that would allow users to do interesting things with systemd via cloud-config. Like changing the default.target from multiuser.target to emergency.target, adding / masking / removing units used in early boot, and "just write fstab" and allow systemd-fstab-generator to process it, and mount things, etc. Or systemd should be fixed to allow dynamically picking up new targets & units always, like upstart did....

@raharper
Copy link
Collaborator

raharper commented Aug 5, 2020

Also, this netplan.target is just a workaround for cloud-init and not used anywhere else, so I'd like to avoid changing public CLI for this workaround.

I'm not terribly happy about the idea of this being a workaround. The primary case for OVS is in deploying cloud-init base images.
Should we not be designing with cloud-init netplan rendering in mind?

I disagree with the usage of word "workaround". Netplan generate creates many units and dependencies which when created at systemd-genetator time are correctly loaded as part of the boot.

However, that precludes at the moment adding new units to the initial boot transaction when netplan yaml is not known ahead of time. For example when cloud-init injects it, just in time.

Hence today, cloud-init cannot dynamically enable/disable networkd or NetworkManager renderers via netplan. And we have to bake networkd/wait-online units as enabled, hoping that cloud-init will inject netplan and generate will be run before those daemons start.

It's not a hope; cloud-init runs before those daemons start, feeding them
config prior to their start.

therefore the new service units and its dependencies, which are generated by manually calling netplan generate are ignored during the first-boot transaction.

This is the core issue. Neither networkd nor NetworkManager utilize dynamic
unit files; When the daemon starts up, it reads configuration which may
be static, or generated prior to the daemon starting.

What service units and deps are generated from netplan generate?

network.target.wants systemd-networkd.service|socket, systemd-networkd-online, NetworkManager.service & its wait-online, wpa-supplicant "WiFi" units, OVS service units to establish OVS and configure bridges & fakevlans, or cleanup old OVS state (as itself is persistent across reboots).

Is this an OVS specific implementation detail of netplan?

No. Any and all renderings of netplan require units to be enabled always. So far we have worked around that by baking in networkd or NM as enabled on server/desktop images.

Networkd and NetworkManager expect to be enabled; just like networking.service
would be expected to be enabled by default.

On service start, the unit can be skipped if there aren't any configuration
files present, and the design of the wait-online units are similar in that if
no interfaces are configured, it will not wait on anything.

So that doesn't sound like a workaround (always enabled), but requirement
for using the service.

Are the OVS/WPA daemons not designed to read config?

Could netplan generate render hooks for network-dispatcher which could invoke the newly generated systemd units at the right time?

That would be too late for some of them. And not guaranteed to complete before we start executing cloud-config / cloud-final stages of cloud-init at which point WiFi/OVS must already be up, running and configured.

OK

netplan-[ovs|wpa]-*.service

Instead of generating custom service files (tell me if I'm incorrect about the content being dynamic); could those services read configuration files that netplan generate writes? In the case that netplan generate has no ovs or wpa content, those services are inactive; ConditionPath=/run/somepath/netplan-generate/writes/{wpa,ovs}?

That would loose granularity though. OVS rejecting setting up one bridge, will mean the rest of bridges are not attempted to be configured. For WPA, we do need a long running Daemon per WiFi network, as it has to continuously renew beacons & roam between APs. And it wouldn't help with starting NetworkManager or networkd.

I'm not following; How are OVS/WPA different than networkd/NM where the
daemon runs and on reload it rereads the on-filesystem config?

I'm suggesting to not write configuration for a network into units. Instead
write configuration files that a single daemon (enabled always like
networkd/NM are). Then there's no need to daemon-reload for dynamic targets.

@raharper alternative to this, is to re-engineer how cloud-init targets work a bit. Instead of booting to default.target, divert the boot to cloud-local.target, fully complete booting to that (no jobs left), then call systemctl daemon-reload, then start default.target with new dependencies calculated. Doing that would allow users to do interesting things with systemd via cloud-config. Like changing the default.target from multiuser.target to emergency.target, adding / masking / removing units used in early boot, and "just write fstab" and allow systemd-fstab-generator to process it, and mount things, etc. Or systemd should be fixed to allow dynamically picking up new targets & units always, like upstart did....

This is certainly an interesting topic; I'm not sure we should depend on this
sort of change to deliver OVS/WPA for first boot though.

@slyon
Copy link
Collaborator Author

slyon commented Aug 13, 2020

therefore the new service units and its dependencies, which are generated by manually calling netplan generate are ignored during the first-boot transaction.

This is the core issue. Neither networkd nor NetworkManager utilize dynamic
unit files; When the daemon starts up, it reads configuration which may
be static, or generated prior to the daemon starting.

Yes. This is the core issue indeed.
But I'd argue that generating dynamic unit files is all about having a systemd generator (like netplan). And yes this is different from networkd or NetworkManager, as those are long running system daemons and not generators. Therefore, IMHO the proper fix to this issue would be cloud-init using the netplan generator as intended (i.e. as a systemd generator at the correct time/boot stage) and not calling it during a already running boot transaction.

Are the OVS/WPA daemons not designed to read config?

I'm not following; How are OVS/WPA different than networkd/NM where the
daemon runs and on reload it rereads the on-filesystem config?

I'm suggesting to not write configuration for a network into units. Instead
write configuration files that a single daemon (enabled always like
networkd/NM are). Then there's no need to daemon-reload for dynamic targets.

It depends. OVS for example is using a database (ovsdb) in the background where it stores (and restores at boot time) its configuration. This database is meant to be modified at runtime using commands like ovs-vsctl .... So it's not always possible to write configuration files on disk.

@raharper alternative to this, is to re-engineer how cloud-init targets work a bit. Instead of booting to default.target, divert the boot to cloud-local.target, fully complete booting to that (no jobs left), then call systemctl daemon-reload, then start default.target with new dependencies calculated. Doing that would allow users to do interesting things with systemd via cloud-config. Like changing the default.target from multiuser.target to emergency.target, adding / masking / removing units used in early boot, and "just write fstab" and allow systemd-fstab-generator to process it, and mount things, etc. Or systemd should be fixed to allow dynamically picking up new targets & units always, like upstart did....

This is certainly an interesting topic; I'm not sure we should depend on this
sort of change to deliver OVS/WPA for first boot though.

IMO this should be the preferred solution here as it solves the problem in a clean way without needing special handling in netplan or backports of systemd features. We should consider if we can wait for this to be implemented, or if we need to integrate some kind of workaround to fix the issue temporarily right now (i.e. the code of this PR).

@raharper
Copy link
Collaborator

IMHO the proper fix to this issue would be cloud-init using the
netplan generator as intended (i.e. as a systemd generator at the correct
time/boot stage) and not calling it during a already running boot
transaction.

cloud-init is calling netplan generator exactly at the right time: after
aquiring network config that wasn't present when the generator previously
ran.

It seems that at some point netplan generate grew emitting units rather
than just configs for the backends. I suspect there wasn't a direct overlap
between this ability in netplan and cloud-init based images such that this
has been a gap for sometime but with little to no intersection of use-cases.

Are the OVS/WPA daemons not designed to read config?
I'm not following; How are OVS/WPA different than networkd/NM where the
daemon runs and on reload it rereads the on-filesystem config?
I'm suggesting to not write configuration for a network into units. Instead
write configuration files that a single daemon (enabled always like
networkd/NM are). Then there's no need to daemon-reload for dynamic targets.

It depends. OVS for example is using a database (ovsdb) in the background where it stores (and restores at boot time) its configuration. This database is meant to be modified at runtime using commands like ovs-vsctl .... So it's not always possible to write configuration files on disk.

When reviewing the ovs/netplan design 8 months ago, I raised this issue of
writing ovs commands into unit files as something we should avoid. I spoke
with a few folks about having an ovsctld or something that would read a
configuration file and interface with ovs rather than unit file execution.

It's unfortunate that we've an implementation and now we're sorting out how
to integrate it into existing working systems.

@raharper alternative to this, is to re-engineer how cloud-init targets work a bit. Instead of booting to default.target, divert the boot to cloud-local.target, fully complete booting to that (no jobs left), then call systemctl daemon-reload, then start default.target with new dependencies calculated. Doing that would allow users to do interesting things with systemd via cloud-config. Like changing the default.target from multiuser.target to emergency.target, adding / masking / removing units used in early boot, and "just write fstab" and allow systemd-fstab-generator to process it, and mount things, etc. Or systemd should be fixed to allow dynamically picking up new targets & units always, like upstart did....

This is certainly an interesting topic; I'm not sure we should depend on this
sort of change to deliver OVS/WPA for first boot though.

IMO this should be the preferred solution here as it solves the problem in a
clean way without needing special handling in netplan or backports of
systemd features. We should consider if we can wait for this to be
implemented, or if we need to integrate some kind of workaround to fix the
issue temporarily right now (i.e. the code of this PR).

I'm very much wary of suggesting additional systemd target/unit changes that
will need to be SRU'ed to every running cloud instance that's running
cloud-init today so that this ovs feature can work. None of these instances
will have netplan with ovs config present but all of these will have risk
if we make these sorts of changes.

How do we scope the changes to minimize any impact?

@slyon
Copy link
Collaborator Author

slyon commented Aug 25, 2020

I opened a discussion / bug report at cloud-init to discuss the "staged boot" targets:
https://bugs.launchpad.net/cloud-init/+bug/1892851

And I'd like to reject this PR in favor of a this "staged boot".

@slyon slyon added the abandoned This PR has been abandoned (will not be merged, might need work) label Aug 28, 2020
@xnox
Copy link
Contributor

xnox commented Sep 7, 2020

I opened a discussion / bug report at cloud-init to discuss the "staged boot" targets:
https://bugs.launchpad.net/cloud-init/+bug/1892851

And I'd like to reject this PR in favor of a this "staged boot".

However, as discussed on irc and in the cloud-init bug report, imposing and changing boot of all systems ever to use multiple-transactions, always, is a far bigger change too.

It would be interesting, if we can still do just in time loading of this unit, in a more opportunistic manner. Such that, it only ever is generated midflight and loaded when needed. With most of the time being a noopt.

@xnox
Copy link
Contributor

xnox commented Sep 7, 2020

To minimize risks/changes, we could do this:

  • make netplan-generator to be sensistive to is-system-running starting / SYSTEMD_UNIT set
  • thus only generate netplan.target deps when we are booting, and things have changed
  • and ensure that netplan generate itself starts the netplan target during boot, without any changes to cloud-init

This would make this PR be a no-opt for most instances;
would not require any cloud-init changes;
would still need versioned dependency on the backported systemd, or "just work" in groovy.

@slyon
Copy link
Collaborator Author

slyon commented Sep 18, 2020

Fixed in #162

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
abandoned This PR has been abandoned (will not be merged, might need work)
Projects
None yet
3 participants