Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using NetworkManager in the initramfs #394

Closed
dustymabe opened this issue Feb 17, 2020 · 11 comments
Closed

Using NetworkManager in the initramfs #394

dustymabe opened this issue Feb 17, 2020 · 11 comments
Assignees

Comments

@dustymabe
Copy link
Member

@dustymabe dustymabe commented Feb 17, 2020

We had been planning to move to using Networkmanager in the initrd for a while. We now have a host of issues that have cropped up that we think would be solved by making this move.

This is a separate ticket (related to #24) that specifically can track moving to networkmanager in the initrd.

@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Feb 17, 2020

There is an existing WIP PR for this (coreos/fedora-coreos-config#259), but it's blocked on another issue.

@dustymabe dustymabe self-assigned this Feb 27, 2020
@dustymabe dustymabe changed the title Using NetworkManager in the initrd Using NetworkManager in the initramfs Mar 10, 2020
@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 16, 2020

After some experimentation with using NetworkManager in the initramfs it's clear that we want to do a few things to keep our life sane. Here are a few points that are interesting:

  • Networking configuration can be delivered by ignition
  • Ignition configuration may need to be fetched over the network, which requires networking

Considering this information there are several scenarios that we want to support:

  1. user provides networking configuration via Ignition
  2. user provides networking configuration via dracut kernel commandline args

However, we need to think about the different scenarios here and what the use case is for each. Below I'll indicate which ways networking configuration was delivered via Ignition: (yes/no) and dracut: (yes/no).

  • Ignition: no dracut: no
    • In this case most platforms will just use the defaults (dhcp).
    • Some platforms may be able to sideload in configuration from metadata services (DigitalOcean, Packet, etc). The configuration from those services will be propagated into the real root and used persistently.
  • Ignition: no dracut: yes
    • any config provided by the user to dracut for networking will get passed on to the real root and persist for all subsequent boots
  • Ignition: yes dracut: no
    • use ignition provided config
  • Ignition: yes dracut: yes
    • In some cases users may have different requirements for initramfs networking than real root networking (one example could be that the requirements for initramfs networking are much simpler and the requirements for real root networking is more complicated).
    • use ignition provided config, do not pass through any initramfs networking configuration.

More simply put:

if ignition:
     use ignition
else
    if dracut
        pass through dracut to real root
@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 16, 2020

Another side effect of the networking split configuration delivery is that we want the networking in the real root itself to be exactly what is going to be brought up on every subsequent reboot of the machine so we'd like to teardown any networking brought up in the initramfs and have the real root properly bring it all the way up again.

Since we are using NetworkManager being able to do something like nmcli networking off would be really useful here, but currently NetworkManager is not persistently running in the initramfs so nmcli can't talk to it over dbus. In the short term we can write our own bash script to teardown networking manually.

@bgilbert

This comment has been minimized.

Copy link
Member

@bgilbert bgilbert commented Mar 16, 2020

There are platform-specific issues too. DigitalOcean and Packet need to bring up just enough networking (link-local for the former, DHCP for the latter) to fetch the rest of their network config from a metadata service. Historically Afterburn has handled this, though we've discussed moving the functionality into an NM plugin.

@cgwalters

This comment has been minimized.

Copy link
Member

@cgwalters cgwalters commented Mar 16, 2020

Might be pretty easy to run the whole initramfs under the equivalent of unshare --net.

@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 16, 2020

@bgilbert I updated #394 (comment) to mention DigitalOcean and Packet.

@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 16, 2020

Might be pretty easy to run the whole initramfs under the equivalent of unshare --net.

interesting

@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 16, 2020

Since we are using NetworkManager being able to do something like nmcli networking off would be really useful here, but currently NetworkManager is not persistently running in the initramfs so nmcli can't talk to it over dbus. In the short term we can write our own bash script to teardown networking manually.

In a discussion with Thomas Haller it was encouraged that we investigate running NM via systemd in the initrd and he asked me to open an RFE so it can be fleshed out: https://bugzilla.redhat.com/show_bug.cgi?id=1814038

In the short term I'll explore the teardown via script approach.

dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 19, 2020
This is a forward port of coreos-teardown-initramfs-network.service
from the spec2x branch [1] (used for RHEL CoreOS). When moving to NM
in the initrd [2] we decided that we also needed a mechanism to take down
the networking between the initramfs and the real root. While we would
like to use NetworkManager's logic to do this operation in the future
it's currently not easily achieved because NetworkManager is not running
persistently in the initramfs [3].

[1] coreos#78
[2] coreos/fedora-coreos-tracker#394
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1814038
dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 19, 2020
…g if desired

The policy here is:

   - If a networking configuration was provided before this point
     (most likely via Ignition) and exists in the real root then
     we do nothing and don't propagate any initramfs networking.
   - If a user did not provide any networking configuration
     then we'll propagate the initramfs networking configuration
     into the real root.

See coreos/fedora-coreos-tracker#394 (comment)
dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 19, 2020
This is a forward port of coreos-teardown-initramfs-network.service
from the spec2x branch [1] (used for RHEL CoreOS). When moving to NM
in the initrd [2] we decided that we also needed a mechanism to take down
the networking between the initramfs and the real root. While we would
like to use NetworkManager's logic to do this operation in the future
it's currently not easily achieved because NetworkManager is not running
persistently in the initramfs [3].

[1] coreos#78
[2] coreos/fedora-coreos-tracker#394
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1814038
dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 19, 2020
…g if desired

The policy here is:

   - If a networking configuration was provided before this point
     (most likely via Ignition) and exists in the real root then
     we do nothing and don't propagate any initramfs networking.
   - If a user did not provide any networking configuration
     then we'll propagate the initramfs networking configuration
     into the real root.

See coreos/fedora-coreos-tracker#394 (comment)
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Mar 19, 2020
Moving to NetworkManager in the initrd should help us solve some
problems we've been having with Networking. For what we want to
do in Fedora CoreOS doing this right requires the network to be
torn down in the initrd and also possibly propagated forward.

Requires: coreos/ignition-dracut#159
Fixes: coreos/fedora-coreos-tracker#394
@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 23, 2020

Existing PRs that should bring NM into the initrd:

Placing a reminder here to review systemd-resolved after we get those merged: coreos/fedora-coreos-config#288

dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Mar 24, 2020
Moving to NetworkManager in the initrd should help us solve some
problems we've been having with Networking. For what we want to
do in Fedora CoreOS doing this right requires the network to be
torn down in the initrd and also possibly propagated forward.

Requires: coreos/ignition-dracut#159
Fixes: coreos/fedora-coreos-tracker#394
dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 24, 2020
This is a forward port of coreos-teardown-initramfs-network.service
from the spec2x branch [1] (used for RHEL CoreOS). When moving to NM
in the initrd [2] we decided that we also needed a mechanism to take down
the networking between the initramfs and the real root. While we would
like to use NetworkManager's logic to do this operation in the future
it's currently not easily achieved because NetworkManager is not running
persistently in the initramfs [3].

[1] coreos#78
[2] coreos/fedora-coreos-tracker#394
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1814038
dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 24, 2020
…g if desired

The policy here is:

   - If a networking configuration was provided before this point
     (most likely via Ignition) and exists in the real root then
     we do nothing and don't propagate any initramfs networking.
   - If a user did not provide any networking configuration
     then we'll propagate the initramfs networking configuration
     into the real root.

See coreos/fedora-coreos-tracker#394 (comment)
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Mar 24, 2020
Moving to NetworkManager in the initrd should help us solve some
problems we've been having with Networking. For what we want to
do in Fedora CoreOS doing this right requires the network to be
torn down in the initrd and also possibly propagated forward.

Requires: coreos/ignition-dracut#159
Fixes: coreos/fedora-coreos-tracker#394
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Mar 24, 2020
Moving to NetworkManager in the initrd should help us solve some
problems we've been having with Networking. For what we want to
do in Fedora CoreOS doing this right requires the network to be
torn down in the initrd and also possibly propagated forward.

We effectively move *to* NetworkManager in the initrd (the default
in Fedora 31+) by putting back in the nm-initrd-generator, which
we were previously removing.

Requires: coreos/ignition-dracut#159
Fixes: coreos/fedora-coreos-tracker#394
dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 24, 2020
This is a forward port of coreos-teardown-initramfs-network.service
from the spec2x branch [1] (used for RHEL CoreOS). When moving to NM
in the initrd [2] we decided that we also needed a mechanism to take down
the networking between the initramfs and the real root. While we would
like to use NetworkManager's logic to do this operation in the future
it's currently not easily achieved because NetworkManager is not running
persistently in the initramfs [3].

[1] coreos#78
[2] coreos/fedora-coreos-tracker#394
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1814038
dustymabe added a commit to dustymabe/ignition-dracut that referenced this issue Mar 24, 2020
…g if desired

The policy here is:

   - If a networking configuration was provided before this point
     (most likely via Ignition) and exists in the real root then
     we do nothing and don't propagate any initramfs networking.
   - If a user did not provide any networking configuration
     then we'll propagate the initramfs networking configuration
     into the real root.

See coreos/fedora-coreos-tracker#394 (comment)
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Mar 24, 2020
Moving to NetworkManager in the initrd should help us solve some
problems we've been having with Networking. For what we want to
do in Fedora CoreOS doing this right requires the network to be
torn down in the initrd and also possibly propagated forward.

We effectively move *to* NetworkManager in the initrd (the default
in Fedora 31+) by putting back in the nm-initrd-generator, which
we were previously removing.

Requires: coreos/ignition-dracut#159
Fixes: coreos/fedora-coreos-tracker#394
dustymabe added a commit to coreos/fedora-coreos-config that referenced this issue Mar 24, 2020
Moving to NetworkManager in the initrd should help us solve some
problems we've been having with Networking. For what we want to
do in Fedora CoreOS doing this right requires the network to be
torn down in the initrd and also possibly propagated forward.

We effectively move *to* NetworkManager in the initrd (the default
in Fedora 31+) by putting back in the nm-initrd-generator, which
we were previously removing.

Requires: coreos/ignition-dracut#159
Fixes: coreos/fedora-coreos-tracker#394
@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 25, 2020

Another side effect of the networking split configuration delivery is that we want the networking in the real root itself to be exactly what is going to be brought up on every subsequent reboot of the machine so we'd like to teardown any networking brought up in the initramfs and have the real root properly bring it all the way up again.

What I describe(d) above is exactly the problem detailed in #233

@dustymabe

This comment has been minimized.

Copy link
Member Author

@dustymabe dustymabe commented Mar 26, 2020

NetworkManager networking in the initramfs is now available in the latest testing release (31.20200323.2.0), which you can find on the download page.

This was implemented by the following two pull requests:

We believe this change will fix the following outstanding issues:

And also possibly:

Please let us know if you encounter any issues because of this change or if this change did in fact resolve any other issues you were seeing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants
You can’t perform that action at this time.