Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NetworkManager-wait-online.service is broken #59603

Closed
M-Gregoire opened this issue Apr 15, 2019 · 8 comments
Closed

NetworkManager-wait-online.service is broken #59603

M-Gregoire opened this issue Apr 15, 2019 · 8 comments
Labels
0.kind: bug 9.needs: port to stable A PR needs a backport to the stable release.

Comments

@M-Gregoire
Copy link
Member

Issue description

NetworkManager service has been renamed recently but it seems NetworkManager-wait-online.service hasn't been updated so it tries to uses NetworkManager.service instead of network-manager.service.

Steps to reproduce

  1. Install network-manager
  2. Run sudo systemctl start NetworkManager-wait-online.service
  3. Watch it fail with:
Failed to start NetworkManager-wait-online.service: Unit NetworkManager.service not found.

Technical details

 - system: `"x86_64-linux"`
 - host os: `Linux 5.0.7, NixOS, 19.03.git.ea497998e4b (Koi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2`
 - channels(root): `"nixos-18.09.2505.46d3867a08a"`
 - nixpkgs: `<config-path>/vendor/nixpkgs-release`
@hedning
Copy link
Contributor

hedning commented Apr 15, 2019

The service was renamed to NetworkManager.service which is what upstream expects, but the change isn't in 19.03, we should probably backport it.

We don't modify NetworkManager-wait-online.service in NixOS so it looks for the standard service name (NetworkManager). I'm guessing this was always broken.

@worldofpeace
Copy link
Contributor

Backport already would feel weird with #57938

@flokli
Copy link
Contributor

flokli commented Apr 15, 2019

Yeah, #51382 wasn't really tested sufficiently enough to backport it to 19.03.

We could fix NetworkManager-wait-online.service by changing the unit name it looks for in nm-online for 19.03, though (without changing anything in master)

@jtojnar jtojnar added 0.kind: bug 9.needs: port to stable A PR needs a backport to the stable release. labels Apr 17, 2019
fricklerhandwerk added a commit to fricklerhandwerk/settings that referenced this issue Jun 26, 2019
@fricklerhandwerk
require network connection for home-config clone
9a76262 
waiting for the network to be up with `network-online.target` will only work
with a NixOS 18.09 build after this commit:
<NixOS/nixpkgs@022eb6a>

you have to build your own image from that, as currently the latest 18.09
release binary is more than two months older. the issue behind this is
essentially inconsistent service names:
<NixOS/nixpkgs#59603>

here is how to build an image:
<https://nixos.org/nixos/manual/index.html#sec-building-cd>

this is what you get if you want to use immature software...
fricklerhandwerk added a commit to fricklerhandwerk/settings that referenced this issue Jun 26, 2019
@fricklerhandwerk
require network connection for home-config clone
d6a2f8a 
the general idea is described here:
<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

unfortunately waiting for the network to be up with
`network-online.target` will not work because of inconsistent service names:
<NixOS/nixpkgs#59603>

it will only work with a NixOS 18.09 build after this commit:
<NixOS/nixpkgs@022eb6a>

it seems that 19.03 still is not fixed in that regard:
<NixOS/nixpkgs#59603>

as we want the service to run immediately after bootstrapping (which is
the whole point of this project), you have to build your own image, as
currently the latest 18.09 release binary does not incorporate said
patch.

here is how to build an image:
<https://nixos.org/nixos/manual/index.html#sec-building-cd>

even more unfortunately the build will fail due to a deprecated kernel
API used by ZFS, used for the NixOS minimal image:
<NixOS/nixpkgs#60907>

    git clone https://github.com/NixOS/nixpkgs \
    # fix is on 18.09, and the repository is huge
    --single-branch --branch release-18.09 --depth 200
    cd nixpkgs
    # commit 44c66417f4e05f0a211735f024b58bdb9770108f
    # Author: Tim Steinbach <tim@nequissimus.com>
    # Date:   Thu May 2 13:19:13 2019 -0400
    #
    #     linux: 4.19.37 -> 4.19.38
    #
    #     (cherry picked from commit e0bf73dbde74e66167bd9ccf385a1fbc228c4036)
    git checkout 44c66417f4e05f0a211735f024b58bdb9770108f~1
    cd nixpkgs/nixos
    nix-build -A config.system.build.isoImage \
    -I nixos-config=modules/installer/cd-dvd/installation-cd-minimal.nix \
    default.nix --max-jobs auto --cores $(nproc)

this is what you get if you want to use immature software...
fricklerhandwerk added a commit to fricklerhandwerk/settings that referenced this issue Jun 26, 2019
@fricklerhandwerk
require network connection for home-config clone
ee4f95f 
the general idea is described here:
<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

unfortunately waiting for the network to be up with
`network-online.target` will not work because of inconsistent service names:
<NixOS/nixpkgs#59603>

it will only work with a NixOS 18.09 build after this commit:
<NixOS/nixpkgs@022eb6a>

it seems that 19.03 still is not fixed in that regard:
<NixOS/nixpkgs#59603>

as we want the service to run immediately after bootstrapping (which is
the whole point of this project), you have to build your own image, as
currently the latest 18.09 release binary does not incorporate said
patch.

here is how to build an image:
<https://nixos.org/nixos/manual/index.html#sec-building-cd>

even more unfortunately the build will fail due to a deprecated kernel
API used by ZFS, used for the NixOS minimal image:
<NixOS/nixpkgs#60907>

    git clone https://github.com/NixOS/nixpkgs \
    # fix is on 18.09, and the repository is huge
    --single-branch --branch release-18.09 --depth 200
    # commit 44c66417f4e05f0a211735f024b58bdb9770108f
    # Author: Tim Steinbach <tim@nequissimus.com>
    # Date:   Thu May 2 13:19:13 2019 -0400
    #
    #     linux: 4.19.37 -> 4.19.38
    #
    #     (cherry picked from commit e0bf73dbde74e66167bd9ccf385a1fbc228c4036)
    git checkout 44c66417f4e05f0a211735f024b58bdb9770108f~1
    cd nixpkgs/nixos
    nix-build -A config.system.build.isoImage \
    -I nixos-config=nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix \
    nixpkgs/nixos/default.nix --max-jobs auto --cores $(nproc)

this is what you get if you want to use immature software...
@fricklerhandwerk
Copy link
Contributor

fricklerhandwerk commented Jun 26, 2019 •
edited

This is fixed on master: 4588661

If you cannot wait for this to get into a release, just add the same lines to your configuration.nix.

fricklerhandwerk added a commit to fricklerhandwerk/settings that referenced this issue Jun 26, 2019
@fricklerhandwerk
require network connection for home-config clone
5864098 
the general idea is described here:
<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

unfortunately waiting for the network to be up with
`network-online.target` will not work because of inconsistent service names:
<NixOS/nixpkgs#59603>

it seems that NixOS 19.03 still is not fixed in that regard:
<NixOS/nixpkgs#59603>

here one could just add these lines to the config as workaround:

    systemd.services.NetworkManager-wait-online = {
      wantedBy = [ "network-online.target" ];
    };

on NixOS 18.09 it will only work after this commit:
<NixOS/nixpkgs@022eb6a>

as we want the service to run immediately after bootstrapping (which is
the whole point of this project), you have to build your own image for
that, as currently the latest 18.09 release binary does not incorporate
above patch.

here is how to build an image:
<https://nixos.org/nixos/manual/index.html#sec-building-cd>

even more unfortunately building `release-18.09` branch will fail due to
a deprecated kernel API used by ZFS, used for the NixOS minimal image:
<NixOS/nixpkgs#60907>

    git clone https://github.com/NixOS/nixpkgs \
    # fix is on 18.09, and the repository is huge
    --single-branch --branch release-18.09 --depth 200
    # commit 44c66417f4e05f0a211735f024b58bdb9770108f
    # Author: Tim Steinbach <tim@nequissimus.com>
    # Date:   Thu May 2 13:19:13 2019 -0400
    #
    #     linux: 4.19.37 -> 4.19.38
    #
    #     (cherry picked from commit e0bf73dbde74e66167bd9ccf385a1fbc228c4036)
    git checkout 44c66417f4e05f0a211735f024b58bdb9770108f~1
    cd nixpkgs/nixos
    nix-build -A config.system.build.isoImage \
    -I nixos-config=nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix \
    nixpkgs/nixos/default.nix --max-jobs auto --cores $(nproc)

this is what you get if you want to use immature software...
fricklerhandwerk added a commit to fricklerhandwerk/settings that referenced this issue Jun 26, 2019
@fricklerhandwerk
require network connection for home-config clone
4930ab1 
the general idea is described here:
<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

unfortunately waiting for the network to be up with
`network-online.target` will not work because of inconsistent service names:
<NixOS/nixpkgs#59603>

it seems that NixOS 19.03 still is not fixed in that regard:
<NixOS/nixpkgs#59603>

on NixOS 18.09 it will only work after this commit:
<NixOS/nixpkgs@022eb6a>

as we want the service to run immediately after bootstrapping (which is
the whole point of this project), you have to build your own image for
that, as currently the latest 18.09 release binary does not incorporate
above patch.

here is how to build an image:
<https://nixos.org/nixos/manual/index.html#sec-building-cd>

even more unfortunately building `release-18.09` branch will fail due to
a deprecated kernel API used by ZFS, used for the NixOS minimal image:
<NixOS/nixpkgs#60907>

    git clone https://github.com/NixOS/nixpkgs \
    # fix is on 18.09, and the repository is huge
    --single-branch --branch release-18.09 --depth 200
    # commit 44c66417f4e05f0a211735f024b58bdb9770108f
    # Author: Tim Steinbach <tim@nequissimus.com>
    # Date:   Thu May 2 13:19:13 2019 -0400
    #
    #     linux: 4.19.37 -> 4.19.38
    #
    #     (cherry picked from commit e0bf73dbde74e66167bd9ccf385a1fbc228c4036)
    git checkout 44c66417f4e05f0a211735f024b58bdb9770108f~1
    cd nixpkgs/nixos
    nix-build -A config.system.build.isoImage \
    -I nixos-config=nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix \
    nixpkgs/nixos/default.nix --max-jobs auto --cores $(nproc)

in both cases you have to add these lines to the config:

    systemd.services.NetworkManager-wait-online = {
      wantedBy = [ "network-online.target" ];
    };

which corresponds to a patch on `master`:
<NixOS/nixpkgs@4588661>

this is what you get if you want to use immature software...
@M-Gregoire
Copy link
Member Author

Thank you for the heads up!

fricklerhandwerk added a commit to fricklerhandwerk/settings that referenced this issue Jul 4, 2019
@fricklerhandwerk
require network connection for home-config clone
ec83b8c 
the general idea is described here:
<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

note that all machines also need `network-manager` for this to work.
otherwise the appropriate `systemd` unit
`NetworkManager-wait-online.service` will be broken (even if it exists,
which is really confusing).

unfortunately waiting for the network to be up with
`network-online.target` will not work because of inconsistent service names:
<NixOS/nixpkgs#59603>

it seems that NixOS 19.03 still is not fixed in that regard:
<NixOS/nixpkgs#59603>

on NixOS 18.09 it will only work after this commit:
<NixOS/nixpkgs@022eb6a>

as we want the service to run immediately after bootstrapping (which is
the whole point of this project), you have to build your own image for
that, as currently the latest 18.09 release binary does not incorporate
above patch.

here is how to build an image:
<https://nixos.org/nixos/manual/index.html#sec-building-cd>

even more unfortunately building `release-18.09` branch will fail due to
a deprecated kernel API used by ZFS, used for the NixOS minimal image:
<NixOS/nixpkgs#60907>

    git clone https://github.com/NixOS/nixpkgs \
    # fix is on 18.09, and the repository is huge
    --single-branch --branch release-18.09 --depth 200
    # commit 44c66417f4e05f0a211735f024b58bdb9770108f
    # Author: Tim Steinbach <tim@nequissimus.com>
    # Date:   Thu May 2 13:19:13 2019 -0400
    #
    #     linux: 4.19.37 -> 4.19.38
    #
    #     (cherry picked from commit e0bf73dbde74e66167bd9ccf385a1fbc228c4036)
    git checkout 44c66417f4e05f0a211735f024b58bdb9770108f~1
    cd nixpkgs/nixos
    nix-build -A config.system.build.isoImage \
    -I nixos-config=nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix \
    nixpkgs/nixos/default.nix --max-jobs auto --cores $(nproc)

in both cases you have to add these lines to the config:

    systemd.services.NetworkManager-wait-online = {
      wantedBy = [ "network-online.target" ];
    };

which corresponds to a patch on `master`:
<NixOS/nixpkgs@4588661>

this is what you get if you want to use immature software...
@sgraf812
Copy link
Contributor

I've recently had some trouble with NetworkManager-wait-online.service timing out after a nixos-rebuild switch.

It turns out that NM was enabled by Gnome for me, so I didn't bother to enable it explicitly in my configuration.nix. After I added networking.networkmanager.enable = true;, the issue went away.

I know that is not strictly the topic of the OP, but the first google result for nixos "Failed to start NetworkManager-wait-online.service" leads here, so I guess it's a good place to share my fix in case anyone else has noticed.

@blaggacao
Copy link
Contributor

networking.networkmanager.enable = true is what enables collaterally NetworkManager-wait-online.service. So what might have looked like causality appears to be just correlation.

systemd.services.NetworkManager-wait-online.enable = false would disable waiting for a network available during boot. Besides that waiting should probably have a timeout...

blaggacao added a commit to blaggacao/nixpkgs that referenced this issue Nov 7, 2022
@blaggacao
mod: add reasonable timeout to nm-wait or users get stuck
70a07ca 
e.g. NixOS#59603 (comment)
@sgraf812
Copy link
Contributor

I think since about 22.11 my accidental fix in #59603 (comment) doesn't work anymore. systemd.services.NetworkManager-wait-online.enable = false as suggested in #59603 (comment) does work around the bug, though it's probably not a satisying solution either. Perhaps it helps until #180175 is fixed, which appears to be caused upstream in systemd. I'll continue discussion over there as it seems like the more appropriate place compared to a closed issue.

fricklerhandwerk added a commit to fricklerhandwerk/settings that referenced this issue Jun 17, 2023
@fricklerhandwerk
require network connection for home-config clone
cafeb78 
the general idea is described here:
<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

note that all machines also need `network-manager` for this to work.
otherwise the appropriate `systemd` unit
`NetworkManager-wait-online.service` will be broken (even if it exists,
which is really confusing).

unfortunately waiting for the network to be up with
`network-online.target` will not work because of inconsistent service names:
<NixOS/nixpkgs#59603>

it seems that NixOS 19.03 still is not fixed in that regard:
<NixOS/nixpkgs#59603>

on NixOS 18.09 it will only work after this commit:
<NixOS/nixpkgs@022eb6a>

as we want the service to run immediately after bootstrapping (which is
the whole point of this project), you have to build your own image for
that, as currently the latest 18.09 release binary does not incorporate
above patch.

here is how to build an image:
<https://nixos.org/nixos/manual/index.html#sec-building-cd>

even more unfortunately building `release-18.09` branch will fail due to
a deprecated kernel API used by ZFS, used for the NixOS minimal image:
<NixOS/nixpkgs#60907>

    git clone https://github.com/NixOS/nixpkgs \
    # fix is on 18.09, and the repository is huge
    --single-branch --branch release-18.09 --depth 200
    # commit 44c66417f4e05f0a211735f024b58bdb9770108f
    # Author: Tim Steinbach <tim@nequissimus.com>
    # Date:   Thu May 2 13:19:13 2019 -0400
    #
    #     linux: 4.19.37 -> 4.19.38
    #
    #     (cherry picked from commit e0bf73dbde74e66167bd9ccf385a1fbc228c4036)
    git checkout 44c66417f4e05f0a211735f024b58bdb9770108f~1
    cd nixpkgs/nixos
    nix-build -A config.system.build.isoImage \
    -I nixos-config=nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix \
    nixpkgs/nixos/default.nix --max-jobs auto --cores $(nproc)

in both cases you have to add these lines to the config:

    systemd.services.NetworkManager-wait-online = {
      wantedBy = [ "network-online.target" ];
    };

which corresponds to a patch on `master`:
<NixOS/nixpkgs@4588661>

this is what you get if you want to use immature software...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0.kind: bug 9.needs: port to stable A PR needs a backport to the stable release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@hedning @flokli @jtojnar @sgraf812 @fricklerhandwerk @blaggacao @M-Gregoire @worldofpeace