Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixops deploy doesn't bring server in desired state because it doesn't start stopped systemd units #1063

Open
nh2 opened this issue Dec 15, 2018 · 4 comments

Comments

Projects
None yet
1 participant
@nh2
Copy link
Contributor

commented Dec 15, 2018

If my nginx shuts down because of some failure in another systemd unit that it depends on, and I fix the issue and deploy with nixops, my nginx isn't actually started.

In general, when I run nixops deploy, the desired state isn't reached if a unit stopped for some reason.

It's only reached with --force-reboot or nixops reboot.

I think this is because nixops deploy doesn't actually systemctl isolate any target like multi-user.target (which is the default target that we put requiredBy on in NixOS service modules if we want it to be started).

I propose that we should probably systemctl isolate multi-user.target during nixops deploy.

Any opposing views?

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2018

In other words, nixops deploy is not congruent in the terminology of https://blog.flyingcircus.io/2016/05/06/thoughts-on-systems-management-methods/.

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2019

Here's some more info on how things work right now:

https://github.com/NixOS/nixpkgs/blob/542ef2b182dff9756abf782a650f80599c515e4a/nixos/modules/system/activation/switch-to-configuration.pl#L69

getActiveUnits uses systemctl list-units --full --no-legend.

After a successful systemctl activate multi-user.target, that looks like this:

...
systemd-udevd-kernel.socket                                                              loaded active running   udev Kernel Socket                                                
basic.target                                                                             loaded active active    Basic System                                                      
encrypted-links.target                                                                   loaded active active    All Encrypted Links                                               
getty.target                                                                             loaded active active    Login Prompts                                                     
local-fs-pre.target                                                                      loaded active active    Local File Systems (Pre)                                          
local-fs.target                                                                          loaded active active    Local File Systems                                                
multi-user.target                                                                        loaded active active    Multi-User System                                                 
network-interfaces.target                                                                loaded active active    All Network Interfaces (deprecated)                               
network-online.target                                                                    loaded active active    Network is Online                                                 
network-pre.target                                                                       loaded active active    Network (Pre)                                                     
network.target                                                                           loaded active active    Network                                                           
nss-lookup.target                                                                        loaded active active    Host and Network Name Lookups                                     
...

and we can see:

# systemctl status multi-user.target
● multi-user.target - Multi-User System
   Loaded: loaded (/nix/store/3hmpbbcv1db42m9g34c9g4q6qinw50x4-systemd-237/example/systemd/system/multi-user.target; linked; vendor preset: enabled)
   Active: active since Thu 2019-01-10 21:00:22 UTC; 1min 15s ago
     Docs: man:systemd.special(7)

But as soon as one service having RequiredBy = [ "multi-user.target" ] stops, the multi-user.target is no longer active. For example, if I systemctl stop myservice then it looks like this:

# systemctl status multi-user.target
● multi-user.target - Multi-User System
   Loaded: loaded (/nix/store/3hmpbbcv1db42m9g34c9g4q6qinw50x4-systemd-237/example/systemd/system/multi-user.target; linked; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-01-10 21:02:27 UTC; 2s ago
     Docs: man:systemd.special(7)

That also makes it disappear from the systemctl list-units --full --no-legend list. Consequently, the bit in https://github.com/NixOS/nixpkgs/blob/542ef2b182dff9756abf782a650f80599c515e4a/nixos/modules/system/activation/switch-to-configuration.pl#L174-L177

is not executed and multi-user.target is not started by switch-to-configuration (which nixops calls).

So, if any service stopped (thus stopping multi-user.target), then nixops deploy will currently not start any declared units, and thus it's not congruent.

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented Jan 10, 2019

A remaining question is whether it should be nixops or switch-to-configuration that is to be made congruent.

nh2 added a commit to nh2/nixops that referenced this issue Jan 11, 2019

Start multi-user.target on deploy. Fixes NixOS#1063.
This makes `nixops deploy` congruent (the declared services
will always be running after deploy, no matter what the state
of the server was before the deploy).

@nh2 nh2 referenced a pull request that will close this issue Jan 11, 2019

Open

Start multi-user.target on deploy #1078

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented Jan 11, 2019

PR at #1078

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.