Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nginx: expose generated config and allow nginx reloads #57429

Merged
merged 5 commits into from Aug 21, 2019

Conversation

@danbst
Copy link
Contributor

commented Mar 11, 2019

Motivation for this change

Fixes: #15906
Another try was done, but not yet merged in #24476

This add 2 new features: ability to review generated Nginx config
(and NixOS has sophisticated generation!) and reloading
of nginx on config changes. This preserves nginx restart on package
updates.

I've modified nginx test to use this new feature and check reload/restart
behavior.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Fixes: #15906
Another try was done, but not yet merged in #24476

This add 2 new features: ability to review generated Nginx config
(and NixOS has sophisticated generation!) and reloading
of nginx on config changes. This preserves nginx restart on package
updates.

I've modified nginx test to use this new feature and check reload/restart
behavior.
@roberth

This comment has been minimized.

Copy link
Contributor

commented Mar 17, 2019

I don't care much about exposing a config file, so my eyes would skip past this option. In a way it's always exposed, through systemctl show, for example, or using nix repl or probably just nixos-option.
So I think the reloading behavior is more relevant than exposed or not. Perhaps the option should be enableReload. Also I'd like the default to be true, although that might break someone's workflow. Either way, this should be in the changelog.

@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Mar 17, 2019

BTW it is not exposed in nix repl, configFile is a private let-binding when making virtual hosts via module system. And systemctl show isn't very fun for debugging...

But I agree this should be default! I just don't want to push this until there is some decision over #24476. I can rename option to enableReload if this will be chosen as a way to go.

cc @fpletz @Infinisil @fadenb @bobvanderlinden

@roberth

This comment has been minimized.

Copy link
Contributor

commented Mar 17, 2019

I guess I was getting philosophical about what expose means :)

Perhaps the rename can be done, and the change of default behavior can wait until a decision is made?

Copy link
Contributor

left a comment

This needs to move forward :(. The current implementation feels broken and we have had quite a few PRs proposed, all 'died' because we couldn't come up with a solution where everyone agrees on all aspects.

This PR looks good. I like the tests 👍.

The name exposeConfig doesn't seem to match the intent, but we might want to split exposeConfig from enableReload and assert enableReload -> exposeConfig. That way a user can explicitly set whether to reload/restart in addition to explicitly defining whether to 'expose' the config.

Another suggestion for the name exposeConfig: configInEtc.

As for the enableReload suggestion, how about reloadIfChanged, inheriting the naming from the systemd configuration?

Lastly, the default should be true to lessen the number of 'wtfs'. But for backwards compatibility it is better to have it on false. Is this a use-case where relying on stateVersion is an option?

You've got my approval. All my suggestions are optional and may be ignored if it helps moving this forward. I feel this PR is at least a step in the right direction, if not the solution to the issue.

@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Apr 13, 2019

@bobvanderlinden unfortunately reloadIfChanged isn't correct name, as service can be restarted instead of reloaded. So

services.nginx.exposeConfig = true;
systemd.services.nginx.reloadIfChanged = false;

and

services.nginx.exposeConfig = true;
systemd.services.nginx.reloadIfChanged = true;

have different behavior. In both cases config will be reloaded, but in second case service will never be restarted.

I also think about making exposeConfig internal and true by default. Not sure people like such breaking changes, but I hope most will be fine with that. @fpletz @Mic92 ?

Is this a use-case where relying on stateVersion is an option?

no, it was stated a few times already that stateVersion is not schemaVersion.

@nixos-discourse

This comment has been minimized.

Copy link

commented Apr 19, 2019

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/20

@wmertens

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

I'm not sure if I am a fan of the expose config - on the one hand, it's nice to see a system-global configuration in /etc (read-only of course), but on the other hand it hides complexity and will make /etc a cesspit if everybody starts doing that.

The rest of the PR is great. Would be cool if you could also do a config check via nginx during the build. I made something similar for Apache but it was problematic because it didn't work if the apache user didn't exist yet.

@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Jun 16, 2019

@wmertens This is more of pragmatic approach. Configuring Nginx via Nix language is fun and nice, but there is a big problem that one can't see the generated result. I had to look into generated config several times, and it is a pain to do when nginx.conf is only in nix store.

The main reason I did this in /etc is because /etc changes are transactional and rollbackable out of box.

Config check is not required for this usecase: if config has problem, reload will fail, and old config will be used by Nginx internally. You'll still get a warning on rebuild-switch that nginx reload failed. But I'm not against it in separate PR

@Infinisil

This comment has been minimized.

Copy link
Member

commented Aug 9, 2019

Nice PR! I'm also for the rename to enableReload and for defaulting it to true. This behavior is preferable for most people by far. We can use the release notes to mention the change in behavior.

An internal exposeConfig option might be nice yeah, but I think this can be done in a separate PR to move this one along fast.

@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Aug 10, 2019

I've rebased this branch on top of master, and test from #48337 stopped working. I'm still investigating into "why". Looks like nginx restart is different from SIGHUP with respect to caching.

@danbst danbst changed the title nginx: expose generated config and allow nginx reloads [WIP] nginx: expose generated config and allow nginx reloads Aug 18, 2019
@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Aug 18, 2019

cc @yorickvP @yegortimoshenko @aszlig

The current blocker for this PR is that ETag test fails when enableReload = true;. The "reload" implies that Nginx is not restarted on configuration change. I would really appreciate if you can dig into this.

@aszlig

This comment has been minimized.

Copy link
Member

commented Aug 18, 2019

@danbst: All the ETag test does is doing a switch-to-configuration to another config (btw. moving that to nesting.clone as well could be a good idea), so I guess if something fails during that, I think it's a bug of this PR, because that's essentially what's happening in nixos-rebuild switch. What's the error you're getting?

nixos/tests/nginx.nix Outdated Show resolved Hide resolved
ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";
Restart = "always";
RestartSec = "10s";
StartLimitInterval = "1min";
};
};

environment.etc."nginx/nginx.conf" = mkIf cfg.enableReload {

This comment has been minimized.

Copy link
@aszlig

aszlig Aug 18, 2019

Member

Hm, why is this only enabled if enableReload is true? Apart from that, I do have mixed feelings about exposing the config in /etc, because I do not want other applications to make certain assumptions about files /etc and generally want to keep as much things out of global paths to ensure they don't behave differently in a Nix build sandbox. That said, I can still mkForce it to an empty attrset, but maybe adding exposeConfigFile with a default of false might be a good idea? Maybe depends on how many users want it to occur in /etc versus how many do not want that behaviour.

This comment has been minimized.

Copy link
@danbst

danbst Aug 18, 2019

Author Contributor

ohhh, the original commit actually called that right 8b036d5#diff-795fa65a7c41a479084de826e4e2e65cR460

It just people suggested that exposeConfig isn't a good name and it should actually be enableReload

Actually, I'd like to write that file in some other place, but only /etc has atomic replaces in NixOS. I don't want to move nginx.conf to runtime dir, as I'd like to have post-mortem (after nginx doesn't start) config checks.

@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Aug 18, 2019

@aszlig

All the ETag test does is doing a switch-to-configuration to another config

yes, but previously this led to nginx restart, and with this PR it leads to SIGHUP, and nginx only reloads it's config. I've tried to debug issue but with no luck.

@aszlig

This comment has been minimized.

Copy link
Member

commented Aug 18, 2019

@GrahamcOfBorg test nginx

@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Aug 18, 2019

@aszlig

subtest: check ETag if serving Nix store paths
webserver: must succeed: curl -v http://localhost/index.html 2>&1 | sed -n -e "s/^< [Ee][Tt][Aa][Gg]: *//p"
webserver: exit status 0
(0.09 seconds)
webserver: must succeed: curl -w "%{http_code}" -X HEAD -H 'If-None-Match: "kiknk38sjy9v04fwamhgg44ndvysncix"' http://localhost/index.html
webserver# Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
webserver# Warning: way you want. Consider using -I/--head instead.
webserver: exit status 0
(0.03 seconds)
webserver: must succeed: /nix/store/9abnqf2zs2af91mla0ix553vyxh7g0lh-nixos-system-newwebserver-19.09.git.deb8cfd/bin/switch-to-configuration test
webserver# [   11.910081] dhcpcd[722]: eth0: Router Advertisement from fe80::2
webserver# [   11.911956] dhcpcd[722]: eth0: adding address fec0::5054:ff:fe12:3456/64
webserver# [   11.914619] dhcpcd[722]: eth0: adding route to fec0::/64
webserver# [   11.917422] dhcpcd[722]: eth0: adding default route via fe80::2
webserver# [   12.590447] nixos[730]: switching to system configuration /nix/store/9abnqf2zs2af91mla0ix553vyxh7g0lh-nixos-system-newwebserver-19.09.git.deb8cfd
webserver# stopping the following units: nginx-config-reload.service, nscd.service
webserver# [   12.615183] systemd[1]: Stopped target Local File Systems.
webserver# [   12.621451] systemd[1]: Stopped target All Network Interfaces (deprecated).
webserver# [   12.626904] systemd[1]: nginx-config-reload.service: Succeeded.
webserver# [   12.631655] systemd[1]: Stopped nginx-config-reload.service.
webserver# [   12.641895] systemd[1]: Stopping Name Service Cache Daemon...
webserver# [   12.654790] systemd[1]: nscd.service: Succeeded.
webserver# [   12.665346] systemd[1]: Stopped Name Service Cache Daemon.
webserver# [   12.669156] systemd[1]: Stopped target Remote File Systems.
webserver# activating the configuration...
webserver# [   13.846253] systemd[1]: Reloading.
webserver# [   14.086708] systemd-fstab-generator[864]: Checking was requested for "store", but it is not a device.
webserver# [   14.092839] systemd-fstab-generator[864]: Checking was requested for "shared", but it is not a device.
webserver# [   14.098623] systemd-fstab-generator[864]: Checking was requested for "xchg", but it is not a device.
webserver# setting up tmpfiles
webserver# restarting the following units: network-addresses-eth1.service
webserver# [   14.411064] systemd[1]: network-local-commands.service: Succeeded.
webserver# [   14.414740] systemd[1]: Stopped Extra networking commands..
webserver# [   14.417935] systemd[1]: Stopping Extra networking commands....
webserver# [   14.424431] systemd[1]: network-setup.service: Succeeded.
webserver# [   14.430591] systemd[1]: Stopped Networking Setup.
webserver# [   14.450544] systemd[1]: Stopping Networking Setup...
webserver# [   14.454923] systemd[1]: Stopping Address configuration of eth1...
webserver# [   14.462867] 9scg4gvwmji8yw8ac73s516w3jwibzjq-unit-script-network-addresses-eth1-pre-stop[874]: /nix/store/9scg4gvwmji8yw8ac73s516w3jwibzjq-unit-script-network-addresses-eth1-pre-stop: line 6: /run/nixos/network/routes/eth1: No such file or directory
webserver# [   14.475282] 9scg4gvwmji8yw8ac73s516w3jwibzjq-unit-script-network-addresses-eth1-pre-stop[874]: deleting address 192.168.1.2/24... done
webserver# [   14.488863] systemd[1]: network-addresses-eth1.service: Succeeded.
webserver# [   14.493069] systemd[1]: Stopped Address configuration of eth1.
webserver# [   14.496286] systemd[1]: Starting Address configuration of eth1...
webserver# [   14.543752] md8b9qyr6r6vjl66br6ap9b5as22fqn9-unit-script-network-addresses-eth1-start[878]: adding address 192.168.1.1/24... done
webserver# [   14.563680] systemd[1]: Started Address configuration of eth1.
webserver# [   14.573148] systemd[1]: Starting Networking Setup...
webserver# starting the following units: nginx-config-reload.service, nscd.service
webserver# [   14.630386] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
webserver# [   14.638203] systemd[1]: Reached target Local File Systems.
webserver# [   14.665714] systemd[1]: Reached target All Network Interfaces (deprecated).
webserver# [   14.672913] systemd[1]: Reached target Remote File Systems.
webserver# [   14.728108] systemd[1]: Condition check resulted in FUSE Control File System being skipped.
webserver# [   14.735574] systemd[1]: Condition check resulted in Kernel Configuration File System being skipped.
webserver# [   14.741661] systemd[1]: Starting Name Service Cache Daemon...
webserver# [   14.752923] systemd[1]: Started nginx-config-reload.service.
webserver# [   14.794703] nscd[897]: 897 monitoring file `/etc/passwd` (1)
webserver# [   14.804899] systemd[1]: Started Networking Setup.
webserver# [   14.814476] nscd[897]: 897 monitoring directory `/etc` (2)
webserver# [   14.825362] systemd[1]: Started Name Service Cache Daemon.
webserver# [   14.829930] nscd[897]: 897 monitoring file `/etc/group` (3)
webserver# [   14.838845] nscd[897]: 897 monitoring directory `/etc` (2)
webserver# [   14.861541] systemd[1]: Starting Extra networking commands....
webserver# [   14.870174] nscd[897]: 897 monitoring file `/etc/hosts` (4)
webserver: exit status 0
(3.25 seconds)
webserver# [   14.873594] systemd[1]: Started Extra networking commands..
webserver# [   14.876806] nscd[897]: 897 monitoring directory `/etc` (2)
webserver: must succeed: curl -v http://localhost/index.html 2>&1 | sed -n -e "s/^< [Ee][Tt][Aa][Gg]: *//p"
webserver# [   14.881266] nscd[897]: 897 monitoring file `/etc/resolv.conf` (5)
webserver# [   14.895692] nscd[897]: 897 monitoring directory `/etc` (2)
webserver# [   14.901782] nscd[897]: 897 monitoring file `/etc/services` (6)
webserver# [   14.906164] systemd[1]: Reloading Nginx Web Server.
webserver# [   14.914249] nscd[897]: 897 monitoring directory `/etc` (2)
webserver# [   14.917200] nscd[897]: 897 disabled inotify-based monitoring for file `/etc/netgroup': No such file or directory
webserver# [   14.926180] nscd[897]: 897 stat failed for file `/etc/netgroup'; will try again later: No such file or directory
webserver# [   14.947036] systemd[1]: Reloaded Nginx Web Server.
webserver# [   14.952421] nixos[730]: finished switching to system configuration /nix/store/9abnqf2zs2af91mla0ix553vyxh7g0lh-nixos-system-newwebserver-19.09.git.deb8cfd
webserver: exit status 0
(0.10 seconds)
webserver: must succeed: curl -w "%{http_code}" -X HEAD -H 'If-None-Match: "kiknk38sjy9v04fwamhgg44ndvysncix"' http://localhost/index.html
webserver# Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
webserver# Warning: way you want. Consider using -I/--head instead.
webserver: exit status 0
(0.03 seconds)
error: Old ETag "kiknk38sjy9v04fwamhgg44ndvysncix" is the same as "kiknk38sjy9v04fwamhgg44ndvysncix" at (eval 21) line 23, <__ANONIO__> line 452.
(3.51 seconds)
@aszlig

This comment has been minimized.

Copy link
Member

commented Aug 18, 2019

@danbst: Hmmm... maybe it's a race condition, so the reload happens after the curl... did you try to do $machine->sleep(10); before the second checkEtag call just to confirm?

@danbst

This comment has been minimized.

Copy link
Contributor Author

commented Aug 18, 2019

@aszlig that was brilliant! Sleep helped indeed!

@aszlig

This comment has been minimized.

Copy link
Member

commented Aug 18, 2019

@danbst: Okay, so we obviously don't want sleeps in test code, so I think we should figure out how to make sure that we properly wait for the reload to finish without sleeping for a fixed timespan.

@aszlig

This comment has been minimized.

Copy link
Member

commented Aug 18, 2019

Eeesh, even nginx itself sleeps X-D

            /* allow new processes to start */
            ngx_msleep(100);
… using `nesting.clone`
@aszlig

This comment has been minimized.

Copy link
Member

commented Aug 18, 2019

@danbst: Okay, it seems we can't easily avoid the sleep, because nginx doesn't have any code for notifying that a reload has finished, which is a bummer... maybe annotate the sleep with an XXX or something like that.

@danbst danbst changed the title [WIP] nginx: expose generated config and allow nginx reloads nginx: expose generated config and allow nginx reloads Aug 18, 2019
@aszlig
aszlig approved these changes Aug 18, 2019
@danbst danbst merged commit 855be67 into NixOS:master Aug 21, 2019
14 checks passed
14 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
nginx on aarch64-linux Success
Details
nginx on x86_64-linux Success
Details
@danbst danbst deleted the danbst:nginx-reload-config branch Aug 21, 2019
restartTriggers = [ configFile ];
script = ''
if ${pkgs.systemd}/bin/systemctl -q is-active nginx.service ; then
${pkgs.systemd}/bin/systemctl reload nginx.service

This comment has been minimized.

Copy link
@Infinisil

Infinisil Aug 21, 2019

Member

This should be config.systemd.package, but I think systemctl is in PATH by default, so it can be just a non-absolute systemctl.

@@ -431,6 +435,16 @@ in
";
};

enableReload = mkOption {
default = false;

This comment has been minimized.

Copy link
@Infinisil

Infinisil Aug 21, 2019

Member

I thought everybody agreed to make this true by default?

This comment has been minimized.

Copy link
@danbst

danbst Aug 21, 2019

Author Contributor

🤔 Maybe I'm just too cautious to change default behavior. Would you like to do that? 😁

@Infinisil

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

(Sorry for the just late review)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.