Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/nixos-option: add `--nesting-limit` option to avoid infinite recursions #79360

Closed
wants to merge 1 commit into from

Conversation

@Ma27
Copy link
Member

@Ma27 Ma27 commented Feb 6, 2020

Motivation for this change

Until now, nixos-option tried to show all sub-attributes of a config
value which can result in a never-ending output e.g. when running
nixos-option nixpkgs.pkgs due to the fixpoint in this attribute set.

By default, nixos-option only steps three levels into a data-structure
(unless --no-nesting-limit is set). The limit can be modified using
--nesting-limit.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
…cursions

Until now, `nixos-option` tried to show all sub-attributes of a config
value which can result in a never-ending output e.g. when running
`nixos-option nixpkgs.pkgs` due to the fixpoint in this attribute set.

By default, `nixos-option` only steps three levels into a data-structure
(unless `--no-nesting-limit` is set). The limit can be modified using
`--nesting-limit`.
@chkno
Copy link
Member

@chkno chkno commented Feb 6, 2020

I see two problems:

  1. 3 is too low of a default nesting limit. Here are some normal config options along with the first nesting limit at which they are fully visible:
4 supportedFeatures    in nix.buildMachines.*
4 groups               in security.sudo.extraRules.*
4 users                in security.sudo.extraRules.*
4 excluded_interfaces  in services.datadog-agent.networkCheck.instances.*
4 modules              in services.xserver.drivers.*
4 encrypted            in swapDevices.*
4 randomEncryption     in swapDevices.*
4 encrypted            in system.build.fileSystems.*
4 options              in system.build.fileSystems.*
5 keys                 in meta.maintainers.*
6 commands             in security.sudo.extraRules.*
6 resources            in services.matrix-synapse.listeners.*

security.sudo.extraRules in particular is really important not to hide by default. The default nesting level would need to be 6 to allow security.sudo.extraRules to be visible.

I suspect setting the nesting limit to 6 wouldn't help with nixpkgs.pkgs though. If a limit of 3 prints nixpkgs.pkgs once -- 10,000 packages -- then presumably a nesting limit of six would print 10,000^4 = 10 quadrillion packages. I didn't actually verify this count because ...

  1. This doesn't prevent recursion into nixpkgs.pkgs down attr-path paths. With this change and with a nixpkgs.pkgs defined, nixos-option -r still gets lost in nixpkgs.pkgs.beam.packages.erlang.beamPackages.beamPackages.beamPackages.beamPackages.beamPackages.beamPackages.beamPackages...

--

I also took a stab at better handling of nixpkgs.pkgs awhile back. Some alternative approaches to consider:

 services.xserver.drivers = [ {
   driverName = "intel";
   modules = [ «derivation /nix/store/...-xf86-video-intel-2018-12-03.drv» ];
-  name = "intel";
+  name = «repeated»;
 } ];

This can be mitigated by only doing this check for attrsets, and then it doesn't break anything.

The other problem is that it didn't actually catch all the fixedpoints. Maybe more careful implementation of this feature could fix this, or maybe it couldn't. (Can the repl printer do it? If yes, then this could also, since it uses the same technique.)

nixpkgs.pkgs = «set with 10856 attributes»;
services.hoogle.haskellPackages = «set with 14421 attributes»;
services.xserver.windowManager.xmonad.haskellPackages = «set with 14421 attributes»;

This actually worked pretty well.

--

I didn't pursue merging these because my attempt to use nixpkgs.pkgs ended in failure. I found that

NixOS and Nixpkgs are distributed together for consistency

Note that using a distinct version of Nixpkgs with NixOS may be an unexpected source of problems. Use this option with care.

in the NixOS manual is true and understated -- I found no way to take a NixOS system that defines nixpkgs.pkgs across the 19.03 → 19.09 upgrade because of the gnome3.corePackagesservices.gnome3.core-shell.enable rename combined with references to these package names in NixOS modules.

I figured that I'd just learned what everyone else already knew -- that nixpkgs.pkgs was a now-unworkable relic of a former era & was just waiting for some well-meaning maintainer to finally come along and remove it. If I learned the wrong lesson here and there is some non-doomed use of nixpkgs.pkgs , then I agree that nixos-option should have good support for working with it.

--

If you end up continuing along this path, consider sticking maxNestingLevel in Context so it doesn't need to be explicitly plumbed through so many other functions (like --large-set-threshold does.)

@chkno
Copy link
Member

@chkno chkno commented Feb 6, 2020

Maybe if printing large attr sets exactly once is what's desired, a combination of attr-set size checking and nesting limit would work: Have the nesting limit be 1, but only increment it when stepping into into huge attr sets. This allows small things like security.sudo.extraRules to be fully printed, but would prevent printing nixpkgs while already inside nixpkgs.

@Ma27
Copy link
Member Author

@Ma27 Ma27 commented Feb 19, 2020

in the NixOS manual is true and understated -- I found no way to take a NixOS system that defines nixpkgs.pkgs across the 19.03 → 19.09 upgrade because of the gnome3.corePackages → services.gnome3.core-shell.enable rename combined with references to these package names in NixOS modules.

You're absolutely right - that's AFAIK a known issue (and one of the main reasons why I'm using nixos-rebuild -I now).

Anyway, thanks a lot for the feedback, I agree that this change only makes sense if you have low recursion limits. Unless you're faster, I'll take a closer look at your suggestions next weekend and file another PR :)

@Ma27 Ma27 closed this Feb 19, 2020
@Ma27 Ma27 deleted the Ma27:nixos-option-recursion branch Feb 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.