New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lib: Use Nix's static scope checking, fix error message, optimize #101139
Conversation
044957f
to
b101812
Compare
Ironically my effort to select the attributes from the right Those inherits should be good now. If anyone is concerned if ofborg and me missed one, we could |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great success!
Ofborg is green without any rebuilds, and I checked the import lists with the static mode, so it looks like this is a no-op (apart from the new symbols exposed).
inherit (lib) | ||
isInt | ||
attrNames | ||
isList | ||
isAttrs | ||
substring | ||
addErrorContext | ||
attrValues | ||
concatLists | ||
concatStringsSep | ||
const | ||
elem | ||
generators | ||
head | ||
id | ||
isDerivation | ||
isFunction | ||
mapAttrs | ||
trace; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn’t importing from lib
instead of builtins
introduce another indirection? Sounds to me like we can get another small speedup by being explicit about the builtins; otoh that means we need to be careful to not import stuff that’s not supported in old nix versions.
Maybe worth investigating whether there’s a small evaluation gain there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had the same thought, but lib
s top-level attribute set is only evaluated once, so the overhead of this indirection amounts to a single lookup. You really need the repeated evaluation of certain expressions to make this optimization effective.
I'd be in favor of adding more builtins to lib
for consistency and convenience. I've only done it for ones I got wrong writing the explicit inherits, which, although admittedly a bit arbitrarily, keeps the scope of this PR in check.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let’s do it in a followup then.
lib/modules.nix
Outdated
let | ||
inherit (lib.attrsets) | ||
mapAttrsRecursiveCond | ||
; | ||
inherit (lib.lists) | ||
any all concatLists concatMap | ||
count filter findFirst foldl foldl' | ||
head imap1 length optional | ||
reverseList sort | ||
; | ||
inherit (lib.options) | ||
isOption | ||
mkOption | ||
showDefs | ||
showFiles | ||
showOption | ||
unknownModule | ||
; | ||
inherit (lib.attrsets) | ||
attrByPath | ||
attrNames | ||
catAttrs | ||
getAttrFromPath | ||
mapAttrs | ||
mapAttrsToList | ||
optionalAttrs | ||
recursiveUpdate | ||
setAttrByPath | ||
toList | ||
; | ||
inherit (lib.types) | ||
types | ||
; | ||
inherit (lib.trivial) | ||
flip | ||
id | ||
isBool | ||
isFunction | ||
isString | ||
min | ||
warn | ||
; | ||
inherit (lib.strings) | ||
optionalString | ||
; | ||
inherit (lib) | ||
elem | ||
isAttrs | ||
; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Import list is correct (checked with the nix-mode in emacs)
@@ -616,7 +659,7 @@ rec { | |||
fixupOptionType = loc: opt: | |||
let | |||
options = opt.options or | |||
(throw "Option `${showOption loc'}' has type optionSet but has no option attribute, in ${showFiles opt.declarations}."); | |||
(throw "Option `${showOption loc}' has type optionSet but has no option attribute, in ${showFiles opt.declarations}."); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this the place that was buggy?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly. Scope checking will now find undeclared local variables, even if not used in any tests.
lib/options.nix
Outdated
let | ||
inherit (lib) | ||
isAttrs isBool isDerivation isFunction isInt isList isString | ||
all collect concatMap concatLists elemAt filter foldl' head length mapAttrs optionals optional take | ||
; | ||
inherit (lib.attrsets) optionalAttrs; | ||
inherit (lib.strings) concatMapStrings concatStringsSep; | ||
inherit (lib.types) mkOptionType; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
correct import list
inherit (builtins) | ||
compareVersions | ||
elem | ||
elemAt | ||
filter | ||
fromJSON | ||
head | ||
isInt | ||
isList | ||
isString | ||
match | ||
parseDrvName | ||
readFile | ||
replaceStrings | ||
split | ||
storeDir | ||
stringLength | ||
substring | ||
tail | ||
toJSON | ||
typeOf | ||
unsafeDiscardStringContext | ||
; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
import list correct
concatStringsSep | ||
escapeNixString | ||
isCoercibleToString | ||
; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
import lists correct
@@ -63,7 +63,7 @@ let | |||
deepSeq elem elemAt filter genericClosure genList getAttr | |||
hasAttr head isAttrs isBool isInt isList isString length | |||
lessThan listToAttrs pathExists readFile replaceStrings seq | |||
stringLength sub substring tail; | |||
stringLength sub substring tail trace; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has been bothering me for a while 👍
Some of my comments only make sense in the full diff view, Github is not a good tool for these reviews. |
Nix can perform static scope checking, but whenever code is inside a `with` expression, the analysis breaks down, because it can't know statically what's in the attribute set whose attributes were brought into scope. In those cases, Nix has to assume that everything works out. Except it doesnt. Removing `with` from lib/ revealed an undefined variable in an error message. If that doesn't convince you that we're better off without `with`, I can tell you that this PR results in a 3% evaluation performance improvement because Nix can look up local variables by index. This adds up with applications like the module system. Furthermore, removing `with` makes the binding site of each variable obvious, which helps with comprehension.
Unlike the other three is* functions in lib.trivial, it was only available as lib.trivial.isFloat
This puts it among the trace* family of functions derived from it.
cde5540
to
fe4a58e
Compare
Rebased and recent lib additions tested. |
I appreciate further improvements to |
@@ -270,7 +270,7 @@ rec { | |||
name = "attrs"; | |||
description = "attribute set"; | |||
check = isAttrs; | |||
merge = loc: foldl' (res: def: mergeAttrs res def.value) {}; | |||
merge = loc: foldl' (res: def: res // def.value) {}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mergeAttrs = x: y: x // y
so this is correct
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me!
I've waited a bit because I wasn't going to be available in the past couple of days. No relevant changes were made to |
Evaluation seems broken:
|
I think there was a silent (i.e. semantic) merge conflict between PR NixOS#101139 and PR NixOS#100456. This commit should fix the error, which manifests as follows: error: undefined variable 'boolToString' at /home/kkini/src/nixpkgs/lib/types.nix:552:42
This is coming from cebf919#diff-64168148acd9f2147ef733b1498b8821c24f2e3f32354b0e147dd421d71274f3, which was merged in the meantime |
Apologies for the breakage. |
FWIW, this breaks |
Was fixed. |
Motivation
Nix can perform static scope checking, but whenever code is inside
a
with
expression, the analysis breaks down, because it can'tknow statically what's in the attribute set whose attributes were
brought into scope. In those cases, Nix has to assume that
everything works out.
Except it doesnt. Removing
with
from lib/ revealed an undefinedvariable in an error message.
If that doesn't convince you that we're better off without
with
,I can tell you that this PR results in a 3% evaluation performance
improvement because Nix can look up local variables by index.
This adds up with applications like the module system.
Furthermore, removing
with
makes the binding site of eachvariable obvious, which helps with comprehension.
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)nix path-info -S
before and after)