[RFC] Replace assert with meta.broken
#36229
Labels
0.kind: enhancement
2.status: stale
https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
2.status: work-in-progress
6.topic: portability
General portability concerns, not specific to cross-compilation or a specific platform
9.needs: community feedback
10.rebuild-darwin: 0
10.rebuild-linux: 0
11.by: nixpkgs-member
Comments
|
👍 from me. I think some asserts are true sanity checks about argument compatibility, but the rest should switch to meta. |
|
Is there any reason not to move argument compatibility sanity checks to |
|
Additinally, with #36226 I don't think there is any more reason for |
|
Well, sometimes these have to be extracted from a nested package set (like |
|
I'd say, if the assert only results from user error, and not any arbitrary broken packages situation, then the { withFoo ? foo.meta.available, foo }:
assert withFoo -> foo.meta.available;The assertion respects the default, so it would only fail in the presence of user overrides. And users should in fact only do overrides to make things less enabled. If some optional feature is needed downstream, but it in that downstream package's |
|
So, I want to make sure I understand correctly. |
|
@Ericson2314 from time to time people use some kind of a deep override; it would be nice if a global deep override of What is the benefit of not putting this condition into |
|
👍 on the idea, as this is on SLNOS TODO list too.
SLNOS motivation: derivations should be lazy (#33057) as this gives a lot of nice bonuses.
But I also completely agree with
I think some asserts are true sanity checks about argument compatibility, but the rest should switch to meta.
My motivation: I want to fuzz package builds that depend on package options (use flags).
I want to check that all allowed combinations of flags actually build and (eventually) that all disallowed don't.
The latter would be implemented with tests for flags (or combinations of them). Possible tests include:
- checking that configure log mentions the relevant optional buildInput,
- checking that the buildInput is referenced in the output,
- actually testing the feature it provides.
In general, most `assert`s are either of the two kinds:
- the ones that check the logic behind flag values like `assert gtkSupport -> x11Support || waylandSupport`,
- the ones that check that the relevant packages are not `null`s like `assert gtkSupport -> gtk2 != null`.
I think the second kind should simply not exist (#35906 (comment)). When all derivations are lazy you don't need to `null` them to hide the ones that won't work on a given system, after #33057 you can still access their attributes, hence you don't need to check they are not `null`s anymore. `null`s now just get in the way of debugging actual package dependencies.
Switching the first kind of `assert`s from assert to `broken` is fine for me, but I would prefer we switch to several tiers of those (names are subject to change):
- `meta.makesNoSense` - logically incompatible combination of flags,
- `meta.wontBuild` - building this would fail (should be implied by and eventually equivalent to `makesNoSense`) (this is easy to fuzz),
- `meta.wontRun` - would build, but won't start (ideally, such expressions should fail to build, but there're things that are hard to test in nix, like they require you to run an Xserver or a database) (this can be fuzzed with nixos tests),
- `meta.wontWork` - would build and run, but won't work as you would expect (same thing) (same thing) (maybe this should be squashed with `wontRun`, I'm not yet sure),
- `meta.broken` - broken, but no reason given.
I also would like those to be more verbose than just `bool`s. Say, like `asserts` and `warnings` of NixOS.
Also note that all of this plays really well when you explicitly mark the use flags. Are you still sure you don't want them (#12877. Wink, wink!)?
|
|
Of all things, running X during a build is trivial: https://github.com/7c6f434c/lang-os/blob/master/firefox-profile.nix Whether we want it is another question. USE flags, multiple kinds of brokenness etc. require maintenance effort to continue making sense, which makes achieving buy-in harder… |
|
@oxij look up rust portability lint. Fuzzing would never exhaust space but we could with SAT solver. @rmhnjoj. I don't think so, because a Linux host platform does not always ensure those packages are available. If it did however, that would be good. |
|
The problem is that we are talking about build failures, and that cannot be just fed to SAT solver. The only thing you can hope for is increasing the chance of a problem being found per minute of build. |
|
I think you over-complicate the problem. Most of the time you can see if the `buildInput` list is correct by the `configure` output log. It's super-cheap. So you can reasonably exhaustively check use flags for most of the packages of `nixpkgs`.
For packages with massive amounts of use flags (like `ffmpeg`) fuzzing `*Support` flags one by one would already be uber-helpful and would find 99% of the errors in minutes.
I imagine that to work like this:
- it sets all use flags to `false`, computes `buildInput` (lazy!), runs `configure`, checks the log for references,
- then it overrides flags one by one with `true`, computes new `buildInput`s, runs `configure`, looks at the log, checks that it now uses the new `buildInput`s.
All of that would not work with overrides with `null`s.
To answer #35906 (comment)
`null` provides a way to quickly remove some dependencies without adding extra support into packages.
I agree. But I consider that to be a hack and I think it should be treated as a hack. Normally, things should be put under flags.
Also, note that the composition of use flags and fuzzing makes maintenance burden _less_, not more: you add a new use flag, then you add the corresponding `buildInput`s, ask the fuzzer to fuzz it, get to see whenever the build would be likely to succeed and do what you expect in seconds instead of hours.
|
|
Also, note that the composition of use flags and fuzzing makes maintenance burden _less_, not more: you add a new use flag, then you add the corresponding `buildInput`s, ask the fuzzer to fuzz it, get to see whenever the build would be likely to succeed and do what you expect in seconds instead of hours.
I think we don't have enough acceptance for multiple versions to have people even look at the fuzzer.
|
|
(Posting this as a separate comment to refer back to specifically this. This seems like a good place as any to remind you that all of those ideas I constantly talk here combined produce some awesome benefits.)
Writing this despite the fact that, most likely, unless I frustrate myself to the point at which this thread will become a flamewar with many non-pretty words, this would just get ignored :(
Anyway.
Anybody skeptic of that not having `null`s can be made short and beautiful is requested to proceed to read `pkgs/applications/audio/cmus/default.nix` on master (because it's an exemplary expression of AFAIK the best thing that can be done with master's infrastructure), and then proceed to imagine an expression that has no duplication of
```
{ stdenv,
...
, cddbSupport ? true, libcddb ? null
...
}:
let
opts = [
...
(mkFlag cddbSupport "CONFIG_CDDB=y" libcddb)
...
];
in
stdenv.mkDerivation {
...
configureFlags = ... ++ concatMap (a: a.flags) opts);
buildInputs = ... ++ concatMap (a: a.deps) opts;
};
```
kind, but instead does this
```
{ lib }: with lib; mkUFDerivation (attrs: with attrs; {
required = mkRequired [ "stdenv" "pkgconfig" ];
# mkYNFlag : (defaultValue : Bool) -> (configureFlag : String) -> (buildInputs : [ String ]) -> mkUFDerivationMagic
darwin = mkYNFlag stdenv.isDarwin null [ "CoreAudio" ];
alsa = mkYNFlag stdenv.isLinux "ALSA" [ "alsaLib" ];
jack = mkYNFlag false "JACK" [ "libjack" ];
samplerate = mkYNFlag useFlags.jack "SAMPLERATE" [ "libsamplerate" ];
...
cddb = mkYNFlag true "CDDB" [ "libcddb" ];
...
}) (attrs: with attrs; stdenv.mkDerivation {
...
nativeBuildInputs = [ pkgconfig ];
configureFlags = useFlags.configureFlags;
buildInputs = useFlags.buildInputs;
...
})
```
- Note how short that is.
Adding a new dep with a use flag and configure options? Its a single line!
Disabling a dep? `nixpkgs.config.<package>.dep = false`!
It's shorter than your shortest `packageOverrides` thing with `null`s and it can also properly check any deps between flags (which `.override` fails to do and `null`s forbid you to even think about, btw).
But "in nixpkgs `nixpkgs.config` is deprecated, you should use overrides instead!" Right? Pfft!
- Support for other build systems is really simple too: just make another kind of `mkYNFlag`.
Isn't that _much_ better than what master has?
SLNOS says "Yes!". Hence let me introduce you into how the above works line by line.
- First line. We call `lib.mkUFDerivation` that does all the magic.
Note that we tie the knot with `attrs` there. That `attrs` feels like `attrs = { inherit useFlags pkgs; } // pkgs;` to the rest of the function.
- Second line adds `stdenv` and `pkgconfig` to required `pkgs`.
- Other attrs use `lib.mkYNFlag` to make up required `pkgs` and `useFlags` attributes.
- The body (second argument of `mkUFDerivation`) gets the same `attrs` and does the rest pretty much as usual, but now it can use `useFlags` for awesome things.
Calling this, as you might have noticed, does require a slightly different kind of infrastructure behind `callPackage`.
Merging such expressions with changes to nixpkgs is paniful, so SLNOS doesn't use them much (yet).
However, in the light of that expression above I fail to see why nixpkgs goes out of the way to reject use flags (#12877).
Like, seriously, just, WHY?!
Now, to completely blow your mind, look at the above expression and then remind yourself that many, if not most, packages in nixpkgs differ only in deps, `src` and some `meta` attributes, the rest is very copy-pastey.
Wouldn't it be cool if all you needed to do to define most packages was: call some magical `lib.mkUFNoCopyPasteyDerivation`, specify use flags with deps, specify `src`, `meta.license` and `meta.description`, and it figures out the rest for you (`homepage` from `src`, `platforms` from deps, etc).
Now, do you see the hidden agenda behind #35075 and
As a side note, `meta.platforms` should be checked (or just autogenerated, or maybe both, maybe with `config.checkMetaPlatforms` or something) against platforms
produced by intersecting platforms of derivations from `buildInputs`. But I have not implemented that properly yet.
of #35906 (comment) ?
Do you want to welcome such an agenda now (if you do, I can port `mkUFDerivation` infra to nixpkgs) or do you want to wait another 3-5 years until, maybe, somebody with commit access implements something like this (usually, worse) with little to no discussion.
Let me remind you that this is, pretty much, my experience with nixpkgs merge process, e.g. see anything nixos related from https://github.com/NixOS/nixpkgs/pulls?page=1&q=is%3Apr+author%3Aoxij+is%3Aunmerged&utf8=%E2%9C%93 , which gave birth to SLNOS.
I want SLNOS to be less than a 100 patches on top of master (most of which would be in OS department), but master makes good things hard to archive in less than 400, which is really painful to merge all the time.
Why can't nixpkgs move in a better direction with use flags? Seriously. So much time is wasted by nixpkgs contributors doing copy-pastey things, and then by SLNOS contributors undoing those things and merging with SLNOS. So frustrating! Grrr.
Can't anything be done with bikeshedding and "not invented here" syndrome in nixpkgs? Like, did you read GuixSD's sources yet? No? You should go read GuixSD's sources! Grrr.
/cc @edolstra @shlevy @vcunat on this
|
|
Oh well. The real problem is: you say «worse», what you mean is it allows strictly less, I agree that the solution in Nixpkgs will be worse, but there is the position that it will do less and will be in some sense more lightweight and so it will be in some sense «better». I have already implemented and committed multiple solutions to remove duplication between Also, I tried to keep a couple of latest versions by default for libraries, and that also was explicitly rejected. I think there is some idea about too many combinations making things untestable or the exact use pattern of library versions hard to establish or risk of having different default dependency versions for downstream packages and problems when combining them further downstream. The thing that happens with 3–5 years and limited implementation is not based as much on NIH, as on accumulation of support for some feature that makes reverting a small enough implementation unattractive. The detail with Ignoring all these flags etc. and just passing Copy-pastey expressions are not enough. when they are enough, if I fully buy all the idea that a separate version should not be specified, come at 16 non-blank lines with 10 lines to actually fill. Platforms from deps is iffy, unifying deps and arguments meets pushback, and the rest actually requires human input. So when you find out that people want the most limited packages possible, there is just nothing to generate. It is probably better to make as much of the distinctions inside Nixpkgs as possible library functions, and not these splits into the lists like In the context of |
|
I have already implemented and committed multiple solutions to remove duplication between `buildInputs` and the function parameters, and all of them were
I'm not aware of anything except `composableDerivation` and things similar to how `cmus` expression is done. Any pointers?
I think there is some idea about too many combinations making things untestable or the exact use pattern of library versions hard to establish or risk of having different default dependency versions for downstream packages and problems when combining them further downstream.
I can get behind that argument. But I also see a solution: simply hide non-default versions by default with something like `knownVulnerabilities`, e.g. `knownBetterVersions` :)
You need it? And it doesn't meet the default nixpkgs inclusion conditions? You just have to enable it explicitly.
The thing that happens with 3–5 years and limited implementation is not based as much on NIH, as on accumulation of support for some feature that makes reverting a small enough implementation unattractive.
That is another plausible explanation, yes, but seeing how NixOS slowly adopts things that were proposed and wholeheartedly rejected years ago I wonder if that is actually the case.
My experience shows that most people hate reading others' source code. In SLNOS we have "show you did your research first" rule (which, you might have noticed, I try to follow in my nixpkgs PRs too). Anything non-trivial has to provide relevant references like "it's a remake of/solution to such and such nixpkgs/Triton/GuixSD/Gentoo/Arch/Debian/etc PR/issue/etc".
and just passing `null` leads to shorter default-case expressions anyway (and that is what too many people care about).
Eh? My whole point that it is not. Did you read the above?
As for `null`s not allowing to check something — why exactly? I can check for `null` just fine.
By
it can also properly check any deps between flags (which `.override` fails to do and `null`s forbid you to even think about, btw)
I mean that flags are strictly more expressive than `null`s. How do you properly disable `x11Support`? You set it to `false`. How you do that with `null`s? You override all the X11 libraries with `null`s? Seriously? Flag dependencies are easy to check, ad-hoc `null` overrides are much harder to check properly (expressions get large and ugly pretty fast).
But if one almost never overrides things because hydra cache then, yes, all of that is purely academic.
Platforms from deps is iffy,
But completely optional and independent from the rest of the above (it's just that I like the idea). Because I like all the ideas that lead to more automation. Computers are cheap. My time is very limited. I hate cross-compiling shit for hours at a time to find out that what I actually needed is not actually supported.
unifying deps and arguments meets pushback,
"Yes, but why?"
and the rest actually requires human input.
If you mean "the expressions need to be rewritten" then, yes, they do.
So when you find out that people want the most limited packages possible, there is just nothing to generate.
=/
It is probably better to make as much of the distinctions inside Nixpkgs as possible library functions, and not these splits into the lists like `nativeBuildInputs`.
It just came to me that you can do that with the SLNOS `mkUFDerivation` too, like so
```
mkUFDerivation (x: {
native = mkNativeBuildInputs [ "pkgconfig" ];
}) body
```
and a proper `stdenv.mkDerivation` (or whatever else) wrapper in the wrapper.
The most expensive part of Nixpkgs is the leaf package expressions, and if one could just give them a different `stdenv.mkDerivation`, the package version/patch database could be maintained together by everyone and the stdenvs could be different.
Hm. That is quite an interesting idea, actually. But I don't see that working for complicated packages with intricate interplay between inputs and build phases, but most packages can be written this way, yes.
I don't see that happening in nixpkgs, though. Such a thing is just too far away from what the core of contributors cares about.
In the context of `meta.broken`, that probably means putting everything into `meta.broken`; maybe with optional-trace-generating wrappers around groups of conditions to hint whether the failure is likely to be a buildtime or a runtime one.
Eh? But why? It isn't like `nix-env` depends specifically on `meta.broken` or something. We can do pretty much whatever with `meta` attributes. Why not do something fuzzable then?
|
|
I find that I run into a bad interaction of having asserts at the top level of a package expression with overrides. Namely, if an assert fires due to an a change from an overlay, then a future use of The only way I've found to fix this is to make the assert lazier by placing it right before nghttp2 is used, i.e. replace with While I do like having these checks, it's a bit annoying to have to manually lazy-ify each one. I hope a better design here can avoid this problem; I haven't spent the time to read through this thread and all the proposals, but for example I think something like #36229 (comment) might still be prone to this issue. |
|
> I have already implemented and committed multiple solutions to remove duplication between `buildInputs` and the function parameters, and all of them were
I'm not aware of anything except `composableDerivation` and things similar to how `cmus` expression is done. Any pointers?
Well, for example `helperArgNames` (up to beginning 2016)
You need it? And it doesn't meet the default nixpkgs inclusion conditions? You just have to enable it explicitly.
I wonder is a bot could maintain `nixpkgs-attic` repository, which would automatically make the version-addressed snapshots of Nixpkgs `master`. If some version is not present in `master` anymore, it just stops getting any changes. A policy could be that the versions that are still present in Nixpkgs should be changed there (this would be ‘‘enforced’’ by the bot actually overwriting the changes), and for old versions anything goes if someone cares.
That is another plausible explanation, yes, but seeing how NixOS slowly adopts things that were proposed and wholeheartedly rejected years ago I wonder if that is actually the case.
I have seen the full spectrum of reactions to changes from committers pushed to master — reverts, negotiating the scale down, complaints with no action. Basically, there is a desire for minimalism, but then there are practical considerations that slowly erode the will to oppose the feature creep.
> and just passing `null` leads to shorter default-case expressions anyway (and that is what too many people care about).
Eh? My whole point that it is not. Did you read the above?
Yes. Passing the null often allows the expression that doesn't care. Your list of supported overrides is in itself longer than many expressions I have written.
How do you properly disable `x11Support`? You set it to `false`. How you do that with `null`s?
Just `libX11`. Yes, flags are more expressive, but they are more explicit, and have to be paid some attention during package updates, and that last part means they will not gain support in the next three years.
But if one almost never overrides things because hydra cache then, yes, all of that is purely academic.
Most high-volume Nixpkgs committers don't use a complex set of overrides (either because Hydra or because they want a set of expressions that receives the most community testing), so your arguments for USE flag — valid arguments — are arguing for an undesirable tradeoff.
> unifying deps and arguments meets pushback,
"Yes, but why?"
Maybe evaluation times, actually. Or evaluation RAM footprint. Maybe also the distant hope of having static typechecking for Nix one day.
> and the rest actually requires human input.
If you mean "the expressions need to be rewritten" then, yes, they do.
No, I mean that of the 8 remaining lines, you can generate zero with Nix. You cannot set maintainer, you do need human input for platforms, you cannot calculate the license in pure Nix.
Hm. That is quite an interesting idea, actually. But I don't see that working for complicated packages with intricate interplay between inputs and build phases, but most packages can be written this way, yes.
Maintaining the interplay between flags and build phases is something that a lot of people in Nixpkgs want to avoid. So this will have to live in a separate repo.
We-ell, if you can beat LibreOffice into submission without going mad, I can Just Commit It and I know from experience that people just take whatever LibreOffice version someone else updates. As long as it resembles a working LibreOffice version. And I also will gratefully accept work from someone else for LibreOffice updates.
In general if the expression is hard enough to get right even for one set of options, I would expect supporting more build variants to be even harder.
I don't see that happening in nixpkgs, though. Such a thing is just too far away from what the core of contributors cares about.
Nixpkgs packages have pressure to minimalism. We just need to steer it into a direction that is slightly more convenient to parse from outside. I mean, most Nixpkgs expressions are already there — if you pass them fake `stdenv`, you get _all_ the valuable information there is (inputs, version, patches) with little effort.
> In the context of `meta.broken`, that probably means putting everything into `meta.broken`; maybe with optional-trace-generating wrappers around groups of conditions to hint whether the failure is likely to be a buildtime or a runtime one.
Eh? But why? It isn't like `nix-env` depends specifically on `meta.broken` or something. We can do pretty much whatever with `meta` attributes. Why not do something fuzzable then?
I have been in a discussion whether some attributes in `meta` should _not_ ever exist. A wrapper function which is just a trace-if-desired could be simpler to negotiate into Nixpkgs mainline, and easy to override in the downstream use-case.
|
|
I think `meta.broken` is currently lazy enough.
|
|
@aneeshusa, right. This exactly my point in
It's shorter than your shortest `packageOverrides` thing with `null`s and it can also properly check any deps between flags (which `.override` fails to do and `null`s forbid you to even think about, btw).
`null`s are evil.
|
|
Well, for example `helperArgNames` (up to beginning 2016)
Hm. From what I can surmise from `git log -S helperArgNames` and #4210 the issue with `builderDefsPackage` was that it was more copy-pastey than plain `mkDerivation`s for packages with small number of flags (which is the most packages of nixpkgs), that is. See e.g. cc9547d or 8878e8e. This is not the case with `mkUFDerivation`.
Any more examples? I seriously want to know why every other attempt failed. Please teach me what to search with. I looked at `git log --numstat pkgs/build-support/ | grep -E '^\s*0\s*[1-9][0-9]*' | sed 's/^[^p]*//' | sort | uniq` but there's nothing else (except already mentioned `builder-defs`) that looks like it might be the thing I want.
ATM I see no drawbacks to `mkUFDerivation` except marginally longer evaluation times.
I wonder is a bot could maintain `nixpkgs-attic` repository
If nix could partially evaluate and dump code as data (Guix has this for free because they do it in LISP, btw) `mkUFDerivation` would cost nothing and attic bot could be implemented pretty easily by just partially evaluating, dumping, and then deduplicating expressions from `all-packages.nix` on every master commit.
Yes. Passing the null often allows the expression that doesn't care. Your list of supported overrides is in itself longer than many expressions I have written.
In which case you could just dump all of it into `required` and get exactly the same lack of extensibility.
so your arguments for USE flag — valid arguments — are arguing for an undesirable tradeoff.
But I also argue that adding fuzzing on top of `mkUFDerivation` changes that tradeoff. Well, whatever. I see your point, you see my point.
Maybe also the distant hope of having static typechecking for Nix one day.
A-ha-ha. (It would be cool, indeed, but knowing how nixpkgs does things now: "a-ha-ha".) I do want to write an alternative interpreter (with incremental and partial evaluation) for nix in Haskell or Rust eventually, though.
you do need human input for platforms, you cannot calculate the license in pure Nix
Well, actually... You know about my fascination with autogenerating platforms already. But you could do that for licenses too. I actually did implement a license checker for `checkMeta` that checked that you didn't link (with `buildInputs`) against GPL packages in non-GPL packages. And got a ton, A TON, of evaluation errors :) The reality is that nobody actually cares about the licenses. If you start properly checking them nothing ever evaluates. But the moral of that story is that if people did actually care about the licenses, you could derive them from package deps too. Just sayin'.
We-ell, if you can beat LibreOffice into submission without going mad, I can Just Commit It and I know from experience that people just take whatever LibreOffice version someone else updates. As long as it resembles a working LibreOffice version. And I also will gratefully accept work from someone else for LibreOffice updates.
:) No thanks. But I'll ask in SLNOS. But the positive result is unlikely. :)
In general if the expression is hard enough to get right even for one set of options, I would expect supporting more build variants to be even harder.
I think fuzzing changes that.
>I don't see that happening in nixpkgs, though. Such a thing is just too far away from what the core of contributors cares about.
Nixpkgs packages have pressure to minimalism. We just need to steer it into a direction that is slightly more convenient to parse from outside. I mean, most Nixpkgs expressions are already there — if you pass them fake `stdenv`, you get _all_ the valuable information there is (inputs, version, patches) with little effort.
... but no information about the options. Since options are the hardest part of an expression, it seems easier to me to maintain a parallel set of expressions that "borrow" src, patches and meta from master's expressions and do the rest themselves.
I have been in a discussion whether some attributes in `meta` should _not_ ever exist. A wrapper function which is just a trace-if-desired could be simpler to negotiate into Nixpkgs mainline, and easy to override in the downstream use-case.
I agree. Fuzzing, though.
|
|
> Well, for example `helperArgNames` (up to beginning 2016)
Hm. From what I can surmise from `git log -S helperArgNames` and #4210 the issue with `builderDefsPackage` was that it was more copy-pastey than plain `mkDerivation`s for packages with small number of flags
helperArgNames were criticised separately (and later iterations builderDefsPackage were non-copy-pastey)
> I wonder is a bot could maintain `nixpkgs-attic` repository
If nix could partially evaluate and dump code as data (Guix has this for free because they do it in LISP, btw)
Well, Lisp gives you dumping the code, but not proper partial evaluation…
With Nix2.0 `nix edit`, though, it is possible to just grab the leaf package file without even parsing Nix code. Of course, there are packages that won't work like that, but they are a minority.
> Yes. Passing the null often allows the expression that doesn't care. Your list of supported overrides is in itself longer than many expressions I have written.
In which case you could just dump all of it into `required` and get exactly the same lack of extensibility.
In the current style I can still pass `null`. Of course, with `required` I can pass something with an empty `outPath`…
> so your arguments for USE flag — valid arguments — are arguing for an undesirable tradeoff.
Well, whatever. I see your point, you see my point.
I think _our_ value disagreement is mostly that I want to have my LibreOffice from a publically mirrored repository, so I won't simply use SLNOS for everything. Instrumentally, I would also prefer to have the package-database maintenance in Nixpkgs a viable shared resource.
> Maybe also the distant hope of having static typechecking for Nix one day.
knowing how nixpkgs does things now: "a-ha-ha".
Hey, that's the point that you _do not_ need to change Nixpkgs much, you can just maintain a small list of weird places (outside Nixpkgs mainline).
> you do need human input for platforms, you cannot calculate the license in pure Nix
The reality is that nobody actually cares about the licenses. If you start properly checking them nothing ever evaluates. But the moral of that story is that if people did actually care about the licenses, you could derive them from package deps too.
It is a fun legal exercise just how optional your GPL optional dependency should be before you can say «the source tarball is MIT but once you enable libSomething support the binary is GPLv3+».
Anyway, the dependencies set a bound on the license, but they don't actually guarantee that the output license is exactly that.
> In general if the expression is hard enough to get right even for one set of options, I would expect supporting more build variants to be even harder.
I think fuzzing changes that.
Sorry, no. If you need to negotiate with build system to get one variant working, you need to negotiate even more about multiple variants working. Fuzzing may help you notice when it breaks, but when it breaks, it needs fixing, and the initial assumption is that this fixing is complicated.
I will gladly and gratefully accept a counterexample in the form of a working configurable LibreOffice package, but
No, thanks.
... but no information about the options. Since options are the hardest part of an expression, it seems easier to me to maintain a parallel set of expressions that "borrow" src, patches and meta from master's expressions and do the rest themselves.
Are you saying that adding options/use-flags everywhere more than doubles the effort?
> I have been in a discussion whether some attributes in `meta` should _not_ ever exist. A wrapper function which is just a trace-if-desired could be simpler to negotiate into Nixpkgs mainline, and easy to override in the downstream use-case.
I agree. Fuzzing, though.
What about fuzzing? You just replace «pick the warning» function with «these kinds of problems are just ignored».
|
|
Well, Lisp gives you dumping the code, but not proper partial evaluation…
But `eval` for LISP is 15 lines of LISP, so you can just write another interpreter right there... It will be slow, though.
In the current style I can still pass `null`. Of course, with `required` I can pass something with an empty `outPath`…
Fine. I see you want things to be dirty sometimes, I get the usefulness of that. `mkUFDerivation` needs to support `null`s somehow. Will do.
I think _our_ value disagreement is mostly that I want to have my LibreOffice from a publically mirrored repository, so I won't simply use SLNOS for everything. Instrumentally, I would also prefer to have the package-database maintenance in Nixpkgs a viable shared resource.
Wait, do I hear you saying that if I port `mkUFDerivation` just to make LibreOffice build sanely, the upstream will adopt the patchset?
Hey, that's the point that you _do not_ need to change Nixpkgs much, you can just maintain a small list of weird places (outside Nixpkgs mainline).
Fair enough.
Anyway, the dependencies set a bound on the license, but they don't actually guarantee that the output license is exactly that.
True, but if you don't do any legal gymnastics most of the time you get no actual choice except GPL. (And I think it's a good thing.)
Sorry, no.
Well, ok, fair enough.
Are you saying that adding options/use-flags everywhere more than doubles the effort?
For packages with a bunch of options/use-flags? Of course. It can be 10x-100x more effort. Look at `ffmpeg` on master. It's so bad that master has _two completely different expressions_ for `ffmpeg-3.4` (`ffmpeg generic`, and `ffmpeg-full`). Options are hard. Hence `mkUFDerivation`.
What about fuzzing? You just replace «pick the warning» function with «these kinds of problems are just ignored».
It seems to me that this way you would just push currently explicit structures of possible problems like `meta.broken`, `meta.knownVulnerabilities`, etc into a function arguments. _Some_ kind of modular handling for those things is due, though, I agree. How about replacing those things with `meta.issues` of `{ kind : String, condition = true : Bool, reason : String, ... }` type, e.g.
```
meta.issues = [
{ kind = "broken.won't-build"; condition = x11Support; reason = "x11 support is broken ATM"; }
{ kind = "broken.won't-work"; condition = pulseaudioSupport; reason = "pulseaudio support is buggy"; }
{ kind = "vulnerable"; id = "CVE-XXXX-XX-XX-XXXX"; }
];
```
? "."-separated strings can be made into lists to make it parse less. Common kinds can be made in lib functions like `markBroken condition reason` or whatnot.
Most of the OP seems to be pretty easy with that.
|
|
> Well, Lisp gives you dumping the code, but not proper partial evaluation…
But `eval` for LISP is 15 lines of LISP, so you can just write another interpreter right there... It will be slow, though.
But Guile isn't LISP 1.5, it's larger and more complicated. But yes, you can go quite far; although it is enough for a parser to be available inside the language — Julia would be fine as well.
> In the current style I can still pass `null`. Of course, with `required` I can pass something with an empty `outPath`…
Fine. I see you want things to be dirty sometimes, I get the usefulness of that. `mkUFDerivation` needs to support `null`s somehow. Will do.
Actually, requiring having `meta` and just allowing an empty `outPath` (up to: literally "") could be a better way. I just want global guarantees, after all (and I am not a SLNOS user and unlikely to migrate to SLNOS as the primary package source), so maybe I want tradeoffs incompatible with yours.
> I think _our_ value disagreement is mostly that I want to have my LibreOffice from a publically mirrored repository, so I won't simply use SLNOS for everything. Instrumentally, I would also prefer to have the package-database maintenance in Nixpkgs a viable shared resource.
Wait, do I hear you saying that if I port `mkUFDerivation` just to make LibreOffice build sanely, the upstream will adopt the patchset?
There is my position and then there is overall reaction. And then there is usage for LibreOffice and usage in general.
If it turns out to be saner than now (not a very high bar, but a nontrivial one — it still has to work with the existing upstream build system) — I guess it has a very high probability to get adopted specifically in LibreOffice. Spreading it will still not be feasible.
I will just say that I am not rejecting a working LO update, whatever the style and merge the PR. I hope that the tolerance level for this specific package is high enough for noone to want to revert.
True, but if you don't do any legal gymnastics most of the time you get no actual choice except GPL. (And I think it's a good thing.)
«Most of the time» is a measure-dependent notion…
> Are you saying that adding options/use-flags everywhere more than doubles the effort?
For packages with a bunch of options/use-flags? Of course. It can be 10x-100x more effort. Look at `ffmpeg` on master. It's so bad that master has _two completely different expressions_ for `ffmpeg-3.4` (`ffmpeg generic`, and `ffmpeg-full`). Options are hard. Hence `mkUFDerivation`.
I think they just default in the opposite direction.
> What about fuzzing? You just replace «pick the warning» function with «these kinds of problems are just ignored».
It seems to me that this way you would just push currently explicit structures of possible problems like `meta.broken`, `meta.knownVulnerabilities`, etc into a function arguments. _Some_ kind of modular handling for those things is due, though, I agree. How about replacing those things with `meta.issues` of `{ kind : String, condition = true : Bool, reason : String, ... }` type, e.g.
```
meta.issues = [
{ kind = "broken.won't-build"; condition = x11Support; reason = "x11 support is broken ATM"; }
{ kind = "broken.won't-work"; condition = pulseaudioSupport; reason = "pulseaudio support is buggy"; }
{ kind = "vulnerable"; id = "CVE-XXXX-XX-XX-XXXX"; }
];
```
? "."-separated strings can be made into lists to make it parse less. Common kinds can be made in lib functions like `markBroken condition reason` or whatnot.
Most of the OP seems to be pretty easy with that.
Hm. I like it. Not sure I want to push for such a change before 18.03, though. This «kill all `assert`» thing has the benefit of clearly not being doable quickly, so nobody will try until the release.
|
|
I've proposed a similar PR a while ago #32019. This would fix #11226 (comment) and #31884 It took only 20 files to edit to make nixpkgs evaluatable on non-cross setup. Also, note that some asserts are used as license checks, so can't be treated as "broken" |
|
Also, note that some asserts are used as license checks, so can't be treated as "broken"
Hm. I wonder if creating `lib.licenses.unsatisfied` (like `unfree` only
worse) and setting it when the EULA has not been accepted is a good idea
(asserts are annoying for overrides and for checking `meta`).
Asserts inside `stdenv` are OK, of course (I started in `pkgs` to
exclude `lib` but forgot about `stdenv`).
|
|
Updated the description re: license check and re: |
Ericson2314
added
the
6.topic: portability
8 tasks
This was referenced Apr 24, 2018
This was referenced Feb 21, 2019
|
@7c6f434c Care to turn this into a proper RFC, so that a “definitive” decision could be made? |
10 tasks
This comment has been minimized.
This comment has been minimized.
stale
bot
added
the
2.status: stale
|
Still relevant. |
|
Closing this. This issue is so outdated, we'd need somebody motivated to evaluate the current situation and open a new issue. I don't think this will require an RFC though. |
12 tasks
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue description
A lot of Nixpkgs packages currently use
assert. Most of the uses ofasserthad been introduced beforemeta.brokenhas been implemented. Usingmeta.brokenwith an expression condition has benefits, for example, it can be queried viameta.availableand catching assertions requiresbuiltins.tryEval, and it is always better not to add more uses oftryEval. See also #36226. Apparently enthusiastically supported by @Ericson2314Tracking
My model of assertions: an assertion happens in a
.nixfile insidepkgs/in a line not starting with a#in the beginning of the line or after one of(){}, before end of line (with condition on the next line) or one of(){}.Sanity check:
(files without assertions as defined above that still have
assertas a alphanumeric string as agrepword haveasserteither as a part of a dash-separated name, or inside quotes)stdenvis a separate thing, the uses ofassert— like inlib, which is outsidepkgs— do not have a convenientmeta.brokenaround.Re: asserts for licenses: should there be
lib.licenses.unsatisfied, likeunfreebut without a permanent override available (I guess$NIXPKGS_ALLOW_UNSATISFIED_LICENSEis enough for all debugging purposes)? The motivation is the same as with things that go tometa.broken: inspection ofname/meta`.Progress countdown:
→ 528
List of files:
The text was updated successfully, but these errors were encountered: