[RFC/WIP] multiple-outputs: Change the order of outputs to have binaries in the primary output #14766

Merged
merged 27 commits into from Aug 30, 2016
@dezgeg
Contributor
dezgeg commented Apr 16, 2016

This series reorders the outputs of each multiple-output splitted package in nixpkgs to have the output that is installed into the user environment be the first one (though I made a single exception: glibc has its libraries first to keep usages like ${stdenv.cc.libc}/lib/ld-linux-x86-64.so.2 working). Previously the dev output was typically the primary one, although there were some exceptions like the kernel package. To accommodate this, mkDerivation was changed to use the dev output by default in packages passed to buildInputs and friends.

The rationale for doing this is to avoid breakage to users' scripts and systemd services that reference binaries like ${pkgs.perl}/bin/perl (e.g. https://github.com/NixOS/nixos-org-configurations/blob/master/delft/common.nix#L62). Currently, those usages will just silently produce a nonsensical path, which IMHO is a significant regression to the pre-multiple-outputs state.

Also this should fix some of our in-tree systemd units which suffer from the same issue and avoid the need for changes like 3e38a71

As a part of doing this, I manually eyeballed all usages of ${pkgs.multiout-package} . I collected this list of packages where the implicit dev is currently incorrectly used in place of the correct output: https://github.com/dezgeg/nixpkgs/commits/bad-multiout-refs. So this PR fixes some of them as a side-effect (though not all, there are other things wrong as well).

Note: I'm still compiling most of this (at least stdenv builds...). It's certainly possible that chooseDevOutputs needs to be used in more places...

cc @edolstra @vcunat @ttuegel @joachifm @lethalman

@vcunat
Member
vcunat commented Apr 16, 2016

Hmm, yes, probably. At least in principle, though I haven't thought through many potential consequences yet. I'm not certain how fast I get to this properly. I suppose I'd first solve issues like #14682, as that seems some easily fixable sloppy programming (of mine).

@cstrahan
Contributor
cstrahan commented Apr 16, 2016 edited

The general idea here looks good to me. The changes to multiple-outputs.sh look good (particularly with respect to the defaulting of $propagaterOutput), and the exception for output order for glibc seems reasonable given how pervasive the current typical use case is. Hopefully this will make things a little easier on newcomers; it could be my own bias from my familiarity with the old status quo, but I think "${foo}/bin/foo" and "${foo.dev}/include" is more intuitive than "${foo.bin}/bin/foo" and "${foo}/include".

👍

@cstrahan
Contributor

I forgot to mention, I also approve of having buildInputs in the stdenv default to using the dev output, expecting it to follow the convention of propagating the other requisite outputs.

I'd like to see this get pushed along, assuming no one comes along with a reasonable objection.

@abbradar
Member

I don't have a strong opinion either way, but to new users this would probably be more consistent with Debian and Red Hat-based distributions' -devel packages.

@joachifm
Member

Agreed, this sounds like a good idea, so better to just deal with any bugs.

@dezgeg
Contributor
dezgeg commented Apr 17, 2016

I built closures.smallContainer.x86_64-linux and the minimal installation CD successfully (way better than I expected...). There are some minor evaluation errors exposed by this series (e.g. GHC using a version of gmp that is not split) that I need to deal with.

@cstrahan cstrahan referenced this pull request Apr 17, 2016
Closed

Search path fixes, v2 #14694

1 of 7 tasks complete
@edolstra
Member

Doesn't this break

buildInputs = [ foo ];

since you would now have to write

buildInputs = [ foo.dev ];

? That was the whole reason for putting dev first.

@dezgeg
Contributor
dezgeg commented Apr 18, 2016

@edolstra: No, that is handled by 090c967

@edolstra
Member

Okay, I see there is some special-casing for this in mkDerivation. But that seems confusing to me, i.e. foo will mean foo.dev in some contexts but not others.

@vcunat
Member
vcunat commented Apr 18, 2016

Well, the whole point is to try guessing what output was meant when none was specified... obviously the best guess depends on context.

@dezgeg
Contributor
dezgeg commented Apr 18, 2016 edited

Yes, that is pretty much inevitable. Even currently (as in before this PR) environment.systemPackages = [ pkgs.curl ]; doesn't refer to curl.dev.

Also cases like services.httpd.package = pkgs.apacheHttp_2_4; - that probably shouldn't refer to the dev output either.

@vcunat
Member
vcunat commented Apr 18, 2016 edited

The last one is correct, I believe. The users of services.httpd.package should choose the particular output, as they might need different on different places.

@dezgeg
Contributor
dezgeg commented Apr 22, 2016

#14902 seems another victim of silent breakage by accidentally using the dev output.

@vcunat
Member
vcunat commented Apr 26, 2016

I don't think there can be any argument against the commits that make output choice explicit (they make majority of the diff). These shouldn't even change hashes and could go to master directly (before conflicts can arise).

@trishume
Contributor
trishume commented May 1, 2016

I ran into some issues with this except with libraries defaulting to dev instead of out, which perhaps makes more sense than binaries. Example issue caused by this problem: #15124

@vcunat vcunat added a commit to vcunat/nixpkgs that referenced this pull request May 19, 2016
@vcunat vcunat Merge: make dev output references explicit
This is a rebase of most commits from #14766,
resolving conflicts and a few other evaluation problems.
dfabb50
@vcunat vcunat added a commit to vcunat/nixpkgs that referenced this pull request May 20, 2016
@vcunat vcunat Merge: make dev output references explicit
This is a rebase of most commits from #14766,
resolving conflicts and a few other evaluation problems.
e06987e
@vcunat vcunat added a commit that referenced this pull request May 22, 2016
@vcunat vcunat Merge: make dev output references explicit
This is a rebase of most commits from #14766,
resolving conflicts and a few other evaluation problems.
c4661e9
dezgeg added some commits Aug 24, 2016
@dezgeg
Contributor
dezgeg commented Aug 29, 2016

Ping on this. I would really prefer to not have ${pkgs.perl}/bin/perl-style references in users' custom scripts or systemd services silently broken when upgrading to 16.09 (or even in any future release where some additional packages might be converted to multiple outputs) and so far no other solutions to this have materialized.

This should be stable enough to not cause big breakage, e.g. Gnome3 and KDE5 on Linux build fine and Darwin seems ok too.

@dezgeg dezgeg added this to the 16.09 milestone Aug 29, 2016
@vcunat
Member
vcunat commented Aug 29, 2016

I suppose we can fixup various details after staging this.

I wonder whether to re-consider the default setting of meta.outputsToInstall, but probably not.

@dezgeg
Contributor
dezgeg commented Aug 29, 2016

Right, I would just like a confirmation from @edolstra if he agrees with this or not. Also @domenkozar might have something to say about one more mass-rebuild thing to 16.09 ;)

@domenkozar
Member

No objections for mass-rebuild if it's done before Thursday and it works (doesn't break too many packages and all tests pass).

@edolstra
Member

Looks good to me.

@lethalman
Member

Good. Can you please update release notes about this? Also the dev / devEnv change of ruby.

@lethalman
Member

There are some more places where .dev needs to be used I think, like for nss nspr gcc glibc glib readline ncurses gio and all these libraries that sometimes are used in build flags of packages. Searching for "-I" might help.

@dezgeg
Contributor
dezgeg commented Aug 30, 2016

I had already done most of those changes in master (e.g. 4db1657) via some regexps just to be able to spot breakages (e.g. d801e8a) but now that you mention, I indeed managed to miss some, thanks.

Should now be in a condition where I can stage this in the morning after some overnight build testing.

@dezgeg
Contributor
dezgeg commented Aug 30, 2016

Oh, and cc @zimbatm for the Ruby change (ba6d94e). Does this rename sound good, or would some other name be better?

Reason for changing is that the name dev is used in the multiple-outputs stuff and causes problem there.

Also, any documentation references to update? Didn't notice any but the name dev is not very greppable.

@zimbatm
Contributor
zimbatm commented Aug 30, 2016

Looks good. I don't think ruby.dev isn't documented right now and I'm considering dropping it in the future.

@dezgeg dezgeg merged commit 03fb2c1 into NixOS:staging Aug 30, 2016

1 check failed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
@dezgeg dezgeg added a commit that referenced this pull request Sep 1, 2016
@dezgeg dezgeg Merge staging into master
Brings in:
    - changed output order for multiple outputs:
      #14766
    - audit disabled by default
      #17916

 Conflicts:
	pkgs/development/libraries/openldap/default.nix
8c4aeb1
@dezgeg dezgeg added a commit that referenced this pull request Sep 1, 2016
@dezgeg dezgeg openexr, libyaml-cpp: Fix output order to match others
This was changed in #14766

@edolstra
75b5922
@lethalman
Member

Wow, this is on staging? Awesome work!

@vcunat
Member
vcunat commented Sep 5, 2016

It's in master, actually, since 8c4aeb1.

@grahamc
Member
grahamc commented Sep 5, 2016

(and in 16.09!)

@joachifm joachifm referenced this pull request Sep 7, 2016
Closed

add nix.optimise service #18378

4 of 7 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment