split off perl-dependent components of groff?

[therealpxc]

Issue description

I noticed that fish pulls in a ~50MB dependency on perl because it depends on groff. See below.

fish's closure is huge, (and it's hardly all perl's fault)! But is taking up ~50MB by itself)

~/C/P/devnix   playground ±…  du -hc --max=0 (nix-store -qR (realpath (which fish))) | sort -h                                                     Mon 26 Jun 2017 09:08:07 PM MST
32K     /nix/store/zgrj2367wbcfg8j4lfvs6hf73maf8apr-libcap-2.25-lib
84K     /nix/store/4717387i4al2fk7kp747na0yg9m8586s-bzip2-1.0.6.0.1
124K    /nix/store/kk8r62xz5g1hn1mn1307fcj24anp5m6g-zlib-1.2.11
172K    /nix/store/mg053x9xlvcc66z50frli7vdv0bd1n9k-attr-2.4.47
200K    /nix/store/y5vkvnhaw6isvzh0sjxk3s0hkxxbqpd1-acl-2.2.52
244K    /nix/store/qcyy8l1z6m2v6ifsnsxifl8wdaznkyjr-bc-1.06
336K    /nix/store/ldjfvjhlvgbi65kqgb1a894bwb4h8497-lz4-131
392K    /nix/store/7xvvm0kybzlwwkrcv1ksj22a1d1vajfz-libpipeline-1.4.1
396K    /nix/store/d67cb599vq2hmlddz0axj93jkhnd78yh-readline-6.3p08
400K    /nix/store/xbd9v28yh7zq10ac2gdff18r0yr1xgbb-xz-5.2.2
504K    /nix/store/ally147d2h11jv34w1saml1ni7kkxmch-pcre-8.40
688K    /nix/store/m74m7c4qbzml7ipfxzlpxddcn9ah8jrs-gdbm-1.12
804K    /nix/store/v4jigflzsh9h2xs62fdn5x7wcffsr05r-libgpg-error-1.26
964K    /nix/store/2nzkvn5b9jxdj75mgw6cdvpl6438sdqw-gnused-4.4
1.1M    /nix/store/616dbi7gz8ik2265k2ii6b91hdf920fj-sqlite-3.17.0
1.1M    /nix/store/yqfgikjsrjjq8gk13ayc2k2w0bcv3bnm-cracklib-2.9.6
1.2M    /nix/store/fi3mbd2ml4pbgzyasrlnp0wyy6qi48fh-bash-4.4-p5
1.2M    /nix/store/vcakrv66mv3a7i0vzz1k9y8fw7ymwzpm-libgcrypt-1.7.7
1.3M    /nix/store/3r21p23hmbzj1ykv2wllqr14k1b0cdmx-gnugrep-3.0
1.5M    /nix/store/1gr6anpzzmqg620l67gkyzdh0a59h11y-pcre2-10.23
1.5M    /nix/store/fxhfl9map3y6anszpq678n32v4ixlsx7-systemd-232-lib
2.0M    /nix/store/cvdhmfc6ppn9phvhsqlqn4v8nl8nq9q9-man-db-2.7.5
2.7M    /nix/store/7kp8g57zabdm3x3zvbwac3pr38asfjh8-glibc-2.25-bin
2.7M    /nix/store/mqz7pd6199lvwkxv68bxj3gamazgfzwg-linux-pam-1.2.1
3.4M    /nix/store/zbjyc3ylb9bj3057rk5payv3sr0gnmkc-openssl-1.0.2l
3.9M    /nix/store/73bkrxyq8y0c8zc9nkch14v9pmn782py-shadow-4.4
4.3M    /nix/store/svrb46nmm9dz7p2azxf5mwabjqzfc29j-db-5.3.28
4.5M    /nix/store/xmb7xxnws01223ica0p6dl0a0f3f5dp2-util-linux-2.29.2-bin
4.7M    /nix/store/89bjrnjb8c9vvhzb92fz832x6fwg3kbj-gcc-5.4.0-lib
7.5M    /nix/store/sj8rnawbay2j2hiqgazsbq05rlpha5lq-ncurses-6.0
7.9M    /nix/store/0wbrl76g10iw43pvlqcyyh2lw3xglpxn-util-linux-2.29.2
8.6M    /nix/store/cjmnrbfqrdnllb4j6a6qlzbv55gqzrih-groff-1.22.3
9.8M    /nix/store/lkdr3z34n6n17g7xiaw8plyw7hv9idsl-gettext-0.19.8
11M     /nix/store/dkcs6dzhsn6l1fqp29z1k2jhqm9nlj82-fish-2.6.0
11M     /nix/store/k8dyhz9hrjydl23ny3131am6cws1rc07-coreutils-8.26
23M     /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25
49M     /nix/store/nsa311yg8h93wfaacjk16c96a98bs09f-perl-5.22.3
49M     /nix/store/zq0lf0fdc7n12zcdcs2qq3viz6mc87vr-python-2.7.13
216M    total

and

~/C/P/devnix   playground ±…  nix-store -q --tree (realpath (which fish))                                                                          Mon 26 Jun 2017 09:08:10 PM MST
/nix/store/dkcs6dzhsn6l1fqp29z1k2jhqm9nlj82-fish-2.6.0
+---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25
|   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
+---/nix/store/1gr6anpzzmqg620l67gkyzdh0a59h11y-pcre2-10.23
|   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   +---/nix/store/1gr6anpzzmqg620l67gkyzdh0a59h11y-pcre2-10.23 [...]
+---/nix/store/2nzkvn5b9jxdj75mgw6cdvpl6438sdqw-gnused-4.4
|   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   +---/nix/store/2nzkvn5b9jxdj75mgw6cdvpl6438sdqw-gnused-4.4 [...]
+---/nix/store/3r21p23hmbzj1ykv2wllqr14k1b0cdmx-gnugrep-3.0
|   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   +---/nix/store/ally147d2h11jv34w1saml1ni7kkxmch-pcre-8.40
|   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   +---/nix/store/ally147d2h11jv34w1saml1ni7kkxmch-pcre-8.40 [...]
|   +---/nix/store/3r21p23hmbzj1ykv2wllqr14k1b0cdmx-gnugrep-3.0 [...]
+---/nix/store/89bjrnjb8c9vvhzb92fz832x6fwg3kbj-gcc-5.4.0-lib
|   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   +---/nix/store/89bjrnjb8c9vvhzb92fz832x6fwg3kbj-gcc-5.4.0-lib [...]
+---/nix/store/cjmnrbfqrdnllb4j6a6qlzbv55gqzrih-groff-1.22.3
|   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   +---/nix/store/2nzkvn5b9jxdj75mgw6cdvpl6438sdqw-gnused-4.4 [...]
|   +---/nix/store/89bjrnjb8c9vvhzb92fz832x6fwg3kbj-gcc-5.4.0-lib [...]
|   +---/nix/store/fi3mbd2ml4pbgzyasrlnp0wyy6qi48fh-bash-4.4-p5
|   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   +---/nix/store/fi3mbd2ml4pbgzyasrlnp0wyy6qi48fh-bash-4.4-p5 [...]
|   +---/nix/store/nsa311yg8h93wfaacjk16c96a98bs09f-perl-5.22.3
|   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   +---/nix/store/k8dyhz9hrjydl23ny3131am6cws1rc07-coreutils-8.26
|   |   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   |   +---/nix/store/mg053x9xlvcc66z50frli7vdv0bd1n9k-attr-2.4.47
|   |   |   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   |   |   +---/nix/store/mg053x9xlvcc66z50frli7vdv0bd1n9k-attr-2.4.47 [...]
|   |   |   +---/nix/store/y5vkvnhaw6isvzh0sjxk3s0hkxxbqpd1-acl-2.2.52
|   |   |   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   |   |   +---/nix/store/mg053x9xlvcc66z50frli7vdv0bd1n9k-attr-2.4.47 [...]
|   |   |   |   +---/nix/store/y5vkvnhaw6isvzh0sjxk3s0hkxxbqpd1-acl-2.2.52 [...]
|   |   |   +---/nix/store/k8dyhz9hrjydl23ny3131am6cws1rc07-coreutils-8.26 [...]
|   |   +---/nix/store/nsa311yg8h93wfaacjk16c96a98bs09f-perl-5.22.3 [...]
|   +---/nix/store/cjmnrbfqrdnllb4j6a6qlzbv55gqzrih-groff-1.22.3 [...]
+---/nix/store/cvdhmfc6ppn9phvhsqlqn4v8nl8nq9q9-man-db-2.7.5
|   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   +---/nix/store/7xvvm0kybzlwwkrcv1ksj22a1d1vajfz-libpipeline-1.4.1
|   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   +---/nix/store/7xvvm0kybzlwwkrcv1ksj22a1d1vajfz-libpipeline-1.4.1 [...]
|   +---/nix/store/fi3mbd2ml4pbgzyasrlnp0wyy6qi48fh-bash-4.4-p5 [...]
|   +---/nix/store/cjmnrbfqrdnllb4j6a6qlzbv55gqzrih-groff-1.22.3 [...]
|   +---/nix/store/svrb46nmm9dz7p2azxf5mwabjqzfc29j-db-5.3.28
|   |   +---/nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25 [...]
|   |   +---/nix/store/89bjrnjb8c9vvhzb92fz832x6fwg3kbj-gcc-5.4.0-lib [...]
|   |   +---/nix/store/svrb46nmm9dz7p2azxf5mwabjqzfc29j-db-5.3.28 [...]
|   +---/nix/store/cvdhmfc6ppn9phvhsqlqn4v8nl8nq9q9-man-db-2.7.5 [...]

I know there is little hope to see fish's closure size drop down to be comparable to that of bash (~11M), but I'd like to see if I can get fish's closure size down smaller. I have some hope that since fish's only use of groff is through calling the nroff command that we could do something similar to what Fedora does and split the groff package. They have a package they call groff-base which includes the executables which don't depend on perl, and then groff-perl which includes all the executables which need perl to run. If we did something similar with our groff package, we could eliminate a ~50MB perl dependency from the dependency closures of fish and probably other packages.

This type of split, where one ‘subpackage’ (as they say in the Fedora link above) needs perlbut another does not, seems like it might be a common use case. fish could make use of the same facility with python, for example: only two fish functions included in fishneed python, and they can be added or removed separately just like executables on the path (as is the case with groff) . Since those functions are used only interactively and optionally, a smaller fish output not depending on python could be used as a dependency for scripts or as a slimmer installable option if the fish package could be split from its python-using components in the same way as Fedora splits groff-base and groff-perl.

Is there already a good way to do this kind of split using multiple outputs? Should groff just be two packages that share the same source tarball, in order to effect this split?

[vcunat]

Yes, I'd add another output for the perl stuff and use moveToOutput on them. Still, it's necessary to check if that breaks some internal groff "links".

Reverse dependencies will also need some work, unless e.g. groff defaults to the perl output which also gets symlinks to the non-perl binaries.

[therealpxc]

How does one add a buildInput (e.g., perl) to only one of the outputs for a multi-output derivation? I didn't see anything about the multiple outputs section of the Nixpkgs manual. Will Nix just figure out that there are references to a perl binary in one of the outputs and not the other?

I also don't fully understand why groff pulls perl into its closure at all with the current package definition-- I can see that perl is needed, but it's only listed as a nativeBuildInput. How come it gets included in the closure for groff? I thought that was runtime dependencies only.

From Section 3.3 of the Nixpkgs manual:

nativeBuildInputs

A list of dependencies used by the new derivation at build-time. I.e. these dependencies should not make it into the package's runtime-closure,

But binaries in the groff package are full of patched perl shebangs, even though perl is only present as a nativeBuildInput. Is this how it is supposed to work?

[edolstra]

The buildInputs are global for the entire derivation. Nix will automatically discover (via hash scanning) which outputs depend on which paths. So you should ensure that the out output has no references to Perl.

The description for nativeBuildInputs is perhaps misleading. The "should not" is a requirement on the package author. If you store the path of a native build input in the output, then it will be part of the closure.

Also, I think Perl should be a buildInput in this case.

[Profpatsch]

The split has been done in the above PR. Now invoking the pdf driver with -T is kind of broken, but that’s a different issue.