Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run time dependencies for modules with bindings are bloated #187

Open
jkryl opened this issue Jun 8, 2020 · 8 comments
Open

Run time dependencies for modules with bindings are bloated #187

jkryl opened this issue Jun 8, 2020 · 8 comments

Comments

@jkryl
Copy link

jkryl commented Jun 8, 2020

I have a project that uses npm modules with C++ bindings. When using node2nix to create a nix package, it has a dependency on gcc, node-source, and nodejs-X packages. However, those are rather build-time dependencies that are not required at run-time. I was able to work around it by doing a couple of changes:

1: I patched the generated nix files and added nodejs-slim package to buildInputs before nodejs package. When shebang in composition script is run, it finds the nodejs-slim package before nodejs package and replaces /usr/bin/env node at the beginning of JS scripts by a path to the slim package.

2: in postInstall script I added a command to remove all files from build subdirectory of modules with bindings except the actual shared library with the bindings. makefiles and object files were referencing various include paths thus pulling in nodejs source package and compiler package.

3: in postInstall script I added a command to strip debug info from shared library (bindings) for the same reason as in 2.

I create a docker image out of the node package generated by nix and the amount of space that was saved is impressive. The size decreased from 981 MB to just 271 MB. I'm wondering if I could accomplish the same with less hassle and without having to patch generated nix files.

@svanderburg
Copy link
Owner

Basically a Nix package (any Nix package, not just NPM-based packages) could get bloated if the resulting binary still contains store paths to dependencies that are not required at runtime. To reduce the footprint of these packages, these dependencies should be eliminated.

Now that I'm thinking about your issue. Perhaps this might be caused by the fact that the strip phase is the standard builder environment is probably unable to strip binaries from NPM packages because they reside in non standard sub paths.

I need to do a bit of investigation to see what the implications are and if there is a way how to solve this generically.

Probably, I need to finish my node-env.nix refactorings first to make this more convenient. In its current form, it is quite hard to make such feature additions.

@W1M0R
Copy link

W1M0R commented May 6, 2021

I would also be interested in a smaller package size. I tried injecting the node-slim package, but the derivation fails down the line because it can't find npm e.g.

  myNodeApp = import ./myNodeApp/default.nix {
    inherit pkgs system;
    nodejs = pkgs."nodejs-14_x";
    # nodejs = pkgs."nodejs-slim-14_x"; # slim does not include npm, causing the build to fail
  };

It looks like the nodejs package also includes python (due to node-gyp), whereas nodejs-slim does not:

❯ nix why-depends nixpkgs.nodejs nixpkgs.python3
/nix/store/y9ay04l5mfm255r296vhcjbxjqkjxp39-nodejs-14.16.1
╚═══lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py: …#!/nix/sto>
    => /nix/store/d44wd6n98f93hjr6q1d1phhh1hw7a17d-python3-3.8.8
❯ nix why-depends nixpkgs.nodejs-slim nixpkgs.python3
'nixpkgs.nodejs-slim' does not depend on 'nixpkgs.python3'

@jkryl
Copy link
Author

jkryl commented May 6, 2021

@W1M0R : The trick I use is to put nodejs-slim before nodejs everywhere where nodejs is listed as a dependency. npm, node-gyp, .. are still provided during the build time, while the runtime dependency is satisfied by the slim version of the package, hence bloated nodejs is not included in the result then. If you want to see a complete example, here is the commit (cluttered with many other changes, but you get the idea): openebs/mayastor@6b03e10

@W1M0R
Copy link

W1M0R commented May 6, 2021

Thanks @jkryl, I see what you did there. That should steer me in the right direction.

@W1M0R
Copy link

W1M0R commented May 6, 2021

@W1M0R : The trick I use is to put nodejs-slim before nodejs everywhere where nodejs is listed as a dependency.

That helped, and reduced my docker image size from 280MB to 200MB. Python is no longer part of my image, however gcc is still there.

❯ nix why-depends nixpkgs.nodejs-slim nixpkgs.gcc
'nixpkgs.nodejs-slim' does not depend on 'nixpkgs.gcc'

The gcc dependency is not because of nodejs-slim, so that is probably introduced by something else in my image.

I see that nodejs-slim still includes perl via openssl, which adds 50MB to the image. I don't think I'll be able to remove that dependency:

❯ nix why-depends nixpkgs.nodejs-slim nixpkgs.perl
/nix/store/7wpvzz0dzsq938778h4aaqrf219ygh2n-nodejs-slim-14.16.1
╚═══bin/node: …                    "/nix/store/apxpi7w14qkvbkpiy89lryarxv9s>
    => /nix/store/apxpi7w14qkvbkpiy89lryarxv9svngy-openssl-1.1.1k-dev
    ╚═══nix-support/propagated-build-inputs: … /nix/store/dzqd6vwynd55fwrsk>
        => /nix/store/dzqd6vwynd55fwrsksilj4p51kwy5xr6-openssl-1.1.1k-bin
        ╚═══bin/c_rehash: …#!/nix/store/6mnbpv86l5ff2vf765hinq95k12dhj85-pe>
            => /nix/store/6mnbpv86l5ff2vf765hinq95k12dhj85-perl-5.32.1

According to these issues, perl should not be a dependency and has been fixed:

  1. openssl.bin depends on perl because of c_rehash NixOS/nixpkgs#19965
  2. openssl: remove build-time dependency to host perl NixOS/nixpkgs#83446
  3. openssl: cross compilation without host perl NixOS/nixpkgs#115911

I'll see if a nixpkgs update will resolve this issue for me.

@bbigras
Copy link

bbigras commented Jun 3, 2021

Could node-packages.nix use nodejs as a nativeBuildInputs? Would that avoid including it as a runtime dep?

@kampka
Copy link
Contributor

kampka commented Dec 29, 2021

Now that I'm thinking about your issue. Perhaps this might be caused by the fact that the strip phase is the standard builder environment is probably unable to strip binaries from NPM packages because they reside in non standard sub paths.

Could it be that the main issue here is that dontStrip defaults to true, which is really an unexpected setting for most users I feel.
Setting dontStrip to false solves the gcc issue for me in most cases.

None the less, nodejs-slim should probably be the default runtime for packages generated by node2nix.
nodejs and the corresponding toolchain (npm et. al.) are really more of a build time dependency.
Sadly, npm packages often include paths, eg. shebangs, to a node interpreter which may need additional monkey patching to get it to use nodejs-slim and remove nodejs from the resulting closure.

@CMCDragonkai
Copy link

CMCDragonkai commented May 10, 2022

As an example when packaging a nodejs app using node2nix into docker, this is the contents:

[nix-shell:~/Projects/TypeScript-Demo-Lib]$ docker save typescript-demo-lib-1.3.1:scg60dr0kpn71fb745bcpkbmq7nyrhvl | tar x --to-stdout --wildcards '*/layer.tar' | tar t --exclude="*/*/*/*"
./
./bin/
./bin/typescript-demo-lib
./lib/
./lib/node_modules/
./tmp/
./nix/
./nix/store/
nix/store/29ci2j5c12188a3akbfb0ggw2glmjka3-icu4c-70.1-dev/
nix/store/5h6q8cmqjd8iqpd99566hrg2a56pwdkc-acl-2.3.1/
nix/store/66rvnvvwp6dmbb4i1nkpsgav09klshi7-zlib-1.2.12-dev/
nix/store/7gkya4n7b0jj12r3k5v2x9gizwqhlbgm-icu4c-70.1/
nix/store/8l8cfgxafnl8hb1z6w2z64pg15w7qg0k-openssl-1.1.1n-dev/
nix/store/9l06npv9sp8avdraahzi4kqhcp607d8p-tzdata-2022a/
nix/store/a0k6rfn47h9f69p15pg415x6pfpxhsl5-gdbm-1.23/
nix/store/a5xpjds3mlln26469h72v1jmd00jq6lv-xz-5.2.5/
nix/store/ayrsyv7npr0lcbann4k9lxr19x813f0z-glibc-2.34-115/
nix/store/b36ilvc5hhfpcp7kv1kvrkgcxxpmxfsd-zlib-1.2.12/
nix/store/clkdigybx5w29rjxnwnsk76q49gb12k7-ncurses-6.3/
nix/store/fcd0m68c331j7nkdxvnnpb8ggwsaiqac-bash-5.1-p16/
nix/store/gm6q7jmajjmnwd29wgbq2jm3x37vsw3h-libffi-3.4.2/
nix/store/hgl0ydlkgs6y6hx9h7k209shw3v7z77j-coreutils-9.0/
nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/
nix/store/ik4qlj53grwmg7avzrfrn34bjf6a30ch-libunistring-1.0/
nix/store/n239ln3v669s5fkir2fd8niqawyg6qrv-attr-2.5.1/
nix/store/nr73vd4apmn4sd6mmzw9fwpg0dq82g03-libuv-1.44.1/
nix/store/psijdi9190zgbp053y6dj3ax4y2l70gk-gcc-11.2.0-lib/
nix/store/qd3g8rk5hx5zkb70idjh6fa12sh6bipg-mailcap-2.1.53/
nix/store/qvs678k05yrv566dmqdnxfbzi4s6ir1n-sqlite-3.38.2/
nix/store/rf3j3p8cvn0dr5wdl65ns9f8wnlca8h6-readline-6.3p08/
nix/store/v8vpzh3slc5hm4d9id5bim4dsb4d2ndh-openssl-1.1.1n/
nix/store/v990x4cib4dssspn4778rlz46jmm3a9k-expat-2.4.7/
nix/store/w3zngkrag7vnm7v1q8vnqb71q6a1w8gn-libidn2-2.3.2/
nix/store/xvacdngzsxn6hwnymncs8iv752aal4j0-perl-5.34.1/
nix/store/zf03nlnk9h724gz7qzzbrzyqif8gbwhq-bzip2-1.0.6.0.2/
nix/store/zl4bvsqfxyx5vn9bbhnrmbmpfvzqj4gd-nodejs-16.14.2/
nix/store/zw7vyav2vrmxm3m3a990i23fx9ig0pj8-openssl-1.1.1n-bin/

About 300 MiB untarred, but 100 MiB as a .tar.gz. You can see gcc is still included there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants