Translate NPM packages to Nix expressions.
What it does
Given the name of one or more packages and an output directory, queries NPM repositories for those packages' definitions and those of their dependencies, and generates a set of nix expressions in the given output directory which not only can be used to build the requested packages, but also to build any of their dependencies as desired.
- The packages generated are easily human readable.
- They can be modified as desired after they are built, and these modifications will remain in place. For example, if a package relies on non-NPM dependencies, or requires extra build steps such as patching, these changes can be added by hand and will be respected down the line.
- Since builds use pre-existing packages, repeated expression generation is fast. For example, if you generate the expression for one package, and then generate the expression for another package which shares dependencies with the first (even if the second is built at a later time), the shared dependencies will not be regenerated.
npm2nix is another tool which can generate nix expressions from an
npm package. However, it has several drawbacks. It generates the
entire dependency tree for an npm package, without availing itself of
results of previous invocations. This is inefficient, since duplicated
packages are built multiple times. The resulting expression is a
single monolithic file which is hard to grok, and hard to
modify. Furthermore, any modifications performed would have to be done
each time the package was regenerated. It also discourages committing
of the resulting package into source control, since it's large and has
to be continually regenerated whenever changes are made. This means
that packages built with it are unlikely to be cached in a nix store
$ git clone https://github.com/adnelson/nixfromnpm $ cd nixfromnpm
Make sure you have nix installed, and
nixpkgs is in your
environment variable. Then run:
$ nix-env --install --attr nixfromnpm --file ./release.nix
If you'd like to try out
nixfromnpm without installing it, or just
hack on it, you can use it in a
$ cd /path/to/nixfromnpm $ nix-shell [nix-shell:nixfromnpm]$ cabal run -- <arguments>
nixpkgs and GHC versions
By default, we pin the version of
nixpkgs in order to maintain a
reliable build. However, if you'd like to build off of
NIX_PATH or some other custom location:
$ nix-env -f release.nix -iA nixfromnpm --arg nixpkgs '<nixpkgs>'
The GHC version can also be set explicitly, although of course, the package is not guaranteed to build with any arbitrary GHC version:
$ nix-env -f release.nix -iA nixfromnpm --argstr compiler ghc802
Note that the above options also apply to
I recommend using the
repo for most applications. This repo contains several thousand node
package definitions already, which means whatever you're trying to
build might already be defined. It also contains some packages which
are hand-written or hand-modified from what had been auto-generated by
nixfromnpm (fixing bugs or performing additional build steps). To use
the repo, clone it and then specify it as an "output" when calling
nixfromnpm. Of course, if the package already exists you can just build
it immediately with
$ git clone https://github.com/adnelson/nix-node-packages $ nixfromnpm -o nix-node-packages -p 'the-package-I-need@the-version-I-need' $ nix-build nix-node-packages -A nodePackages.the-package-I-need_the-version-I-need
See that repo's
README.md for information.
If you don't use the repo, the commands below will still work, but some of the packages which the repo provides fixes for might fail to build.
Generating an expression for a package
The most basic usage is providing an
--output) flag and one or
$ nixfromnpm -o /some/path -p package_name -p other_package_name
This will build the packages called
other_package_name, and put all of the generated expressions in
/some/path. That path will be created if it doesn't exist. If the
output path does exist, a package will only be fetched if a nix
expression for it doesn't already exist.
You can also specify a version bound on the packages you are fetching,
$ nixfromnpm -p package_name@version_bound -o /some/path
Any NPM version bound is valid; so for example:
$ nixfromnpm -p email@example.com -o /some/path $ nixfromnpm -p 'foo@>=0.8 <0.9' -o /some/path $ nixfromnpm -p 'foo@~1.0.0' -o /some/path
Generating an expression from a package.json file
You can also generate an expression for a project on the local disk by
passing in the path to a
package.json file, or a directory
containing one. The generated expression will be placed in a file
project.nix in the same directory as the
default.nix will be created in the directory as well which
project.nix. As with normal usage, the
-o flag is used
to specify a path to where generated expressions will be placed;
however, only the downsteam dependencies will be put here, while the
expression itself will be in the
$ nixfromnpm -f /path/to/package.json -o /path/to/dependency/set
You can give multiple
-f arguments to build multiple expressions on
disk, and it can be used alongside
-p arguments as well.
nixfromnpm lets you generate expressions for development
dependencies at a maximum depth. For example, depth
0 means don't
make any development dependency expressions; depth
1 means create
expressions for the package being built, but not any of their
development dependencies, etc.
$ nixfromnpm -p package_name -o /some/path --dev-depth 1
For a package in a private registry located at
$ nixfromnpm -p private_package -o /some/path -r https://my.registry:2345
For npm packages which fetch from git, if an authorization token is required:
$ nixfromnpm -p package -o /some/path --github-token llnl23uinlaskjdno34nedhoaidjn5o48wugn
This can also be set by a
GITHUB_TOKEN environment variable.
Private NPM namespaces
NPM offers private packaging, where you can specify a namespace or
scope under which a package lives. For example, a package might be
@foo/bar, where the package is called
bar and the
namespace is called
nixfromnpm supports this, as long as you
have set up your NPM repo to use authorization tokens (see
documentation for details). To use this, set an environment variable
NPM_AUTH_TOKENS, with the following format:
Where tokens are keyed on namespaces. Then when building the
expression set, if a namespaced package is encountered,
will look up the namespace in this environment variable to determine
what token to use for authentication. The same environment variable is
read when the packages are built with
nix-build. The path to the
package in the generated expressions is slightly different:
$ export NPM_AUTH_TOKENS="foo=a1a1a1a1a1a1a1a1a1a" $ nixfromnpm -o my_expressions -p '@foo/bar' $ nix-build my_expressions -A namespaces.foo.bar
Caching of packages
nixfromnpm will discover all existing packages in the
specified output directory (provided via tha
-o flag). However, if
you would like to generate all of these from scratch, you can disable
Troubleshooting a package that doesn't build
There are any number of reasons why a package might not build. Some of the most common ones are:
nixfromnpmtool wasn't able to generate the definition of one of the package's dependencies. It will insert in the
brokenPackagefunction, which, as might be anticipated, never builds. Looking at the call to
brokenPackagewill tell you why it couldn't build it. In my experience, this is because
nixfromnpm's version range checker is not completely up to spec, and it's unable to find a version that satisfies the bounds given by a
package.json. If this is the case, the easiest way to fix it is to:
- See what version range
nixfromnpmfailed to resolve. E.g.
npmto manually build the package at the given version bounds. E g.
npm install foo@>=1.2.3-bar <2.3.4-baz.qux.
- See what version it ends up building. E.g.
nixfromnpmon that version. E.g.
nixfromnpm -o /path/to/nix-node-packages -p 'firstname.lastname@example.org'.
- Replace the call to
- See what version range
- The build fails with
npmcomplaining about HTTP errors. This is usually caused by a dependency that wasn't satified, likely because
nixfromnpmcalculated the wrong dependency. In this case, use steps similar to the above to find out what the actual dependency should be, and modify the package definition to include the correct one.
- A package build script is attempting to do some hacky bullshit like modifying its dependencies. This, of course, is not kosher in the
nixview of things. In this case, you'll probably want to
nix-shellinto the package and see what it's trying to do. Figure out how to stop it from doing these things, and supply
postPatchsteps to apply those changes.
The good news is that if you patch or otherwise fix a broken package, it will not be overwritten by subsequent invocations of
nixfromnpm (although, I highly recommend keeping your expressions in source control in case bad things happen!).
This project has gone from what I thought would be a weekend project to a pretty significant undertaking. I think it has the potential to be pretty useful to JS developers who are interested in using nix, and it's already usable, with thousands of packages defined (although there's no guarantee, of course, that these all work). However, there are still a number of issues which need to be addressed. I very heartily welcome contributions, whether error reporting or pull requests.