Translate NPM packages to Nix expressions.
Given the name of one or more packages and an output directory, queries NPM repositories for those packages' definitions and those of their dependencies, and generates a set of nix expressions in the given output directory which not only can be used to build the requested packages, but also to build any of their dependencies as desired.
- The packages generated are easily human readable.
- They can be modified as desired after they are built, and these modifications will remain in place. For example, if a package relies on non-NPM dependencies, or requires extra build steps such as patching, these changes can be added by hand and will be respected down the line.
- Since builds use pre-existing packages, repeated expression generation is fast. For example, if you generate the expression for one package, and then generate the expression for another package which shares dependencies with the first (even if the second is built at a later time), the shared dependencies will not be regenerated.
npm2nix
is another tool which can generate nix expressions from an
npm package. However, it has several drawbacks. It generates the
entire dependency tree for an npm package, without availing itself of
results of previous invocations. This is inefficient, since duplicated
packages are built multiple times. The resulting expression is a
single monolithic file which is hard to grok, and hard to
modify. Furthermore, any modifications performed would have to be done
each time the package was regenerated. It also discourages committing
of the resulting package into source control, since it's large and has
to be continually regenerated whenever changes are made. This means
that packages built with it are unlikely to be cached in a nix store
or repo.
Clone the nixfromnpm
repo.
$ git clone https://github.com/adnelson/nixfromnpm
$ cd nixfromnpm
Make sure you have nix installed, and nixpkgs
is in your NIX_PATH
environment variable. Then run:
$ nix-env --install --attr nixfromnpm --file ./release.nix
If you'd like to try out nixfromnpm
without installing it, or just
hack on it, you can use it in a nix-shell
:
$ cd /path/to/nixfromnpm
$ nix-shell
[nix-shell:nixfromnpm]$ cabal run -- <arguments>
By default, we pin the version of nixpkgs
in order to maintain a
reliable build. However, if you'd like to build off of nixpkgs
in
your NIX_PATH
or some other custom location:
$ nix-env -f release.nix -iA nixfromnpm --arg nixpkgs '<nixpkgs>'
The GHC version can also be set explicitly, although of course, the package is not guaranteed to build with any arbitrary GHC version:
$ nix-env -f release.nix -iA nixfromnpm --argstr compiler ghc802
Note that the above options also apply to nix-build
, nix-shell
, etc.
I recommend using the
nix-node-packages
repo for most applications. This repo contains several thousand node
package definitions already, which means whatever you're trying to
build might already be defined. It also contains some packages which
are hand-written or hand-modified from what had been auto-generated by
nixfromnpm
(fixing bugs or performing additional build steps). To use
the repo, clone it and then specify it as an "output" when calling
nixfromnpm
. Of course, if the package already exists you can just build
it immediately with nix-build
.
$ git clone https://github.com/adnelson/nix-node-packages
$ nixfromnpm -o nix-node-packages -p 'the-package-I-need@the-version-I-need'
$ nix-build nix-node-packages -A nodePackages.the-package-I-need_the-version-I-need
See that repo's README.md
for information.
If you don't use the repo, the commands below will still work, but some of the packages which the repo provides fixes for might fail to build.
The most basic usage is providing an -o
(--output
) flag and one or
more -p
flags:
$ nixfromnpm -o /some/path -p package_name -p other_package_name
This will build the packages called package_name
and
other_package_name
, and put all of the generated expressions in
/some/path
. That path will be created if it doesn't exist. If the
output path does exist, a package will only be fetched if a nix
expression for it doesn't already exist.
You can also specify a version bound on the packages you are fetching,
using @
:
$ nixfromnpm -p package_name@version_bound -o /some/path
Any NPM version bound is valid; so for example:
$ nixfromnpm -p foo@0.8.6 -o /some/path
$ nixfromnpm -p 'foo@>=0.8 <0.9' -o /some/path
$ nixfromnpm -p 'foo@~1.0.0' -o /some/path
You can also generate an expression for a project on the local disk by
passing in the path to a package.json
file, or a directory
containing one. The generated expression will be placed in a file
called project.nix
in the same directory as the package.json
file,
and a default.nix
will be created in the directory as well which
calls into project.nix
. As with normal usage, the -o
flag is used
to specify a path to where generated expressions will be placed;
however, only the downsteam dependencies will be put here, while the
expression itself will be in the project.nix
file.
$ nixfromnpm -f /path/to/package.json -o /path/to/dependency/set
You can give multiple -f
arguments to build multiple expressions on
disk, and it can be used alongside -p
arguments as well.
NPM packages differentiate between dependencies needed at runtime and dependencies required only when developing on a package (e.g. packages for testing, transpilers for compile-to-javascript languages, etc). A package might have a great number of development dependencies, and there might be circular dependencies with development packages because the packages are not required at runtime. For this reason it's generally better to avoid generating expressions for development dependencies unless they're needed, because they add a lot of extra work and generate a lot of files.
nixfromnpm
lets you generate expressions for development
dependencies at a maximum depth. For example, depth 0
means don't
make any development dependency expressions; depth 1
means create
expressions for the package being built, but not any of their
development dependencies, etc.
$ nixfromnpm -p package_name -o /some/path --dev-depth 1
For a package in a private registry located at https://my.registry:2345
:
$ nixfromnpm -p private_package -o /some/path -r https://my.registry:2345
For npm packages which fetch from git, if an authorization token is required:
$ nixfromnpm -p package -o /some/path --github-token llnl23uinlaskjdno34nedhoaidjn5o48wugn
This can also be set by a GITHUB_TOKEN
environment variable.
NPM offers private packaging, where you can specify a namespace or
scope under which a package lives. For example, a package might be
designated as @foo/bar
, where the package is called bar
and the
namespace is called foo
. nixfromnpm
supports this, as long as you
have set up your NPM repo to use authorization tokens (see
documentation for details). To use this, set an environment variable
NPM_AUTH_TOKENS
, with the following format:
mynamespace=mytoken:myothernamespace=myothertoken:...
Where tokens are keyed on namespaces. Then when building the
expression set, if a namespaced package is encountered, nixfromnpm
will look up the namespace in this environment variable to determine
what token to use for authentication. The same environment variable is
read when the packages are built with nix-build
. The path to the
package in the generated expressions is slightly different:
$ export NPM_AUTH_TOKENS="foo=a1a1a1a1a1a1a1a1a1a"
$ nixfromnpm -o my_expressions -p '@foo/bar'
$ nix-build my_expressions -A namespaces.foo.bar
By default, nixfromnpm
will discover all existing packages in the
specified output directory (provided via tha -o
flag). However, if
you would like to generate all of these from scratch, you can disable
caching with --no-cache
.
There are any number of reasons why a package might not build. Some of the most common ones are:
- The
nixfromnpm
tool wasn't able to generate the definition of one of the package's dependencies. It will insert in thebrokenPackage
function, which, as might be anticipated, never builds. Looking at the call tobrokenPackage
will tell you why it couldn't build it. In my experience, this is becausenixfromnpm
's version range checker is not completely up to spec, and it's unable to find a version that satisfies the bounds given by apackage.json
. If this is the case, the easiest way to fix it is to:- See what version range
nixfromnpm
failed to resolve. E.g.foo@>=1.2.3-bar <2.3.4-baz.qux
. - Use
npm
to manually build the package at the given version bounds. E g.npm install foo@>=1.2.3-bar <2.3.4-baz.qux
. - See what version it ends up building. E.g.
foo@1.2.3-xyz
. - Call
nixfromnpm
on that version. E.g.nixfromnpm -o /path/to/nix-node-packages -p 'foo@1.2.3-xyz'
. - Replace the call to
brokenPackage
withfoo_1-2-3-xyz
.
- See what version range
- The build fails with
npm
complaining about HTTP errors. This is usually caused by a dependency that wasn't satified, likely becausenixfromnpm
calculated the wrong dependency. In this case, use steps similar to the above to find out what the actual dependency should be, and modify the package definition to include the correct one. - A package build script is attempting to do some hacky bullshit like modifying its dependencies. This, of course, is not kosher in the
nix
view of things. In this case, you'll probably want tonix-shell
into the package and see what it's trying to do. Figure out how to stop it from doing these things, and supplyprePatch
orpostPatch
steps to apply those changes.
The good news is that if you patch or otherwise fix a broken package, it will not be overwritten by subsequent invocations of nixfromnpm
(although, I highly recommend keeping your expressions in source control in case bad things happen!).
This project has gone from what I thought would be a weekend project to a pretty significant undertaking. I think it has the potential to be pretty useful to JS developers who are interested in using nix, and it's already usable, with thousands of packages defined (although there's no guarantee, of course, that these all work). However, there are still a number of issues which need to be addressed. I very heartily welcome contributions, whether error reporting or pull requests.