NiJS: An internal DSL for Nix in JavaScript

Many programming language environments provide their own language specific package manager, implementing features that are already well supported by generic ones. This package is a response to that phenomenon.

This package contains a library and command-line utility providing an internal DSL for the Nix package manager in JavaScript. Nix is a generic package manager that borrows concepts from purely functional programming languages to make deployment reliable, reproducible and efficient. It serves as the basis of the NixOS Linux distribution, but it can also be used seperately on regular Linux distributions, FreeBSD, Mac OS X and Windows (through Cygwin).

The internal JavaScript DSL makes it possible for developers to convienently perform deployment operations, such as building, upgrading and installing packages with the Nix package manager from JavaScript programs. Moreover, it offers a number of additional features.

Prerequisites

• In order to use the components in this package, a Node.js installation is required.
• To use the nijs-build and nijs-execute command-line utilities, we require the optparse library to be installed either through Nix or NPM
• The slasp library is required to make asynchronous programming more convenient.
• Of course, since this package provides a feature for Nix, we require the Nix package manager to be installed

Installation

There are two ways this package can installed.

To install this package through the Nix package manager, obtain a copy of Nixpkgs and run:

$ nix-env -f '<nixpkgs>' -iA nodePackages.nijs

Alternatively, this package can also be installed through NPM by running:

$ npm install -g nijs

Usage

This package offers a number of interesting features.

Calling a Nix function from JavaScript

The most important use case of this package is to be able to call Nix functions from JavaScript. To call a Nix function, we must create a simple proxy that translates a JavaScript function call to a string containing a semantically equivalent Nix function invocation.

The following code fragment demonstrates a JavaScript function proxy that relays a call to the stdenv.mkDerivation {} function in Nix:

var nijs = require('nijs');

function mkDerivation(args) {
  return new nijs.NixFunInvocation({
    funExpr: new nijs.NixExpression("pkgs.stdenv.mkDerivation"),
    paramExpr: args
  });
}

As can be observed, the function proxy has a very simple structure. It takes an arbitrary JavaScript object as a parameter and returns a JavaScript object representing a Nix function invocation to stdenv.mkDerivation {} using the args object as an argument.

The conversion to Nix is done automatically by NiJS through a function called jsToNix().

Compiling JavaScript language constructs into Nix language constructs

The jsToNix() function is used by NiJS to transform JavaScript language constructs to Nix expression language constructs.

JavaScript objects are translated into semantically equivalent (or similar) Nix expression language objects as follows:

• null references are translated into null values
• Objects of type boolean and number are translated verbatim
• Objects of type string and xml are translated verbatim and are automatically escaped
• Arrays of objects are recursively translated into lists of objects.
• "Ordinary" objects are recursively translated into attribute sets. Keys are translated into identifiers unless they contain characters not allowing it do to so. If the latter is the case, they are translated into strings.
• A JavaScript Function is wrapped into a function proxy so that it can be invoked from a Nix expression (see section: 'Calling JavaScript functions from Nix expressions')
• Object members whose values are undefined are not included in the generated attribute set. In all other cases undefined translates to null.

Some Nix expression language constructs have no semantic equivalent in JavaScript. Nonetheless, they can be generated by composing an abstract syntax tree of objects that are instances of prototypes that inherit from NixObject:

• An object instance of NixFile can be used to specify a relative or absolute path to a file. Nix checks whether the file exists and imports it into the Nix store.
• To encode URLs, an object instance of NixURL can be used.
• Recursive attribute sets (in which attributes are allowed to refer to each other) can be defined by creating an object instance of the NixRecursiveAttrSet prototype.
• Referring to an attribute of an attribute set can be done by creating an object instance of NixAttrReference
• Defining functions in the Nix expression language instead of JavaScript can be done by instantiating NixFunction.
• A Nix function can be invoked by creating an object that is an instance of the NixFunInvocation prototype.
• An external Nix expression file can be imported by creating a NixImport object that refers to an external file.
• Referring to existing Nix store paths can be done by creating objects that are instances of NixStorePath.
• An if-then-else block can be generated by defining an NixIf object.
• An assert block can be defined with an NixAssert object.
• A let-block containing private values can be defined by means of a NixLet object.
• A value can be imported into the lexical scope of a block by assigning a member of an object, NixLet, or NixRecursiveAttrSet to an object instance of NixInherit.
• Attributes member of an attribute set can be imported into the lexical scope by creating a NixWith object.
• Two attribute sets can be merged by creating a NixMergeAttrs object.
• Defining build instructions in JavaScript (as opposed to bash code embedded in strings) can be done by creating a NixInlineJS object (see section: 'Writing inline JavaScript code in a NiJS package specification').
• To literally do stuff in the Nix expression language, compose objects that are instances of the NixExpression prototype.

Check the API documentation for more details on how to use the above prototypes. More details on the Nix expression language can be found in the Nix manual.

Specifying packages in JavaScript

The stdenv.mkDerivation() function shown earlier is a very important function in Nix. It's directly and indirectly used by nearly every package recipe to perform a build from source code. In the tests/ folder, we have defined a packages repository: pkgs.js that provides a proxy to this function.

By using this proxy we can also describe our own package specifications in JavaScript, instead of the Nix expression language. Every package build recipe can be written as a CommonJS module, that may look as follows:

var nijs = require('nijs');

exports.pkg = function(args) {
  return args.stdenv().mkDerivation ({
    name : "hello-2.10",

    src : args.fetchurl()({
      url : new nijs.NixURL("mirror://gnu/hello/hello-2.10.tar.gz"),
      sha256 : "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
    }),

    doCheck : true,

    meta : {
      description : "A program that produces a familiar, friendly greeting",
      homepage : new nijs.NixURL("http://www.gnu.org/software/hello/manual"),
      license : "GPLv3+"
    }
  });
};

A package module exports the pkg property, which refers to a function definition taking the build-time dependencies of the package as argument object. In the body of the function, we return the result of an invocation to the mkDerivation() function that builds a package from source code. To this function we pass essential build parameters, such as the URL from which the source code can be obtained.

Nix has special types for URLs and files to check whether they are in the valid format and that they are automatically imported into the Nix store for purity. As they are not in the JavaScript language, we can artificially create them through objects that are instances of the NixFile and NixURL prototypes.

Moreover, there are more prototypes for some other Nix expression language constructs that have no JavaScript equivalent. Check the API documentation for more information.

Composing packages

As with ordinary Nix expressions, we cannot use this CommonJS module to build a package directly. We have to compose it by calling it with its required function arguments. Composition is done in the composition module: pkgs.js in the tests/ folder. The structure of this file is as follows:

var pkgs = {

  stdenv : function() {
    return require('./pkgs/stdenv.js').pkg;
  },

  fetchurl : function() {
    return require('./pkgs/fetchurl/fetchurl.js').pkg({
      stdenv : pkgs.stdenv
    });
  },

  hello : function() {
    return require('./pkgs/hello.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl
    });
  },
  
  ...
};

exports.pkgs = pkgs;

The above module exports the pkgs property that refers to an object in which each member refers to a function definition. These functions call the package modules with its required parameters. By evaluating these functions, a Nix expression gets generated that can be built by the Nix package manager.

Building packages programmatically

The callNixBuild() function can be used to build a generated Nix expression:

var nijs = require('nijs');
var pkgs = require('pkgs.js').pkgs;

nijs.callNixBuild({
  nixExpression : pkgs.hello(),
  params : [],
  pkgsExpression : "import <nixpkgs> {}", /* Optional parameter, which defaults to this value */
  callback : function(err, result) {
    if(err) {
      process.stderr.write(err);
      process.exit(1);
    } else {
      process.stdout.write(result + "\n");
    }
  }
});

In the code fragment above we call the callNixBuild function, in which we evaluate the hello package that gets built asynchronously by Nix. The callback function parameter is called when the build has been done, providing either an err parameter that is not null if some error occured or result referring to the resulting Nix store path. Finally, the store path is printed on the standard output.

Building packages through a command-line utility

As the previous code example is so common, there is also a command-line utility that can do the same. The following instruction builds the hello package from the composition module (pkgs.js):

$ nijs-build pkgs.js -A hello

It may also be useful to see what kind of Nix expression is generated for debugging or testing purposes. The --eval-only option prints the generated Nix expression on the standard output:

$ nijs-build pkgs.js -A hello --eval-only

We can also nicely format the generated expression to improve readability:

$ nijs-build pkgs.js -A hello --eval-only --format

Building NiJS packages from a Nix expression

We can also call the composition CommonJS module from a Nix expression. This is useful to build NiJS packages from Hydra, a continuous build and integration server built around Nix.

The following Nix expression builds the hello package defined in pkgs.js shown earlier:

{nixpkgs, system, nijs}:

let
  nijsImportPackage = import "${nijs}/lib/node_modules/nijs/lib/importPackage.nix" {
    inherit nixpkgs system nijs;
  };
in
{
  hello = nijsImportPackage { pkgsJsFile = ./tests/pkgs/pkgs.js; attrName = "hello"; };
  ...
}

Calling JavaScript functions from Nix expressions

Another use case of NiJS is to call JavaScript functions from Nix expressions. This can be done by using the nijsFunProxy Nix function. The following code fragment shows a Nix expression using the sum() JavaScript function to add two integers and writes the result to a text file in the Nix store:

{stdenv, lib, nodejs, nijs}:

let
  nijsFunProxy = import "${nijs}/lib/node_modules/nijs/lib/funProxy.nix" {
    inherit stdenv lib nodejs nijs;
  };

  sum = a: b: nijsFunProxy {
    name = "sum"; # Optional, but allows one to read function names in the traces if an error occurs
    function = ''
      function sum(a, b) {
        return a + b;
      }
    '';
    args = [ a b ];
  };
in
stdenv.mkDerivation {
  name = "sum";

  buildCommand = ''
    echo ${toString (sum 1 2)} > $out
  '';
}

As can be observed, the nijsFunProxy is a very thin layer that propagates the Nix function parameters to the JavaScript function (Nix objects are converted to JavaScript object) and the resulting JavaScript object is translated back to a Nix expression.

JavaScript function calls have a number of caveats that a developer should take in mind:

• We cannot use variables outside the scope of the function, e.g. global variables.
• We must always return something. If nothing is returned, we will have an undefined object, which cannot be converted.
• Functions with a variable number of positional arguments are not supported, as Nix functions don't support this.

Using CommonJS modules in embedded JavaScript functions

It may be very annoying to only use self contained JavaScript code fragments, as we cannot access anything outside the function's scope. Fortunately, the nijsFunProxy can also take Node packages through Nix as parameters and make them available to that function. The following code fragment uses the Underscore library to convert a list of integers to a list of strings:

{stdenv, lib, nodejs, nijs, underscore}:

let
  nijsFunProxy = import "${nijs}/lib/node_modules/nijs/lib/funProxy.nix" {
    inherit stdenv lib nodejs nijs;
  };

  underscoreTestFun = numbers: nijsFunProxy {
    name = "underscoreTestFun";
    function = ''
      function underscoreTestFun(numbers) {
        var words = [ "one", "two", "three", "four", "five" ];
        var result = [];

        _.each(numbers, function(elem) {
          result.push(words[elem - 1]);
        });

        return result;
      }
    '';
    args = [ numbers ];
    modules = [ underscore ];
    requires = [
      { var = "_"; module = "underscore"; }
    ];
  };
in
stdenv.mkDerivation {
  name = "underscoreTest";

  buildCommand = ''
    echo ${toString (underscoreTestFun [ 5 4 3 2 1 ])} > $out
  '';
}

The modules parameter is a list of Node.JS packages (provided through Nixpkgs) and the require parameter is a list taking var and module pairs. The latter parameter is used to automatically generate a collection of require statements. In this case, it adds the following line before the function definition:

var _ = require('underscore');

Calling asynchronous JavaScript functions from Nix expressions

In Node.js, most of the standard utility functions are asynchronous, which will return immediately, and invoke a callback function when the task is done. To allow these functions to be used inside a Nix expression, we must set the async parameter to true in the nijsFunProxy. Furthermore, instead of returning an object or throwing an exception, we must call the nijsCallbacks.callback(err, result) function.

The following example uses a timer that calls the success callback function after three seconds, with a standard greeting message:

{stdenv, lib, nodejs, nijs}:

let
  nijsFunProxy = import "${nijs}/lib/node_modules/nijs/lib/funProxy.nix" {
    inherit stdenv lib nodejs nijs;
  };

  timerTest = message: nijsFunProxy {
    name = "timerTest";
    function = ''
      function timerTest(message) {
        setTimeout(function() {
          nijsCallbacks.callback(null, message);
        }, 3000);
      }
    '';
    args = [ message ];
    async = true;
  };
in
stdenv.mkDerivation {
  name = "timerTest";

  buildCommand = ''
    echo ${timerTest "Hello world! The timer test works!"} > $out
  '';
}

Writing inline JavaScript code in Nix expressions

All the examples so far use generic building procedures or refer to JavaScript functions that expose themselves as Nix functions. Sometimes it may also be required to implement a custom build procedure for a package. Nix uses the Bash shell as its default builder, hence it requires developers to implement custom build steps as shell code embedded in strings.

It may also be desired to implement custom build procedure steps as embedded JavaScript code, instead of embedded shell code. The nijsInlineProxy function allows a developer to write inline JavaScript code inside a Nix expression:

{stdenv, lib, writeTextFile, nodejs, nijs}:

let
  nijsInlineProxy = import "${nijs}/lib/node_modules/nijs/lib/inlineProxy.nix" {
    inherit stdenv lib writeTextFile nodejs nijs;
  };
in
stdenv.mkDerivation {
  name = "createFileWithMessage";
  buildCommand = nijsInlineProxy {
    name = "createFileWithMessage-buildCommand"; # Optional, but allows one to read function names in the traces if an error occurs
    requires = [
      { var = "fs"; module = "fs"; }
      { var = "path"; module = "path"; }
    ];
    code = ''
      fs.mkdirSync(process.env['out']);
      var message = "Hello world written through inline JavaScript!";
      fs.writeFileSync(path.join(process.env['out'], "message.txt"), message);
    '';
  };
}

The above example Nix expression implements a custom build procedure that creates a Nix component containing a file named message.txt with a standard greeting message. As you may see, instead of providing a custom buildCommand that contains shell code we invoke the nijsInlineProxy that uses two CommonJS modules. The code implements our custom build procedure in JavaScript.

As with ordinary Nix expressions, the parameters passed to stdenv.mkDerivation and its generic properties are accessible as environment variables inside the builder. In our example, process.env['out'] is an environment variable containing the Nix store output path of our package.

The nijsInlineProxy has the same limitations as the nijsFunProxy, such as the fact that global variables cannot be accessed. Moreover, like the nijsFunProxy it can also take the modules parameter allowing one to utilise external node.js packages.

Writing inline JavaScript code in a NiJS package specification

When implementing a custom build procedure in a NiJS package module, we may also run into the same inconvenience of having to embed custom build steps as shell code embedded in strings. We can also use the nijsInlineProxy from a NiJS package module, by creating an object that is an instance of the NixInlineJS prototype:

var nijs = require('nijs');

exports.pkg = function(args) {
  return args.stdenv().mkDerivation ({
    name : "createFileWithMessageTest",
    buildCommand : new nijs.NixInlineJS({
      requires : [
        { "var" : "fs", "module" : "fs" },
        { "var" : "path", "module" : "path" }
      ],
      code : function() {
        fs.mkdirSync(process.env['out']);
        var message = "Hello world written through inline JavaScript!";
        fs.writeFileSync(path.join(process.env['out'], "message.txt"), message);
      }
    })
  });
};

The above NiJS package module shows the NiJS equivalent of our first Nix expression example containing inline JavaScript code.

The buildCommand parameter is bound to an instance of the NixInlineJS prototype. The code parameter can be either a JavaScript function (that takes no parameters) or a string that contains embedded JavaScript code. The former case (the function approach) has the advantage that its syntax can be checked or visualised by an editor, interpreter or compiler.

Specifying packages asynchronously

Earlier, we have shown that we can describe package specifications in JavaScript. The example shown previously (specifying GNU Hello) is a synchronous package specification. We can also specify packages asynchronously, for example:

var nijs = require('nijs');
var slasp = require('slasp');

exports.pkg = function(args, callback) {
  slasp.sequence([
    function(callback) {
      args.fetchurl()({
        url : new nijs.NixURL("mirror://gnu/hello/hello-2.8.tar.gz"),
        sha256 : "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6"
      }, callback);
    },

    function(callback, src) {
      args.stdenv().mkDerivation ({
        name : "hello-2.8",
        src : src,
 
        doCheck : true,
        meta : {
          description : "A program that produces a familiar, friendly greeting",
          homepage : new nijs.NixURL("http://www.gnu.org/software/hello/manual"),
          license : "GPLv3+"
        }
      }, callback);
    }
  ], callback);
};

The above expression has the same meaning as the synchronous GNU Hello expression, but implements an interface using callbacks. Moreover, it uses the slasp library to flatten the code structure to make it better readable and maintainable.

Composing packages asynchronously

Asynchronous packages also have to be composed by passing the required dependencies of a package as function parameters:

var pkgs = {

  stdenv : function(callback) {
    return require('./pkgs-async/stdenv.js').pkg;
  },
  
  fetchurl : function(callback) {
    return require('./pkgs-async/fetchurl').pkg({
      stdenv : pkgs.stdenv
    }, callback);
  },
  
  hello : function(callback) {
    return require('./pkgs-async/hello.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl
    }, callback);
  },

  ...
};

exports.pkgs = pkgs;

The above composition module has the same meaning as the synchronous composition module shown earlier. The minor difference is that all functions provide a callback interface.

Building asynchronous packages through a command-line utility

The NiJS build tool can also compile asynchronous packages to Nix expressions and build them with Nix.

To allow the asynchronous modules to be recognized we need to pass the --async parameter to nijs-build:

$ nijs-build pkgs-async.js -A hello --async

Executing asynchronous package specifications directly

Another use case of asynchronous packages (which cannot be done with synchronous package specifications) is that they can be built directly, without using the Nix package manager.

The following command uses NiJS to build the same package:

$ nijs-execute pkgs-async.js -A hello

Packages are stored in a so-called NiJS store. By default, this folder resides in $HOME/.nijs/store but NiJS can be configured to store it elsewhere.

Although NiJS is capable of building packages, its features and facilities are quite primitive. It should only be used for simple (test) use cases or as a toy.

Building asynchronous NiJS packages from a Nix expression

As with synchronous package specifications, we can also call asynchronous composition modules from Nix expressions.

To do this the nijsImportPackageAsync {} function should be used:

{nixpkgs, system, nijs}:

let
  nijsImportPackageAsync = import "${nijs}/lib/node_modules/nijs/lib/importPackageAsync.nix" {
    inherit nixpkgs system nijs;
  };
in
{
  hello = nijsImportPackageAsync { pkgsJsFile = ./tests/pkgs/pkgs-async.js; attrName = "hello"; };
  ...
}

Transforming custom object structures into Nix expressions

As explained earlier, NiJS transforms objects in the JavaScript language to semantically equivalent (or similar) constructs in the Nix expression language.

Sometimes it may also be desired to generate Nix expressions from a domain model, designed for solving a specific non-deployment related problem, with properties and structures that cannot be literally translated into a representation in the Nix expression language.

It is also possible to specify for an object how to generate a Nix expression from it. This can be done by inheriting from the NixASTNode prototype and overriding the toNixAST method.

For example, we may have a system already providing a representation of a file that should be downloaded from an external source:

function HelloSourceModel(args) {
    this.args = args;
    this.src = "mirror://gnu/hello/hello-2.10.tar.gz";
    this.sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
}

The above module defines a constructor function composing an object that refers to the GNU Hello package provided by a GNU mirror site.

A direct translation of the above constructed object to the Nix expression language does not provide anything meaningful -- it can, for example, not be used to let Nix fetch the package from the mirror site.

We can inherit from NixASTNode and implement our own custom toNixAST() function to provide a more meaningful Nix translation:

var nijs = require('nijs');
var inherit = require('nijs/lib/ast/util/inherit.js').inherit;

/* HelloSourceModel inherits from NixASTNode */
inherit(nijs.NixASTNode, HelloSourceModel);

/**
 * @see NixASTNode#toNixAST
 */
HelloSourceModel.prototype.toNixAST = function() {
    return this.args.fetchurl()({
        url: new nijs.NixURL(this.src),
        sha256: this.sha256
    });
};

The toNixAST() function shown above composes an abstract syntax tree (AST) for a function invocation to fetchurl {} in the Nix expression language with the url and sha256 properties a parameters.

An object that inherits from the NixASTNode prototype also indirectly inherits from NixObject. This means that we can directly attach such an object to any other AST object. The generator uses the underlying toNixAST() function to automatically convert it to its AST representation.

For example, we can also define the GNU Hello package as an object composed by a constructor function:

function HelloModel(args) {
    this.args = args;

    this.name = "hello-2.10";
    this.source = new HelloSourceModel(args);
    this.meta = {
        description: "A program that produces a familiar, friendly greeting",
        homepage: "http://www.gnu.org/software/hello/manual",
        license: "GPLv3+"
    };
}

In the above function, we construct a build recipe for GNU Hello using a convention that is not directly usable in Nix. The object also has a reference to a source object composed by the constructor function shown in the previous example.

By inheriting from NixASTNode and overriding toNixAST() we can construct an AST for a Nix expression that builds the package:

/* HelloModel inherits from NixASTNode */
inherit(nijs.NixASTNode, HelloModel);

/**
 * @see NixASTNode#toNixAST
 */
HelloModel.prototype.toNixAST = function() {
    return this.args.stdenv().mkDerivation ({
        name: this.name,
        src: this.source,
        doCheck: true,
        meta: this.meta
    });
};

The above function constructs an AST for a function invocation to stdenv.mkDerivation {}, using the object's properties a parameters. It directly refers to the source object (this.source) without any conversions -- because the source object inherits from NixAST and (indirectly) from NixObject, the generator will automatically convert it to an AST for a fetchurl {} function invocation.

In some cases, it may not be possible to inherit from NixASTNode, for example, when the object already inherits from another prototype that is beyond the user's control.

It is also possible to use the NixASTNode constructor function as an adapter. For example, we can take any object with a toNixAST() function (such as an object creating a wrapper for the metadata):

var self = this;

var metadataWrapper = {
    toNixAST: function() {
        return {
            description: self.meta.description,
            homepage: new nijs.NixURL(self.meta.homepage),
            license: self.meta.license
        };
    }
};

By wrapping the metadataWrapper object in the NixASTNode constructor, we can convert it to an object that is an instance of NixASTNode:

new nijs.NixASTNode(metadataWrapper)

Examples

The tests/ directory contains a number of interesting example cases:

• pkgs.js is a composition module containing a collection of NiJS packages
• proxytests.nix is a composition Nix expression containing a collection of JavaScript function invocations from Nix expressions
• pkgs-async.js is a composition module containing a collection of NiJS packages that are specified asynchronously

API documentation

This package includes API documentation, which can be generated with JSDoc.

License

The contents of this package is available under the MIT license