NixOS Planet

nixbuild.net: Introducing nixbuild.net

support@nixbuild.net (nixbuild.net) — Tue, 18 Feb 2020 00:00:00 +0000

Exactly one month ago, I announced the nixbuild.net service. Since then, there have been lots of work on functionality, performance and stability of the service. As of today, nixbuild.net is exiting alpha and entering private beta phase. If you want to try it out, just send me an email.

Today, I’m also launching the nixbuild.net blog, which is intended as an outlet for anything related to the nixbuild.net service. Announcements, demos, technical articles and various tips and tricks. We’ll start out with a proper introduction of nixbuild.net; why it was built, what it can help you with and what the long-term goals are.

+ Craige McWhirter: Building Daedalus Flight on NixOS + http://mcwhirter.com.au//craige/blog/2020/Building_Daedalus_Flight_on_NixOS/ + http://mcwhirter.com.au//craige/blog/2020/Building_Daedalus_Flight_on_NixOS/ +

+ +

Daedalus Flight was recently released +and this is how you can build and run this version of +Deadalus on NixOS.

+ +

If you want to speed the build process up, you can add the +IOHK Nix cache to your own NixOS configuration:

+ +

iohk.nix:

+ +

nix.binaryCaches = [
+  "https://cache.nixos.org"
+  "https://hydra.iohk.io"
+];
+nix.binaryCachePublicKeys = [
+  "hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ="
+];
+

Why nixbuild.net?

Nix has great built-in support for distributing builds to remote machines. You just need to setup a standard Nix enviroment on your build machines, and make sure they are accessible via SSH. Just like that, you can offload your heavy builds to a couple of beefy build servers, saving your poor laptop’s fan from spinning up.

However, just when you’ve tasted those sweet distributed builds you very likely run into the issue of scaling.

What if you need a really big server to run your builds, but only really need it once or twice per day? You’ll be wasting a lot of money keeping that build server available.

And what if you occasionally have lots and lots of builds to run, or if your whole development team wants to share the build servers? Then you probably need to add more build servers, which means more wasted money when they are not used.

So, you start looking into auto-scaling your build servers. This is quite easy to do if you use some cloud provider like AWS, Azure or GCP. But, this is where Nix will stop cooperating with you. It is really tricky to get Nix to work nicely together with an auto-scaled set of remote build machines. Nix has only a very coarse view of the “current load” of a build machine and can therefore not make very informed decisions on exactly how to distribute the builds. If there are multiple Nix instances (one for each developer in your team) fighting for the same resources, things get even trickier. It is really easy to end up in a situation where a bunch of really heavy builds are fighting for CPU time on the same build server while the other servers are idle or running lightweight build jobs.

If you use Hydra, the continous build system for Nix, you can find scripts for using auto-scaled AWS instances, but it is still tricky to set it up. And in the end, it doesn’t work perfectly since Nix/Hydra has no notion of “consumable” CPU/memory resources so the build scheduling is somewhat hit-and-miss.

Even if you manage to come up with a solution that can handle your workload in an acceptable manner, you now have a new job: maintaining uniquely configured build servers. Possibly for your whole company.

Through my consulting company, Immutable Solutions, I’ve done a lot of work on Nix-based deployments, and I’ve always struggled with half-baked solutions to the Nix build farm problem. This is how the idea of the nixbuild.net service was born — a service that can fill in the missing pieces of the Nix distributed build puzzle and package it as a simple, no-maintenance, cost-effective service.

Who are We?

nixbuild.net is developed and operated by me (Rickard Nilsson) and my colleague David Waern. We both have extensive experience in building Nix-based solutions, for ourselves and for various clients.

We’re bootstrapping nixbuild.net, and we are long-term committed to keep developing and operating the service. Today, nixbuild.net can be productively used for its main purpose — running Nix builds in a scalable and cost-effective way — but there are lots of things that can (and will) be built on top of and around that core. Read more about this below.

What does nixbuild.net Look Like?

To the end-user, a person or team using Nix for building software, nixbuild.net behaves just like any other remote build machine. As such, you can add it as an entry in your /etc/nix/machines file:

beta.nixbuild.net x86_64-linux - 100 1 big-parallel,benchmark

The big-parallel,benchmark assignment is something that is called system features in Nix. You can use that as a primitive scheduling strategy if you have multiple remote machines. Nix will only submit builds that have been marked as requiring a specific system feature to machines that are assigned that feature.

The number 100 in the file above tells Nix that it is allowed to submit up to 100 simultaneous builds to beta.nixbuild.net. Usually, you use this property to balance builds between remote machines, and to make sure that a machine doesn’t run too many builds at the same time. This works OK when you have rather homogeneous builds, and only one single Nix client is using a set of build servers. If multiple Nix clients use the same set of build servers, this simplistic scheduling breaks down, since a given Nix client loses track on how many builds are really running on a server.

However, when you’re using nixbuild.net, you can set this number to anything really, since nixbuild.net will take care of the scheduling and scaling on its own, and it will not let multiple Nix clients step on each other’s toes. In fact each build that nixbuild.net runs is securely isolated from other builds and by default gets exclusive access to the resources (CPU and memory) it has been assigned.

Apart from setting up the distributed Nix machines, you need to configure SSH. When you register an account on nixbuild.net, you’ll provide us with a public SSH key. The corresponding private key is used for connecting to nixbuild.net. This private key needs to be readable by the user that runs the Nix build. This is usually the root user, if you have a standard Nix setup where the nix-daemon process runs as the root user.

That’s all there is to it, now we can run builds using nixbuild.net!

Let’s try building the following silly build, just so we can see some action:

let pkgs = import <nixpkgs> { system = "x86_64-linux"; };
+If you haven't already, you can clone the Daedalus
+repo and specifically the
+1.0.0 tagged commit:
 
-in pkgs.runCommand "silly" {} ''
-  n=0
-  while (($n < 12)); do
-    date | tee -a $out
-    sleep 10
-    n=$(($n + 1))
-  done
-''

This build will run for 2 minutes and output the current date every ten seconds:

$ nix-build silly.nix
-these derivations will be built:
-  /nix/store/cy14fc13d3nzl65qp0sywvbjnnl48jf8-silly.drv
-building '/nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv' on 'ssh://beta.nixbuild.net'...
-Mon Feb 17 20:53:47 UTC 2020
-Mon Feb 17 20:53:57 UTC 2020
-Mon Feb 17 20:54:07 UTC 2020

You can see that Nix is telling us that the build is running on nixbuild.net!

The nixbuild.net Shell

nixbuild.net supports a simple shell interface that you can access through SSH. This shell allows you to retrieve information about your builds on the service.

For example, we can list the currently running builds:

$ ssh beta.nixbuild.net shell
-nixbuild.net> list builds --running
-10524 2020-02-17 21:05:20Z [40.95s] [Running]
-      /nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv

We can also get information about any derivation or nix store path that has been built:

nixbuild.net> show drv /nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv
-Derivation
-  path = /nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv
-  builds = 1
-  successful builds = 1
+$ git clone --branch 1.0.0 https://github.com/input-output-hk/daedalus.git
+
 
-Outputs
-  out -> /nix/store/8c7sndr3npwmskj9zzp4347cnqh5p8q0-silly
+Once you've cloned the repo and checked you're on the 1.0.0 tagged commit,
+you can build Daedalus flight with the following command:
 
-Builds
-  10524 2020-02-17 21:05:20Z [02:01] [Built]

This shell is under development, and new features are added continuously. A web-based frontend will also be implemented.

The Road Ahead

To finish up this short introduction to nixbuild.net, let’s talk a bit about our long-term goals for the service.

The core purpose of nixbuild.net is to provide Nix users with pay-per-use distributed builds that are simple to set up and integrate into any workflow. The build execution should be performant and secure.

There are a number of features that basically just are nice side-effects of the design of nixbuild.net:

Building a large number of variants of the same derivation (a build matrix or some sort of parameter sweep) will take the same time as running a single build, since nixbuild.net can run all builds in parallel.
Running repeated builds to find issues related to non-determinism/reproducability will not take longer than running a single build.
A whole team/company can share the same account in nixbuild.net letting builds be shared in a cost-effective way. If everyone in a team delegates builds to nixbuild.net, the same derivation will never have to be built twice. This is similar to having a shared Nix cache, but avoids having to configure a cache and perform network uploads for each build artifact. Of course, nixbuild.net can be combined with a Nix cache too, if desired.

Beyond the above we have lots of thoughts on where we want to take nixbuild.net. I’m not going to enumerate possible directions here and now, but one big area that nixbuild.net is particularly suited for is advanced build analysis and visualisation. The sandbox that has been developed to securely isolate builds from each other also gives us a unique way to analyze exactly how a build behaves. One can imagine nixbuild.net being able give very detailed feedback to users about build bottlenecks, performance regressions, unused dependencies etc.

With that said, our primary focus right now is to make nixbuild.net a robust workhorse for your Nix builds, enabling you to fully embrace Nix without being limited by local compute resources. Please get in touch if you want try out nixbuild.net, or if you have any questions or comments!

Sander van der Burg: A declarative process manager-agnostic deployment framework based on Nix tooling

noreply@blogger.com (Sander van der Burg) — Sat, 15 Feb 2020 20:07:00 +0000

In a previous blog post written two months ago, I have introduced a new experimental Nix-based process framework, that provides the following features:

It uses the Nix expression language for configuring running process instances, including their dependencies. The configuration process is based on only a few simple concepts: function definitions to define constructors that generate process manager configurations, function invocations to compose running process instances, and Nix profiles to make collections of process configurations accessible from a single location.
The Nix package manager delivers all packages and configuration files and isolates them in the Nix store, so that they never conflict with other running processes and packages.
It identifies process dependencies, so that a process manager can ensure that processes are activated and deactivated in the right order.
The ability to deploy multiple instances of the same process, by making conflicting resources configurable.
Deploying processes/services as an unprivileged user.
Advanced concepts and features, such as namespaces and cgroups, are not required.

Another objective of the framework is that it should work with a variety of process managers on a variety of operating systems.

In my previous blog post, I was deliberately using sysvinit scripts (also known as LSB Init compliant scripts) to manage the lifecycle of running processes as a starting point, because they are universally supported on Linux and self contained -- sysvinit scripts only require the right packages installed, but they do not rely on external programs that manage the processes' life-cycle. Moreover, sysvinit scripts can also be conveniently used as an unprivileged user.

I have also developed a Nix function that can be used to more conveniently generate sysvinit scripts. Traditionally, these scripts are written by hand and basically require that the implementer writes the same boilerplate code over and over again, such as the activities that start and stop the process.

The sysvinit script generator function can also be used to directly specify the implementation of all activities that manage the life-cycle of a process, such as:


{createSystemVInitScript, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSystemVInitScript {
  name = instanceName;
  description = "Nginx";
  activities = {
    start = ''
      mkdir -p ${nginxLogDir}
      log_info_msg "Starting Nginx..."
      loadproc ${nginx}/bin/nginx -c ${configFile} -p ${stateDir}
      evaluate_retval
    '';
    stop = ''
      log_info_msg "Stopping Nginx..."
      killproc ${nginx}/bin/nginx
      evaluate_retval
    '';
    reload = ''
      log_info_msg "Reloading Nginx..."
      killproc ${nginx}/bin/nginx -HUP
      evaluate_retval
    '';
    restart = ''
      $0 stop
      sleep 1
      $0 start
    '';
    status = "statusproc ${nginx}/bin/nginx";
  };
  runlevels = [ 3 4 5 ];

  inherit dependencies instanceName;
}

In the above Nix expression, we specify five activities to manage the life-cycle of Nginx, a free/open source web server:

The start activity initializes the state of Nginx and starts the process (as a daemon that runs in the background).
stop stops the Nginx daemon.
reload instructs Nginx to reload its configuration
restart restarts the process
status shows whether the process is running or not.

Besides directly implementing activities, the Nix function invocation shown above can also be used on a much higher level -- typically, sysvinit scripts follow the same conventions. Nearly all sysvinit scripts implement the activities described above to manage the life-cycle of a process, and these typically need to be re-implemented over and over again.

We can also generate the implementations of these activities automatically from a high level specification, such as:


{createSystemVInitScript, nginx,  stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSystemVInitScript {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile "-p" stateDir ];
  runlevels = [ 3 4 5 ];

  inherit dependencies instanceName;
}

You could basically say that the above createSystemVInitScript function invocation makes the configuration process of a sysvinit script "more declarative" -- you do not need to specify the activities that need to be executed to manage processes, but instead, you specify the relevant characteristics of a running process.

From this high level specification, the implementations for all required activities will be derived, using conventions that are commonly used to write sysvinit scripts.

After completing the initial version of the process management framework that works with sysvinit scripts, I have also been investigating other process managers. I discovered that their configuration processes have many things in common with the sysvinit approach. As a result, I have decided to explore these declarative deployment concepts a bit further.

In this blog post, I will describe a declarative process manager-agnostic deployment approach that we can integrate into the experimental Nix-based process management framework.

Writing declarative deployment specifications for managed running processes

As explained in the introduction, I have also been experimenting with other process managers than sysvinit. For example, instead of generating a sysvinit script that manages the life-cycle of a process, such as the Nginx server, we can also generate a supervisord configuration file to define Nginx as a program that can be managed with supervisord:


{createSupervisordProgram, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSupervisordProgram {
  name = instanceName;
  command = "mkdir -p ${nginxLogDir}; "+
    "${nginx}/bin/nginx -c ${configFile} -p ${stateDir}";
  inherit dependencies;
}

Invoking the above function will generate a supervisord program configuration file, instead of a sysvinit script.

With the following Nix expression, we can generate a systemd unit file so that Nginx's life-cycle can be managed by systemd:


{createSystemdService, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSystemdService {
  name = instanceName;
  Unit = {
    Description = "Nginx";
  };
  Service = {
    ExecStartPre = "+mkdir -p ${nginxLogDir}";
    ExecStart = "${nginx}/bin/nginx -c ${configFile} -p ${stateDir}";
    Type = "simple";
  };

  inherit dependencies;
}

What you may probably notice when comparing the above two Nix expressions with the last sysvinit example (that captures process characteristics instead of activities), is that they all contain very similar properties. Their main difference is a slightly different organization and naming convention, because each abstraction function is tailored towards the configuration conventions that each target process manager uses.

As discussed in my previous blog post about declarative programming and deployment, declarativity is a spectrum -- the above specifications are (somewhat) declarative because they do not capture the activities to manage the life-cycle of the process (the how). Instead, they specify what process we want to run. The process manager derives and executes all activities to bring that process in a running state.

sysvinit scripts themselves are not declarative, because they specify all activities (i.e. shell commands) that need to be executed to accomplish that goal. supervisord configurations and systemd services configuration files are (somewhat) declarative, because they capture process characteristics -- the process manager executes derives all required activities to bring the process in a running state.

Despite the fact that I am not specifying any process management activities, these Nix expressions could still be considered somewhat a "how specification", because each configuration is tailored towards a specific process manager. A process manager, such as syvinit, is a means to accomplish something else: getting a running process whose life-cycle can be conveniently managed.

If I would revise the above specifications to only express what I kind of running process I want, disregarding the process manager, then I could simply write:


{createManagedProcess, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createManagedProcess {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile" -p" "${stateDir}/${instanceName}" ];

  inherit dependencies instanceName;
}

The above Nix expression simply states that we want to run a managed Nginx process (using certain command-line arguments) and before starting the process, we want to initialize the state by creating the log directory, if it does not exists yet.

I can translate the above specification to all kinds of configuration artifacts that can be used by a variety of process managers to accomplish the same outcome. I have developed six kinds of generators allowing me to target the following process managers:

sysvinit scripts, also known as LSB Init compliant scripts.
supervisord programs
systemd services
launchd services
BSD rc scripts
Windows services (via Cygwin's cygrunsrv)

Translating the properties of the process manager-agnostic configuration to a process manager-specific properties is quite straight forward for most concepts -- in many cases, there is a direct mapping between a property in the process manager-agnostic configuration to a process manager-specific property.

For example, when we intend to target supervisord, then we can translate the process and args parameters to a command invocation. For systemd, we can translate process and args to the ExecStart property that refers to a command-line instruction that starts the process.

Although the process manager-agnostic abstraction function supports enough features to get some well known system services working (e.g. Nginx, Apache HTTP service, PostgreSQL, MySQL etc.), it does not facilitate all possible features of each process manager -- it will provide a reasonable set of common features to get a process running and to impose some restrictions on it.

It is still possible work around the feature limitations of process manager-agnostic deployment specifications. We can also influence the generation process by defining overrides to get process manager-specific properties supported:


{createManagedProcess, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createManagedProcess {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile" -p" "${stateDir}/${instanceName}" ];

  inherit dependencies instanceName;

  overrides = {
    sysvinit = {
      runlevels = [ 3 4 5 ];
    };
  };
}

In the above example, we have added an override specifically for sysvinit to tell that the init system that the process should be started in runlevels 3, 4 and 5 (which implies the process should stopped in the remaining runlevels: 0, 1, 2, and 6). The other process managers that I have worked with do not have a notion of runlevels.

Similarly, we can use an override to, for example, use systemd-specific features to run a process in a Linux namespace etc.

Simulating process manager-agnostic concepts with no direct equivalents

For some process manager-agnostic concepts, process managers do not always have direct equivalents. In such cases, there is still the possibility to apply non-trivial simulation strategies.

Foreground processes or daemons

What all deployment specifications shown in this blog post have in common is that their main objective is to bring a process in a running state. How these processes are expected to behave is different among process managers.

sysvinit and BSD rc scripts expect processes to daemonize -- on invocation, a process spawns another process that keeps running in the background (the daemon process). After the initialization of the daemon process is done, the parent process terminates. If processes do not deamonize, the startup process execution blocks indefinitely.

Daemons introduce another complexity from a process management perspective -- when invoking an executable from a shell session in background mode, the shell can you tell its process ID, so that it can be stopped when it is no longer necessary.

With deamons, an invoked processes forks another child process (or when it supposed to really behave well: it double forks) that becomes the daemon process. The daemon process gets adopted by the init system, and thus remains in the background even if the shell session ends.

The shell that invokes the executable does not know the PIDs of the resulting daemon processes, because that value is only propagated to the daemon's parent process, not the calling shell session. To still be able to control it, a well-behaving daemon typically writes its process IDs to a so-called PID file, so that it can be reliably terminated by a shell command when it is no longer required.

sysvinit and BSD rc scripts extensively use PID files to control daemons. By using a process' PID file, the managing sysvinit/BSD rc script can tell you whether a process is running or not and reliably terminate a process instance.

"More modern" process managers, such as launchd, supervisord, and cygrunsrv, do not work with processes that daemonize -- instead, these process managers are daemons themselves that invoke processes that work in "foreground mode".

One of the advantages of this approach is that services can be more reliably controlled -- because their PIDs are directly propagated to the controlling daemon from the fork() library call, it is no longer required to work with PID files, that may not always work reliably (for example: a process might abrubtly terminate and never clean its PID file, giving the system the false impression that it is still running).

systemd improves process control even further by using Linux cgroups -- although foreground process may be controlled more reliably than daemons, they can still fork other processes (e.g. a web service that creates processes per connection). When the controlling parent process terminates, and does not properly terminate its own child processes, they may keep running in the background indefintely. With cgroups it is possible for the process manager to retain control over all processes spawned by a service and terminate them when a service is no longer needed.

systemd has another unique advantage over the other process managers -- it can work both with foreground processes and daemons, although foreground processes seem to have to preference according to the documentation, because they are much easier to control and develop.

Many common system services, such as OpenSSH, MySQL or Nginx, have the ability to both run as a foreground process and as a daemon, typically by providing a command-line parameter or defining a property in a configuration file.

To provide an optimal user experience for all supported process managers, it is typically a good thing in the process manager-agnostic deployment specification to specify both how a process can be used as a foreground process and as a daemon:


{createManagedProcess, nginx, stateDir, runtimeDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createManagedProcess {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-p" "${stateDir}/${instanceName}" "-c" configFile ];
  foregroundProcessExtraArgs = [ "-g" "daemon off;" ];
  daemonExtraArgs = [ "-g" "pid ${runtimeDir}/${instanceName}.pid;" ];

  inherit dependencies instanceName;

  overrides = {
    sysvinit = {
      runlevels = [ 3 4 5 ];
    };
  };
}

In the above example, we have revised Nginx expression to both specify how the process can be started as a foreground process and as a daemon. The only thing that needs to be configured differently is one global directive in the Nginx configuration file -- by default, Nginx runs as a deamon, but by adding the daemon off; directive to the configuration we can run it in foreground mode.

When we run Nginx as daemon, we configure a PID file that refers to the instance name so that multiple instances can co-exist.

To make this conveniently configurable, the above expression does the following:

The process parameter specifies the process that needs to be started both in foreground mode and as a daemon. The args parameter specifies common command-line arguments that both the foreground and daemon process will use.
The foregroundProcessExtraArgs parameter specifies additional command-line arguments that are only used when the process is started in foreground mode. In the above example, it is used to provide Nginx the global directive that disables the daemon setting.
The daemonExtraArgs parameter specifies additional command-line arguments that are only used when the process is started as a daemon. In the above example, it used to provide Nginx a global directive with a PID file path that uniquely identifies the process instance.

For custom software and services implemented in different language than C, e.g. Node.js, Java or Python, it is far less common that they have the ability to daemonize -- they can typically only be used as foreground processes.

Nonetheless, we can still daemonize foreground-only processes, by using an external tool, such as libslack's daemon command:


$ daemon -U -i myforegroundprocess

The above command deamonizes the foreground process and creates a PID file for it, so that it can be managed by the sysvinit/BSD rc utility scripts.

The opposite kind of "simulation" is also possible -- if a process can only be used as a daemon, then we can use a proxy process to make it appear as a foreground process:


export _TOP_PID=$$

# Handle to SIGTERM and SIGINT signals and forward them to the daemon process
_term()
{
    trap "exit 0" TERM
    kill -TERM "$pid"
    kill $_TOP_PID
}

_interrupt()
{
    kill -INT "$pid"
}

trap _term SIGTERM
trap _interrupt SIGINT

# Start process in the background as a daemon
${executable} "$@"

# Wait for the PID file to become available.
# Useful to work with daemons that don't behave well enough.
count=0

while [ ! -f "${_pidFile}" ]
do
    if [ $count -eq 10 ]
    then
        echo "It does not seem that there isn't any pid file! Giving up!"
        exit 1
    fi

    echo "Waiting for ${_pidFile} to become available..."
    sleep 1

    ((count=count++))
done

# Determine the daemon's PID by using the PID file
pid=$(cat ${_pidFile})

# Wait in the background for the PID to terminate
${if stdenv.isDarwin then ''
  lsof -p $pid +r 3 &>/dev/null &
'' else if stdenv.isLinux || stdenv.isCygwin then ''
  tail --pid=$pid -f /dev/null &
 '' else if stdenv.isBSD || stdenv.isSunOS then ''
   pwait $pid &
 '' else
   throw "Don't know how to wait for process completion on system: ${stdenv.system}"}

# Wait for the blocker process to complete.
# We use wait, so that bash can still
# handle the SIGTERM and SIGINT signals that may be sent to it by
# a process manager
blocker_pid=$!
wait $blocker_pid

The idea of the proxy script shown above is that it runs as a foreground process as long as the daemon process is running and relays any relevant incoming signals (e.g. a terminate and interrupt) to the daemon process.

Implementing this proxy was a bit tricky:

In the beginning of the script we configure signal handlers for the TERM and INT signals so that the process manager can terminate the daemon process.
We must start the daemon and wait for it to become available. Although the parent process of a well-behaving daemon should only terminate when the initialization is done, this turns out not be a hard guarantee -- to make the process a bit more robust, we deliberately wait for the PID file to become available, before we attempt to wait for the termination of the daemon.
Then we wait for the PID to terminate. The bash shell has an internal wait command that can be used to wait for a background process to terminate, but this only works with processes in the same process group as the shell. Daemons are in a new session (with different process groups), so they cannot be monitored by the shell by using the wait command.

From this Stackoverflow article, I learned that we can use the tail command of GNU Coreutils, or lsof on macOS/Darwin, and pwait on BSDs and Solaris/SunOS to monitor processes in other process groups.
When a command is being executed by a shell script (e.g. in this particular case: tail, lsof or pwait), the shell script can no longer respond to signals until the command completes. To still allow the script to respond to signals while it is waiting for the daemon process to terminate, we must run the previous command in background mode, and we use the wait instruction to block the script. While a wait command is running, the shell can respond to signals.

The generator function will automatically pick the best solution for the selected target process manager -- this means that when our target process manager are sysvinit or BSD rc scripts, the generator automatically picks the configuration settings to run the process as a daemon. For the remaining process managers, the generator will pick the configuration settings that runs it as a foreground process.

If a desired process model is not supported, then the generator will automatically simulate it. For instance, if we have a foreground-only process specification, then the generator will automatically configure a sysvinit script to call the daemon executable to daemonize it.

A similar process happens when a daemon-only process specification is deployed for a process manager that cannot work with it, such as supervisord.

State initialization

Another important aspect in process deployment is state initialization. Most system services require the presence of state directories in which they can store their PID, log and temp files. If these directories do not exist, the service may not work and refuse to start.

To cope with this problem, I typically make processes self initializing -- before starting the process, I check whether the state has been intialized (e.g. check if the state directories exist) and re-initialize the initial state if needed.

With most process managers, state initialization is easy to facilitate. For sysvinit and BSD rc scripts, we just use the generator to first execute the shell commands to initialize the state before the process gets started.

Supervisord allows you to execute multiple shell commands in a single command directive -- we can just execute a script that initializes the state before we execute the process that we want to manage.

systemd has a ExecStartPre directive that can be used to specify shell commands to execute before the main process starts.

Apple launchd and cygrunsrv, however, do not have a generic shell execution mechanism or some facility allowing you to execute things before a process starts. Nonetheless, we can still ensure that the state is going to be initialized by creating a wrapper script -- first the wrapper script does the state initialization and then executes the main process.

If a state initialization procedure was specified and the target process manager does not support scripting, then the generator function will transparently wrap the main process into a wrapper script that supports state initialization.

Process dependencies

Another important generic concept is process dependency management. For example, Nginx can act as a reverse proxy for another web application process. To provide a functional Nginx service, we must be sure that the web application process gets activated as well, and that the web application is activated before Nginx.

If the web application process is activated after Nginx or missing completely, then Nginx is (temporarily) unable to redirect incoming requests to the web application process causing end-users to see bad gateway errors.

The process managers that I have experimented with all have a different notion of process dependencies.

sysvinit scripts can optionally declare dependencies in their comment sections. Tools that know how to interpret these dependency specifications can use it to decide the right activation order. Systems using sysvinit typically ignore this specification. Instead, they work with sequence numbers in their file names -- each run level configuration directory contains a prefix (S or K) followed by two numeric digits that defines the start or stop order.

supervisord does not work with dependency specifications, but every program can optionally provide a priority setting that can be used to order the activation and deactivation of programs -- lower priority numbers have precedence over high priority numbers.

From dependency specifications in a process management expression, the generator function can automatically derive sequence numbers for process managers that require it.

Similar to sysvinit scripts, BSD rc scripts can also declare dependencies in their comment sections. Contrary to sysvinit scripts, BSD rc scripts can use the rcorder tool to parse these dependencies from the comments section and automatically derive the order in which the BSD rc scripts need to be activated.

cygrunsrv also allows you directly specify process dependencies. The Windows service manager makes sure that the service get activated in the right order and that all process dependencies are activated first. The only limitation is that cygrunsrv only allows up to 16 dependencies to be specified per service.

To simulate process dependencies with systemd, we can use two properties. The Wants property can be used to tell systemd that another service needs to be activated first. The After property can be used to specify the ordering.

Sadly, it seems that launchd has no notion of process dependencies at all -- processes can be activated by certain events, e.g. when a kernel module was loaded or through socket activation, but it does not seem to have the ability to configure process dependencies or the activation ordering. When our target process manager is launchd, then we simply have to inform the user that proper activation ordering cannot be guaranteed.

Changing user privileges

Another general concept, that has subtle differences in each process manager, is changing user privileges. Typically for the deployment of system services, you do not want to run these services as root user (that has full access to the filesystem), but as an unprivileged user.

sysvinit and BSD rc scripts have to change users through the su command. The su command can be used to change the user ID (UID), and will automatically adopt the primary group ID (GID) of the corresponding user.

Supervisord and cygrunsrv can also only change user IDs (UIDs), and will adopt the primary group ID (GID) of the corresponding user.

Systemd and launchd can both change the user IDs and group IDs of the process that it invokes.

Because only changing UIDs are universally supported amongst process managers, I did not add a configuration property that allows you to change GIDs in a process manager-agnostic way.

Deploying process manager-agnostic configurations

With a processes Nix expression, we can define which process instances we want to run (and how they can be constructed from source code and their dependencies):


{ pkgs ? import  { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange processManager;
  };                                                                                                                                                                                               
in                                                                                                                                                                                                 
rec {                                                                                                                                                                                              
  webapp = rec {                                                                                                                                                                                   
    port = 5000;                                                                                                                                                                                   
    dnsName = "webapp.local";                                                                                                                                                                      
                                                                                                                                                                                                   
    pkg = constructors.webapp {                                                                                                                                                                    
      inherit port;                                                                                                                                                                                
    };                                                                                                                                                                                             
  };                                                                                                                                                                                               
                                                                                                                                                                                                   
  nginxReverseProxy = rec {
    port = 8080;

    pkg = constructors.nginxReverseProxy {
      webapps = [ webapp ];
      inherit port;
    } {};
  };
}

In the above Nix expression, we compose two running process instances:

webapp is a trivial web application process that will simply return a static HTML page by using the HTTP protocol.
nginxReverseProxy is a Nginx server configured as a reverse proxy server. It will forward incoming HTTP requests to the appropriate web application instance, based on the virtual host name. If a virtual host name is webapp.local, then Nginx forwards the request to the webapp instance.

To generate the configuration artifacts for the process instances, we refer to a separate constructors Nix expression. Each constructor will call the createManagedProcess function abstraction (as shown earlier) to construct a process configuration in a process manager-agnostic way.

With the following command-line instruction, we can generate sysvinit scripts for the webapp and Nginx processes declared in the processes expression, and run them as an unprivileged user with the state files managed in our home directory:


$ nixproc-build --process-manager sysvinit \
  --state-dir /home/sander/var \
  --force-disable-user-change processes.nix

By adjusting the --process-manager parameter we can also generate artefacts for a different process manager. For example, the following command will generate systemd unit config files instead of sysvinit scripts:


$ nixproc-build --process-manager systemd \
  --state-dir /home/sander/var \
  --force-disable-user-change processes.nix

The following command will automatically build and deploy all processes, using sysvinit as a process manager:


$ nixproc-sysvinit-switch --state-dir /home/sander/var \
  --force-disable-user-change processes.nix

We can also run a life-cycle management activity on all previously deployed processes. For example, to retrieve the statuses of all processes, we can run:


$ nixproc-sysvinit-runactivity status

We can also traverse the processes in reverse dependency order. This is particularly useful to reliably stop all processes, without breaking any process dependencies:


$ nixproc-sysvinit-runactivity -r stop

Similarly, there are command-line tools to use the other supported process managers. For example, to deploy systemd units instead of sysvinit scripts, you can run:


$ nixproc-systemd-switch processes.nix

Distributed process manager-agnostic deployment with Disnix

As shown in the previous process management framework blog post, it is also possible to deploy processes to machines in a network and have inter-dependencies between processes. These kinds of deployments can be managed by Disnix.

Compared to the previous blog post (in which we could only deploy sysvinit scripts), we can now also use any process manager that the framework supports. The Dysnomia toolset provides plugins that supports all process managers that this framework supports:


{ pkgs, distribution, invDistribution, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "sysvinit"
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange processManager;
  };

  processType =
    if processManager == "sysvinit" then "sysvinit-script"
    else if processManager == "systemd" then "systemd-unit"
    else if processManager == "supervisord" then "supervisord-program"
    else if processManager == "bsdrc" then "bsdrc-script"
    else if processManager == "cygrunsrv" then "cygrunsrv-service"
    else throw "Unknown process manager: ${processManager}";
in
rec {
  webapp = rec {
    name = "webapp";
    port = 5000;
    dnsName = "webapp.local";
    pkg = constructors.webapp {
      inherit port;
    };
    type = processType;
  };

  nginxReverseProxy = rec {
    name = "nginxReverseProxy";
    port = 8080;
    pkg = constructors.nginxReverseProxy {
      inherit port;
    };
    dependsOn = {
      inherit webapp;
    };
    type = processType;
  };
}

In the above expression, we have extended the previously shown processes expression into a Disnix service expression, in which every attribute in the attribute set represents a service that can be distributed to a target machine in the network.

The type attribute of each service indicates which Dysnomia plugin needs to manage its life-cycle. We can automatically select the appropriate plugin for our desired process manager by deriving it from the processManager parameter.

The above Disnix expression has a drawback -- in a heteregenous network of machines (that run multiple operating systems and/or process managers), we need to compose all desired variants of each service with configuration files for each process manager that we want to use.

It is also possible to have target-agnostic services, by delegating the translation steps to the corresponding target machines. Instead of directly generating a configuration file for a process manager, we generate a JSON specification containing all parameters that are passed to createManagedProcess. We can use this JSON file to build the corresponding configuration artefacts on the target machine:


{ pkgs, distribution, invDistribution, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? null
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange processManager;
  };
in
rec {
  webapp = rec {
    name = "webapp";
    port = 5000;
    dnsName = "webapp.local";
    pkg = constructors.webapp {
      inherit port;
    };
    type = "managed-process";
  };

  nginxReverseProxy = rec {
    name = "nginxReverseProxy";
    port = 8080;
    pkg = constructors.nginxReverseProxy {
      inherit port;
    };
    dependsOn = {
      inherit webapp;
    };
    type = "managed-process";
  };
}

In the above services model, we have set the processManager parameter to null causing the generator to print JSON presentations of the function parameters passed to createManagedProcess.

The managed-process type refers to a Dysnomia plugin that consumes the JSON specification and invokes the createManagedProcess function to convert the JSON configuration to a configuration file used by the preferred process manager.

In the infrastructure model, we can configure the preferred process manager for each target machine:


{
  test1 = {
    properties = {
      hostname = "test1";
    };
    containers = {
      managed-process = {
        processManager = "sysvinit";
      };
    };
  };

  test2 = {
    properties = {
      hostname = "test2";
    };
    containers = {
      managed-process = {
        processManager = "systemd";
      };
    };
  };
}

In the above infrastructure model, the managed-proces container on the first machine: test1 has been configured to use sysvinit scripts to manage processes. On the second test machine: test2 the managed-process container is configured to use systemd to manage processes.

If we distribute the services in the services model to targets in the infrastructure model as follows:


{infrastructure}:

{
  webapp = [ infrastructure.test1 ];
  nginxReverseProxy = [ infrastructure.test2 ];
}

and the deploy the system as follows:


$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Then the webapp process will distributed to the test1 machine in the network and will be managed with a sysvinit script.

The nginxReverseProxy will be deployed to the test2 machine and managed as a systemd job. The Nginx reverse proxy forwards incoming connections to the webapp.local domain name to the web application process hosted on the first machine.

Discussion

In this blog post, I have introduced a process manager-agnostic function abstraction making it possible to target all kinds of process managers on a variety of operating systems.

By using a single set of declarative specifications, we can:

Target six different process managers on four different kinds of operating systems.
Implement various kinds of deployment scenarios: production deployments, test deployments as an unprivileged user.
Construct multiple instances of processes.

In a distributed-context, the advantage is that we can uniformly target all supported process managers and operating systems in a heterogeneous environment from a single declarative specification.

This is particularly useful to facilitate technology diversity -- for example, one of the key selling points of Microservices is that "any technology" can be used to implement them. In many cases, technology diversity is "restricted" to frameworks, programming languages, and storage technologies.

One particular aspect that is rarely changed is the choice of operating systems, because of the limitations of deployment tools -- most deployment solutions for Microservices are container-based and heavily rely on Linux-only concepts, such as Namespaces and cgroups.

With this process managemenent framework and the recent Dysnomia plugin additions for Disnix, it is possible to target all kinds of operating systems that support the Nix package manager, making the operating system component selectable as well. This, for example, allows you to also pick the best operating system to implement a certain requirement -- for example, when performance is important you might pick Linux, and when there is a strong emphasis on security, you could pick OpenBSD to host a mission criticial component.

Limitations

The following table, summarizes the differences between the process manager solutions that I have investigated:

	sysvinit	bsdrc	supervisord	systemd	launchd	cygrunsrv
Process type	daemon	daemon	foreground	foreground daemon	foreground	foreground
Process control method	PID files	PID files	Process PID	cgroups	Process PID	Process PID
Scripting support	yes	yes	yes	yes	no	no
Process dependency management	Numeric ordering	Dependency-based	Numeric ordering	Dependency-based + dependency loading	None	Dependency-based + dependency loading
User changing capabilities	user	user	user and group	user and group	user and group	user
Unprivileged user deployments	yes*	yes*	yes	yes*	no	no
Operating system support	Linux	FreeBSD >OpenBSD NetBSD	Many UNIX-like: Linux macOS FreeBSD Solaris	Linux (+glibc) only	macOS (Darwin)	Windows (Cygwin)

Although we can facilitate lifecycle management from a common specification with a variety of process managers, only the most important common features are supported.

Not every concept can be done in a process manager agnostic way. For example, we cannot generically do any isolation of resources (except for packages, because we use Nix). It is difficult to generalize these concepts because these they are not standardized, e.g. the POSIX standard does not descibe namespaces and cgroups (or similar concepts).

Furthermore, most process managers (with the exception of supervisord) are operating system specific. As a result, it still matters what process manager is picked.

Related work

Process manager-agnostic deployment is not entirely a new idea. Dysnomia already has a target-agnostic 'process' plugin for quite a while, that translates a simple deployment specification (constisting of key-value pairs) to a systemd unit configuration file or sysvinit script.

The features of Dysnomia's process plugin are much more limited compared to the createManagedProcess abstraction function described in this blog post. It does not support any other than process managers than sysvint and systemd, and it can only work with foreground processes.

Furthermore, target agnostic configurations cannot be easily extended -- it is possible to (ab)use the templating mechanism, but it has no first class overridde facilities.

I also found a project called pleaserun that also has the objective to generate configuration files for a variety of process managers (my approach and pleaserunit, both support sysvinit scripts, systemd and launchd).

It seems to use template files to generate the configuration artefacts, and it does not seem to have a generic extension mechanism. Furthermore, it provides no framework to configure the location of shared resources, automatically install package dependencies or to compose multiple instances of processes.

Some remaining thoughts

Although the Nix package manager (not the NixOS distribution), should be portable amongst a variety of UNIX-like systems, it turns out that the only two operating systems that are well supported are Linux and macOS. Nix was reported to work on a variety of other UNIX-like systems in the past, but recently it seems that many things are broken.

To make Nix work on FreeBSD 12.1, I have used the latest stable Nix package manager version with patches from this repository. It turns out that there is still a patch missing to work around in a bug in FreeBSD that incorrectly kills all processes in a process group. Fortunately, when we run Nix as as unprivileged user, this bug does not seem to cause any serious problems.

Recent versions of Nixpkgs turn out to be horribly broken on FreeBSD -- the FreeBSD stdenv does not seem to work at all. I tried switching back to stdenv-native (a stdenv environment that impurely uses the host system's compiler and core executables), but that also no longer seems to work in the last three major releases -- the Nix expression evaluation breaks in several places. Due to the intense amount of changes and assumptions that the stdenv infrastructure currently makes, it was as good as impossible for me to fix the infrastructure.

As another workaround, I reverted back very to a very old version of Nixpkgs (version 17.03 to be precise), that still has a working stdenv-native environment. With some tiny adjustments (e.g. adding some shell aliases for some GNU variants of certain shell executables to stdenv-native), I have managed to get some basic Nix packages working, including Nginx on FreeBSD.

Surprisingly, running Nix on Cygwin was less painful than FreeBSD (because of all the GNUisms that Cygwin provides). Similar to FreeBSD, recent versions of Nixpkgs also appear to be broken, including the Cygwin stdenv environment. By reverting back to release-18.03 (that still has a somewhat working stdenv for Cygwin), I have managed to build a working Nginx version.

As a future improvement to Nixpkgs, I would like to propose a testing solution for stdenv-native. Although I understand that is difficult to dedicate manpower to maintain all unconventional Nix/Nixpkgs ports, stdenv-native is something that we can also convienently test on Linux and prevent from breaking in the future.

Availability

The latest version of my experimental Nix-based process framework, that includes the process manager-agnostic configuration function described in this blog post, can be obtained from my GitHub page.

In addition, the repository also contains some example cases, including the web application system described in this blog post, and a set of common system services: MySQL, Apache HTTP server, PostgreSQL and Apache Tomcat.

Cachix: CDN and double storage size

support@cachix.org (Domen Kožar) — Wed, 29 Jan 2020 08:00:00 +0000

Cachix - Nix binary cache hosting, has grown quite a bit in recent months in terms of day to day usage and that was mostly noticable on bandwidth. -Over 3000 GB were served in December 2019. -CDN by CloudFlare Increased usage prompted a few backend machine instance upgrades to handle concurrent upload/downloads, but it became clear it’s time to abandon single machine infrastructure. -As of today, all binary caches are served by CloudFlare CDN.

Mayflower: __structuredAttrs in Nix

support@nixbuild.net (nixbuild.net) — Mon, 20 Jan 2020 12:00:00 +0000

Up until now, nixbuild.net has not supported directly fetching build dependencies from binary caches like cache.nixos.org or Cachix. All build dependencies have instead been uploaded from the user’s local machine to nixbuild.net the first time they’ve been needed.

Today, this bottleneck has been removed, since nixbuild.net now can fetch build dependencies directly from binary caches, without taxing users’ upload bandwidth.

+ +

By default, the official Nix binary cache (cache.nixos.org) is added to all nixbuild.net accounts, but a nixbuild.net user can freely decide on which caches that should be queried for build dependencies (including Cachix caches).

An additional benefit of the new support for binary caches is that users that trust the same binary caches automatically share build dependencies from those caches. This means that if one user’s build has triggered a download from for example cache.nixos.org, the next user that comes along and needs the same build dependency doesn’t have to spend time on downloading that dependency.

For more information on how to use binary caches with nixbuild.net, see the documentation.

Hercules Labs: Hercules CI & Cachix split up

Tue, 14 Jan 2020 00:00:00 +0000

After careful consideration of how to balance between the two products, we’ve decided to split up. Each of the two products will be a separate entity:

+ Graham Christensen: Erase your darlings + http://grahamc.com//blog/erase-your-darlings + http://grahamc.com/blog/erase-your-darlings +

I erase my systems at every boot.

+ +

Over time, a system collects state on its root partition. This state +lives in assorted directories like /etc and /var, and represents +every under-documented or out-of-order step in bringing up the +services.

+ +

+
“Right, run myapp-init.”
+

+ +

These small, inconsequential “oh, oops” steps are the pieces that get +lost and don’t appear in your runbooks.

+ +

+
“Just download ca-certificates to … to fix …”
+

+ +

Each of these quick fixes leaves you doomed to repeat history in three +years when you’re finally doing that dreaded RHEL 7 to RHEL 8 upgrade.

+ +

+
“Oh, touch /etc/ipsec.secrets or the l2tp tunnel won’t work.”
+

+ +

Immutable infrastructure gets us so close

+ +

Immutable infrastructure is a wonderfully effective method of +eliminating so many of these forgotten steps. Leaning in to the pain +by deleting and replacing your servers on a weekly or monthly basis +means you are constantly testing and exercising your automation and +runbooks.

+ +

The nugget here is the regular and indiscriminate removal of system +state. Destroying the whole server doesn’t leave you much room to +forget the little tweaks you made along the way.

+ +

These techniques work great when you meet two requirements:

Hercules CI becomes part of Robert Hensing’s Ensius B.V.
Cachix becomes part of Domen Kožar’s Enlambda OÜ
you can provision and destroy servers with an API call
the servers aren’t inherently stateful

For customers there will be no changes, except for the point of contact in support requests.

Long running servers

Domen & Robert

Mayflower: Windows-on-NixOS, part 1: Migrating bare-metal to a VM

Wed, 27 Nov 2019 06:00:00 +0000

This is part 1 of a series of blog posts explaining how we took an existing Windows installation on hardware and moved it into a VM running on top of NixOS. -Background We have a decently-equipped desktop PC sitting in our office, which is designated for data experiments using TensorFlow and such. During off-hours, it's also used for games, and for that purpose it has Windows installed on it. We decided to try moving Windows into a VM within NixOS so that we could run both operating systems in parallel.

Craige McWhirter: Deploying and Configuring Vim on NixOS

Thu, 14 Nov 2019 04:18:37 +0000

There are lots of cases in which immutable infrastructure doesn’t +work, and the dirty secret is those servers need good tools the +most.

I had a need to deploy vim and my particular preferred -configuration both system-wide and across multiple systems (via -NixOps).

Long-running servers cause long outages. Their runbooks are outdated +and incomplete. They accrete tweaks and turn in to an ossified, +brittle snowflake — except its arms are load-bearing.

I started by creating a file named vim.nix that would be imported into either -/etc/nixos/configuration.nix or an appropriate NixOps Nix file. This example -is a stub that shows a number of common configuration items:

Let’s bring the ideas of immutable infrastructure to these systems +too. Whether this system is embedded in a stadium’s jumbotron, in a +datacenter, or under your desk, we can keep the state under control.

vim.nix:

FHS isn’t enough

with import <nixpkgs> {};
+The hard part about applying immutable techniques to long running
+servers is knowing exactly where your application state ends and the
+operating system, software, and configuration begin.
 
-vim_configurable.customize {
-  name = "vim";   # Specifies the vim binary name.
-  # Below you can specify what usually goes into `~/.vimrc`
-  vimrcConfig.customRC = ''
-    " Preferred global default settings:
-    set number                    " Enable line numbers by default
-    set background=dark           " Set the default background to dark or light
-    set smartindent               " Automatically insert extra level of indentation
-    set tabstop=4                 " Default tabstop
-    set shiftwidth=4              " Default indent spacing
-    set expandtab                 " Expand [TABS] to spaces
-    syntax enable                 " Enable syntax highlighting
-    colorscheme solarized         " Set the default colour scheme
-    set t_Co=256                  " use 265 colors in vim
-    set spell spelllang=en_au     " Default spell checking language
-    hi clear SpellBad             " Clear any unwanted default settings
-    hi SpellBad cterm=underline   " Set the spell checking highlight style
-    hi SpellBad ctermbg=NONE      " Set the spell checking highlight background
-    match ErrorMsg '\s\+$'        "
+This is hard because legacy operating systems and the Filesystem
+Hierarchy Standard poorly separate these areas of concern. For
+example, /var/lib is for state information, but how much of this do
+you actually care about tracking? What did you configure in /etc on
+purpose?
 
-    let g:airline_powerline_fonts = 1   " Use powerline fonts
-    let g:airline_theme='solarized'     " Set the airline theme
+The answer is probably not a lot.
 
-    set laststatus=2   " Set up the status line so it's coloured and always on
+You may not care, but all of this accumulation of junk is a tarpit.
+Everything becomes harder: replicating production, testing changes,
+undoing mistakes.
 
-    " Add more settings below
-  '';
-  # store your plugins in Vim packages
-  vimrcConfig.packages.myVimPackage = with pkgs.vimPlugins; {
-    start = [               # Plugins loaded on launch
-      airline               # Lean & mean status/tabline for vim that's light as air
-      solarized             # Solarized colours for Vim
-      vim-airline-themes    # Collection of themes for airlin
-      vim-nix               # Support for writing Nix expressions in vim
-    ];
-    # manually loadable by calling `:packadd $plugin-name`
-    # opt = [ phpCompletion elm-vim ];
-    # To automatically load a plugin when opening a filetype, add vimrc lines like:
-    # autocmd FileType php :packadd phpCompletion
-  };
-}
-

New computer smell

I then needed to import this file into my system packages stanza:

Getting a new computer is this moment of cleanliness. The keycaps +don’t have oils on them, the screen is perfect, and the hard drive +is fresh and unspoiled — for about an hour or so.

  environment = {
-    systemPackages = with pkgs; [
-      someOtherPackages   # Normal package listing
-      (
-        import ./vim.nix
-      )
-    ];
-  };
-

Let’s get back to that.

This will then install and configure Vim as you've defined it.

How is this possible?

If you'd like to give this build a run in a non-production space, I've written vim_vm.nix with which you can build a VM, ssh into afterwards and test the Vim configuration:

NixOS can boot with only two directories: /boot, and /nix.

$ nix-build '<nixpkgs/nixos>' -A vm --arg configuration ./vim_vm.nix
-...
-$ export QEMU_OPTS="-m 4192"
-$ export QEMU_NET_OPTS="hostfwd=tcp::18080-:80,hostfwd=tcp::10022-:22"
-$ ./result/bin/run-vim-vm-vm
-

/nix contains read-only system configurations, which are specified +by your configuration.nix and are built and tracked as system +generations. These never change. Once the files are created in /nix, +the only way to change the config’s contents is to build a new system +configuration with the contents you want.

Then, from a another terminal:

Any configuration or files created on the drive outside of /nix is +state and cruft. We can lose everything outside of /nix and /boot +and have a healthy system. My technique is to explicitly opt in and +choose which state is important, and only keep that.

$ ssh nixos@localhost -p 10022
-

How this is possible comes down to the boot sequence.

And you should be in a freshly baked NixOS VM with your Vim config ready to be -used.

For NixOS, the bootloader follows the same basic steps as a standard +Linux distribution: the kernel starts with an initial ramdisk, and the +initial ramdisk mounts the system disks.

There's an always current example of my production Vim -configuration -in my mio-ops repo.

Hercules Labs: Hercules CI Agent 0.6.1

Tue, 12 Nov 2019 00:00:00 +0000

We’ve released hercules-ci-agent 0.6.1, days after 0.6.0 release.

And here is where the similarities end.

Everyone is encouraged to upgrade, as it brings performance improvements, a bugfix to IFD and better onboarding experience.

NixOS’s early startup

0.6.1 - 2019-11-06

NixOS configures the bootloader to pass some extra information: a +specific system configuration. This is the secret to NixOS’s +bootloader rollbacks, and also the key to erasing our disk on each +boot. The parameter is named systemConfig.

Fixed

On every startup the very early boot stage knows what the system’s +configuration should be: the entire system configuration is stored in +the read-only /nix/store, and the directory passed through +systemConfig has a reference to the config. Early boot then +manipulates /etc and /run to match the chosen setup. Usually this +involves swapping out a few symlinks.

-
Fix token leak to system log when reporting an HTTP exception. This was introduced by a library upgrade. -This was discovered after tagging 0.6.0 but before the release was -announced and before moving of the stable branch. -Only users of the hercules-ci-agent master branch and the unannounced -tag were exposed to this leak. -We recommend to follow the stable branch.
-
-
Temporarily revert a Nix GC configuration change that might cause problems -until agent gc root behavior is improved.
-

If /etc simply doesn’t exist, however, early boot creates /etc +and moves on like it were any other boot. It also creates /var, +/dev, /home, and any other core directories that must be present.

0.6.0 - 2019-11-04

Simply speaking, an empty / is not surprising to NixOS. In fact, +the NixOS netboot, EC2, and installation media all start out this way.

Changed

Opting out

Switch to Nix 2.3 and NixOS 19.09. You should update your deployment to reflect the NixOS upgrade, unless you’re using terraform or nix-darwin, where it’s automatic.
Increased parallellism during push to cachix
Switch to NixOS 19.09
Enable min-free/max-free Nix GC

Before we can opt in to saving data, we must opt out of saving data +by default. I do this by setting up my filesystem in a way that +lets me easily and safely erase the unwanted data, while preserving +the data I do want to keep.

Fixed

My preferred method for this is using a ZFS dataset and rolling it +back to a blank snapshot before it is mounted. A partition of any +other filesystem would work just as well too, running mkfs at boot, +or something similar. If you have a lot of RAM, you could skip the +erase step and make / a tmpfs.

Transient errors during source code fetching are now retried
Fixed a bug related to narinfo caching in the context of IFD
Fixed an exception when the root of ci.nix is a list, although lists are unsupported

Opting out with ZFS

When installing NixOS, I partition my disk with two partitions, one +for the boot partition, and another for a ZFS pool. Then I create and +mount a few datasets.

What we do

My root dataset:

Automated hosted infrastructure for Nix, reliable and reproducible developer tooling, -to speed up adoption and lower integration cost. We offer -Continuous Integration and Binary Caches.

Sander van der Burg: A Nix-based functional organization for managing processes

noreply@blogger.com (Sander van der Burg) — Mon, 11 Nov 2019 22:43:00 +0000

The Nix expression language and the Nix packages repository follow a number of unorthodox, but simple conventions that provide all kinds of benefits, such as the ability to conveniently construct multiple variants of packages and store them safely in isolation without any conflicts.

The scope of the Nix package manager, however, is limited to package deployment only. Other tools in the Nix project extend deployment to other kinds of domains, such as machine level deployment (NixOS), networks of machines (NixOps) and service-oriented systems (Disnix).

In addition to packages, there is also a category of systems (such as systems following the microservices paradigm) that are composed of running processes.

Recently, I have been automating deployments of several kinds of systems that are composed of running processes and I have investigated how we can map the most common Nix packaging conventions to construct specifications that we can use to automate the deployment of these kinds of systems.

Some common Nix packaging conventions

The Nix package manager implements a so-called purely functional deployment model. In Nix, packages are constructed in the Nix expression language from pure functions in which side effects are eliminated as much as possible, such as undeclared dependencies residing in global directories, such as /lib and /bin.

The function parameters of a build function refer to all required inputs to construct the package, such as the build instructions, the source code, environment variables and all required build-time dependencies, such as compilers, build tools and libraries.

A big advantage of eliminating side effects (or more realistically: significantly reducing side effects) is to support reproducible deployment -- when building the same package with the same inputs on a different machine, we should get a (nearly) bit-identical result.

Strong reproducibility guarantees, for example, make it possible to optimize package deployments by only building a package from source code once and then downloading binary substitutes from remote servers that can be trusted.

In addition to the fact that packages are constructed by executing pure functions (with some caveats), the Nixpkgs repository -- that contains a large set of well known free and open source packages -- follows a number of conventions. One of such conventions is that most package build recipes reside in separate files and that each recipe declares a function.

An example of such a build recipe is:

{ stdenv, fetchurl, pkgconfig, glib, gpm, file, e2fsprogs
, perl, zip, unzip, gettext, libssh2, openssl}:

stdenv.mkDerivation rec {
  pname = "mc";
  version = "4.8.23";

  src = fetchurl {
    url = "http://www.midnight-commander.org/downloads/${pname}-${version}.tar.xz";
    sha256 = "077z7phzq3m1sxyz7li77lyzv4rjmmh3wp2vy86pnc4387kpqzyx";
  };

  buildInputs = [
    pkgconfig perl glib slang zip unzip file gettext libssh2 openssl
  ];

  configureFlags = [ "--enable-vfs-smb" ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    maintainers = [ stdenv.lib.maintainers.sander ];
    platforms = with stdenv.lib.platforms; linux ++ darwin;
  };
}

The Nix expression shown above (pkgs/tools/misc/mc/default.nix) describes how to build the Midnight Commander from source code and its inputs:

The first line declares a function in which the function arguments refer to all dependencies required to build Midnight Commander: stdenv refers to an environment that provides standard UNIX utilities, such as cat and ls and basic build utilities, such as gcc and make. fetchurl is a utility function that can be used to download artifacts from remote locations and that can verify the integrity of the downloaded artifact.

The remainder of the function arguments refer to packages that need to be provided as build-time dependencies, such as tools and libraries.
In the function body, we invoke the stdenv.mkDerivation function to construct a Nix package from source code.

By default, if no build instructions are provided, it will automatically execute the standard GNU Autotools/GNU Make build procedure: ./configure; make; make install, automatically downloads and unpacks the tarball specified by the src parameter, and uses buildInputs to instruct the configure script to automatically find the dependencies it needs.

A function definition that describes a package build recipe is not very useful on its own -- to be able to build a package, it needs to be invoked with the appropriate parameters.

A Nix package is composed in a top-level Nix expression (pkgs/top-level/all-packages.nix) that declares one big data structure: an attribute set, in which every attribute name refers to a possible variant of a package (typically only one) and each value to a function invocation that builds the package, with the desired versions of variants of the dependencies that a package may need:

{ system ? builtins.currentSystem }:

rec {
  stdenv = ...
  fetchurl = ...
  pkgconfig = ...
  glib = ...

  ...

  openssl = import ../development/libraries/openssl {
    inherit stdenv fetchurl zlib ...;
  };

  mc = import ../tools/misc/mc {
    inherit stdenv fetchurl pkgconfig glib gpm file e2fsprogs perl;
    inherit zip unzip gettext libssh2 openssl;
  };
}

The last attribute (mc) in the attribute set shown above, builds a specific variant of Midnight Commander, by passing the dependencies that it needs as parameters. It uses the inherit language construct to bind the parameters that are declared in the same lexical scope.

All the dependencies that Midnight Commander needs are declared in the same attribute set and composed in a similar way.

(As a sidenote: in the above example, we explicitly propagate all function parameters, which is quite verbose and tedious. In Nixpkgs, it is also possible to use a convenience function called: callPackage that will automatically pass the attributes with the same names as the function arguments as parameters.)

With the composition expression above and running the following command-line instruction:

$ nix-build all-packages.nix -A mc
/nix/store/wp3r8qv4k510...-mc-4.8.23

The Nix package manager will first deploy all build-time dependencies that Midnight Commander needs, and will then build Midnight Commander from source code. The build result is stored in the Nix store (/nix/store/...-mc-4.8.23), in which all build artifacts reside in isolation in their own directories.

We can start Midnight Commander by providing the full path to the mc executable:

$ /nix/store/wp3r8qv4k510...-mc-4.8.23/bin/mc

The prefix of every artifact in the Nix store is a SHA256 hash code derived from all inputs provided to the build function. The SHA256 hash prefix makes it possible to safely store multiple versions and variants of the same package next to each other, because they never share the same name.

If Nix happens to compute a SHA256 that is already in the Nix store, then the build result is exactly the same, preventing Nix from doing the same build again.

Because the Midnight Commander build recipe is a function, we can also adjust the function parameters to build different variants of the same package. For example, by changing the openssl parameter, we can build a Midnight Commander variant that uses a specific version of OpenSSL that is different than the default version:

{ system ? builtins.currentSystem }:

rec {
  stdenv = ...
  fetchurl = ...
  pkgconfig = ...
  glib = ...

  ...

  openssl_1_1_0 = import ../development/libraries/openssl/1.1.0.nix {
    inherit stdenv fetchurl zlib ...;
  };

  mc_alternative = import ../tools/misc/mc {
    inherit stdenv fetchurl pkgconfig glib gpm file e2fsprogs perl;
    inherit zip unzip gettext libssh2;
    openssl = openssl_1_1_0; # Use a different OpenSSL version
  };
}

We can build our alternative Midnight Commander variant as follows:

$ nix-build all-packages.nix -A mc_alternative
/nix/store/0g0wm23y85nc0y...-mc-4.8.23

As may be noticed, we get a different Nix store path, because we build Midnight Commander with different build inputs.

Although the purely functional model provides all kinds of nice benefits (such as reproducibility, the ability conveniently construct multiple variants of a package, and storing them in isolation without any conflicts), it also has a big inconvenience from a user point of view -- as a user, it is very impractical to remember the SHA256 hash prefixes of a package to start a program.

As a solution, Nix also makes it possible to construct user environments (probably better known as Nix profiles), by using the nix-env tool or using the buildEnv {} function in Nixpkgs.

User environments are symlink trees that blend the content of a set of packages into a single directory in the Nix store so that they can be accessed from one single location. By adding the bin/ sub folder of a user environment to the PATH environment variable, it becomes possible for a user to start a command-line executable without specifying a full path.

For example, with the nix-env tool we can install the Midnight Commander in a Nix profile:

$ nix-env -f all-packages.nix -iA mc

and then start it as follows:

$ mc

The above command works if the Nix profile is in the PATH environment variable of the user.

Mapping packaging conventions to process management

There are four important packaging conventions that the Nix package manager and the Nixpkgs repository follow that I want to emphasize:

Invoking the derivation function (typically through stdenv.mkDerivation or an abstraction built around it) builds a package from its build inputs.
Every package build recipe defines a function in which the function parameters refer to all possible build inputs. We can use this function to compose all kinds of variants of a package.
Invoking a package build recipe function constructs a particular variant of a package and stores the result in the Nix store.
Nix profiles blend the content of a collection of packages into one directory and makes them accessible from a single location.

(As a sidenote: There is some discussion in the Nix community about these concepts. For example, one of the (self-)criticisms is that the Nix expression language, that is specifically designed as a DSL for package management, has no package concept in the language.

Despite this oddity, I personally think that functions are a simple and powerful concept. The only thing that is a bit of a poor decision in my opinion is to call the mechanism that executes a build: derivation).

Process management is quite different from package management -- we need to have an executable deployed first (typically done by a package manager, such as Nix), but in addition, we also need to manage the life-cycle of a process, such as starting and stopping it. These facilities are not Nix's responsibility. Instead, we need to work with a process manager that can facilitate these.

Furthermore, systems composed of running processes have a kind of dependency relationship that Nix does not manage -- they may also communicate with other processes (e.g. via a network connection or UNIX domain sockets).

As a consequence, they require the presence of other processes in order to work. This means that processes need to be activated in the right order or, alternatively, the communication between two dependent processes need to be queued until both are available.

If these dependency requirements are not met, then a system may not work. For example, a web application process is useless if the database backend is not available.

In order to fully automate the deployment of systems that are composed of running processes, we can do package management with Nix first and then we need to:

Integrate with a process manager, by generating artifacts that a process manager can work with, such as scripts and/or configuration files.
Make it possible to specify the process dependencies so that they can be managed (by a process manager or by other means) and activated in the right order.

Generating sysvinit scripts

There a variety of means to manage processes. A simple (and for today's standards maybe an old fashioned and perhaps controversial) way to manage processes is by using sysvinit scripts (also known as LSB Init compliant scripts).

A sysvinit script implements a set of activities and a standardized interface allowing us to manage the lifecycle of a specific process, or a group of processes.

For example, on a traditional Linux distribution, we can start a process, such as the Nginx web server, with the following command:

$ /etc/init.d/nginx start

and stop it as follows:

$ /etc/init.d/nginx stop

A sysvinit script is straight forward to implement and follows a number of conventions:

#!/bin/bash

## BEGIN INIT INFO
# Provides:      nginx
# Default-Start: 3 4 5
# Default-Stop:  0 1 2 6
# Should-Start:  webapp
# Should-Stop:   webapp
# Description:   Nginx
## END INIT INFO

. /lib/lsb/init-functions

case "$1" in
  start)
    log_info_msg "Starting Nginx..."
    mkdir -p /var/nginx/logs
    start_daemon /usr/bin/nginx -c /etc/nginx.conf -p /var/nginx 
    evaluate_retval
    ;;

  stop)
    log_info_msg "Stopping Nginx..."
    killproc /usr/bin/nginx
    evaluate_retval
    ;;

  reload)
    log_info_msg "Reloading Nginx..."
    killproc /usr/bin/nginx -HUP
    evaluate_retval
    ;;

  restart)
    $0 stop
    sleep 1
    $0 start
    ;;

  status)
    statusproc /usr/bin/nginx
    ;;

  *)
    echo "Usage: $0 {start|stop|reload|restart|status}"
    exit 1
    ;;
esac

A sysvinit script typically starts by providing some metadata, such a description, in which runlevels it needs to be started and stopped, and which dependencies the script has.

In classic Linux distributions, meta information is typically ignored, but more sophisticated process managers, such as systemd, can use it to automatically configure the activation/deactivation ordering.
The body defines a case statement that executes a requested activity.
Activities use a special construct (in the example above it is: evaluate_retval) to display the status of an instruction, typically whether a process has started or stopped successfully or not, using appropriate colors (e.g. red in case of a failure, green in case of sucess).
sysvinit scripts typically define a number of commonly used activities: start starts a process, stop stops a process, reload sends a HUP signal to the process to let it reload its configuration (if applicable), restart restarts the process, status indicates the status, and there is a fallback activity that displays the usage to the end user to show which activities can be executed.

sysvinit scripts use number of utility functions that are defined by the Linux Standards Base (LSB):

start_daemon is a utility function that is typically used for starting a process. It has the expectation that the process daemonizes -- a process that daemonizes will fork another process that keeps running in the background and then terminates immediately.

Controlling a daemonized processes is a bit tricky -- when spawning a process the shell can tell you its process id (PID), so that it can be controlled, but it cannot tell you the PID of the process that gets daemonized by the invoked process, because that is beyond the shell's control.

As a solution, most programs that daemonize will write a PID file (e.g. /var/run/nginx.pid) that can be used to determine the PID of the daemon so that it can be controlled.

To do proper housekeeping, the start_daemon function will check whether such a PID file already exists, and will only start the process when it needs to.
Stopping a process, or sending it a different kind of signal, is typically done with the killproc function.

This function will search for the corresponding PID file of the process (by default, a PID file that has the same name as the executable or a specified PID file) and uses the corresponding PID content to terminate the daemon. As a fallback, if no PID file exists, it will scan the entire process table and kills the process with the same name.
We can determine the status of a process (e.g. whether it is running or not), with the statusproc function that also consults the corresponding PID file or scans the process table if needed.

Most common system software have the ability to deamonize, such as nginx, the Apache HTTP server, MySQL and PostgreSQL. Unfortunately, application services (such as microservices) that are implemented with technologies such as Python, Node.js or Java Springboot do not have this ability out of the box.

Fortunately, we can use an external utility, such as libslack's daemon command, to let these foreground-only processes daemonize. Although it is possible to conveniently daemonize external processes, this functionality is not part of the LSB standard.

For example, using the following command to start the web application front-end process will automatically daemonize a foreground process, such as a simple Node.js web application, and creates a PID file so that it can be controlled by the sysvinit utility functions:

$ daemon -U -i /home/sander/webapp/app.js

In addition to manually starting and stopping sysvinit scripts, sysvinit scripts are also typically started on startup and stopped on shutdown, or when a user switches between runlevels. These processes are controlled by symlinks that reside in an rc.d directory that have specific prefixes:

/etc/
  init.d/
    webapp
    nginx
  rc0.d/
    K98nginx -> ../init.d/nginx
    K99webapp -> ../init.d/webapp
  rc1.d/
    K98nginx -> ../init.d/nginx
    K99webapp -> ../init.d/webapp
  rc2.d/
    K98nginx -> ../init.d/nginx
    K99webapp -> ../init.d/webapp
  rc3.d/
    S00webapp -> ../init.d/nginx
    S01nginx -> ../init.d/webapp
  rc4.d/
    S00webapp -> ../init.d/nginx
    S01nginx -> ../init.d/webapp
  rc5.d/
    S00webapp -> ../init.d/nginx
    S01nginx -> ../init.d/webapp
  rc6.d/
    K98nginx -> ../init.d/nginx
    K99webapp -> ../init.d/webapp

In the above directory listing, every rc?.d directory contains symlinks to scripts in the init.d directory.

The first character of each symlink file indicates whether an init.d script should be started (S) or stopped (K). The two numeric digits that follow indicate the order in which the scripts need to be started and stopped.

Each runlevel has a specific purpose as described in the LSB standard. In the above situation, when we boot the system in multi-user mode on the console (run level 3), first our Node.js web application will be started, followed by nginx. On a reboot (when we enter runlevel 6) nginx and then the web application will be stopped. Basically, the stop order is the reverse of the start order.

To conveniently automate the deployment of sysvinit scripts, I have created a utility function called: createSystemVInitScript that makes it possible to generate sysvinit script with the Nix package manager.

We can create a Nix expression that generates a sysvinit script for nginx, such as:

{createSystemVInitScript, nginx}:

let
  configFile = ./nginx.conf;
  stateDir = "/var";
in
createSystemVInitScript {                                                                                                                                                                                          
  name = "nginx";
  description = "Nginx";
  activities = {
    start = ''
      mkdir -p ${stateDir}/logs
      log_info_msg "Starting Nginx..."
      loadproc ${nginx}/bin/nginx -c ${configFile} -p ${stateDir}
      evaluate_retval
    '';
    stop = ''
      log_info_msg "Stopping Nginx..."
      killproc ${nginx}/bin/nginx
      evaluate_retval
    '';
    reload = ''
      log_info_msg "Reloading Nginx..."
      killproc ${nginx}/bin/nginx -HUP
      evaluate_retval
    '';
    restart = ''
      $0 stop
      sleep 1
      $0 start
    '';
    status = "statusproc ${nginx}/bin/nginx";
  };
  runlevels = [ 3 4 5 ];
}

The above expression defines a function in which the function parameters refer to all dependencies that we need to construct the sysvinit script to manage a nginx server: createSystemVInitScript is the utility function that creates sysvinit scripts, nginx is the package that provides Nginx.

In the body, we invoke the: createSystemVInitScript to construct a sysvinit script:

The name corresponds to name of the sysvinit script and the description to the description displayed in the metadata header.
The activities parameter refers to an attribute set in which every name refers to an activity and every value to the shell commands that need to be executed for this activity.

We can use this parameter to specify the start, stop, reload, restart and status activities for nginx. The function abstraction will automatically configure the fallback activity that displays the usage to the end-user including the activities that the script supports.
The runlevels parameter indicates in which runlevels the init.d script should be started. For these runlevels, the function will create start symlinks. An implication is that for the runlevels that are not specified (0, 1, 2, and 6) the script will automatically create stop symlinks.

As explained earlier, sysvinit script use conventions. One of such conventions is that most activities typically display a description, then execute a command, and finally display the status of that command, such as:

log_info_msg "Starting Nginx..."
loadproc ${nginx}/bin/nginx -c ${configFile} -p ${stateDir}
evaluate_retval

The createSystemVInit script also a notion of instructions, that are automatically translated into activities displaying task descriptions (derived from the general description) and the status. Using the instructions parameter allows us to simplify the above expression to:

{createSystemVInitScript, nginx}:

let
  configFile = ./nginx.conf;
  stateDir = "/var";
in
createSystemVInitScript {                                                                                                                                                                                          
  name = "nginx";
  description = "Nginx";
  instructions = {
    start = {
      activity = "Starting";
      instruction = ''
        mkdir -p ${stateDir}/logs
        loadproc ${nginx}/bin/nginx -c ${configFile} -p ${stateDir}
      '';
    };
    stop = {
      activity = "Stopping";
      instruction = "killproc ${nginx}/bin/nginx";
    };
    reload = {
      activity = "Reloading";
      instruction = "killproc ${nginx}/bin/nginx -HUP";
    };
  };
  activities = {
    status = "statusproc ${nginx}/bin/nginx";
  };
  runlevels = [ 3 4 5 ];
}

In the above expression, the start, stop and reload activities have been simplified by defining them as instructions allowing us to write less repetitive boilerplate code.

We can reduce the amount of boilerplate code even further -- the kind of activities that we need to implement for managing process are typically mostly the same. When we want to manage a process, we typically want a start, stop, restart, status activity and, if applicable, a reload activity if a process knows how to handle the HUP signal.

Instead of speciying activities or instructions, it is also possible to specify which process we want to manage, and what kind of parameters the process should take:

{createSystemVInitScript, nginx}:

let
  configFile = ./nginx.conf;
  stateDir = "/var";
in
createSystemVInitScript {                                                                                                                                                                                          
  name = "nginx";
  description = "Nginx";
  initialize = ''
    mkdir -p ${stateDir}/logs
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile "-p" stateDir ];
  runlevels = [ 3 4 5 ];
}

From the process and args parameters, the createSystemVInitScript automatically derives all relevant activities that we need to manage the process. It is also still possible to augment or override the generated activities by means of the instructions or activities parameters.

Besides processes that already have the ability to daemonize, it is also possible to automatically daemonize foreground processes with this function abstraction. This is particularly useful to generate a sysvinit script for the Node.js web application service, that lacks this ability:

{createSystemVInitScript}:

let
  webapp = (import ./webapp {}).package;
in
createSystemVInitScript {
  name = "webapp";
  process = "${webapp}/lib/node_modules/webapp/app.js";
  processIsDaemon = false;
  runlevels = [ 3 4 5 ];
  environment = {
    PORT = 5000;
  };
}

In the above Nix expression, we set the parameter: processIsDaemon to false (the default value is: true) to indicate that the process is not a deamon, but a foreground process. The createSystemVInitScript function will generate a start activity that invokes the daemon command to daemonize it.

Another interesting feature is that we can specify process dependency relationships. For example, an nginx server can act as a reverse proxy for the Node.js web application.

To reliably activate the entire system, we must make sure that the web application process is deployed before Nginx is deployed. If we activate the system in the opposite order, then the reverse proxy may redirect users to an non-existent web application causing them to see 502 bad gateway errors.

We can use the dependency parameter with a reference to a sysvinit script to indicate that this sysvinit script has a dependency. For example, we can revise the Nginx sysvinit script expression as follows:

{createSystemVInitScript, nginx, webapp}:

let
  configFile = ./nginx.conf;
  stateDir = "/var";
in
createSystemVInitScript {                                                                                                                                                                                          
  name = "nginx";
  description = "Nginx";
  initialize = ''
    mkdir -p ${stateDir}/logs
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile "-p" stateDir ];
  runlevels = [ 3 4 5 ];
  dependencies = [ webapp ];
}

In the above example, we pass the webapp sysvinit script as a dependency (through the dependencies parameter). Adding it as a dependency causes the generator to compute a start sequence number for the nginx script that will be higher than the web app sysvinit script and stop sequence number that will be lower than the web app script.

The different sequence numbers ensure that webapp is started before nginx starts, and that the nginx stops before the webapp stops.

Configuring managed processes

So far composing sysvinit scripts is still very similar to composing ordinary Nix packages. We can also extend the four Nix packaging conventions described in the introduction to create a process management discipline.

Similar to the convention in which every package is in a separate file, and defines a function in which the function parameters refers to all package dependencies, we can extend this convention for processes to also include relevant parameters to configure a service.

For example, we can write a Nix expression for the web application process as follows:

{createSystemVInitScript, port ? 5000}:

let
  webapp = (import /home/sander/webapp {}).package;
in
createSystemVInitScript {
  name = "webapp";
  process = "${webapp}/lib/node_modules/webapp/app.js";
  processIsDaemon = false;
  runlevels = [ 3 4 5 ];
  environment = {
    PORT = port;
  };
}

In the above expression, the port function parameter allows us to configure the TCP port where the web application listens to (and defaults to 5000).

We can also make the configuration of nginx configurable. For example, we can create a function abstraction that creates a configuration for nginx to let it act as a reverse proxy for the web application process shown earlier:

{createSystemVInitScript, stdenv, writeTextFile, nginx
, runtimeDir, stateDir, logDir, port ? 80, webapps ? []}:

let
  nginxStateDir = "${stateDir}/nginx";
in
import ./nginx.nix {
  inherit createSystemVInitScript nginx instanceSuffix;
  stateDir = nginxStateDir;

  dependencies = map (webapp: webapp.pkg) webapps;

  configFile = writeTextFile {
    name = "nginx.conf";
    text = ''
      error_log ${nginxStateDir}/logs/error.log;
      pid ${runtimeDir}/nginx.pid;

      events {
        worker_connections 190000;
      }

      http {
        ${stdenv.lib.concatMapStrings (dependency: ''
          upstream webapp${toString dependency.port} {
            server localhost:${toString dependency.port};
          }
        '') webapps}

        ${stdenv.lib.concatMapStrings (dependency: ''
          server {
            listen ${toString port};
            server_name ${dependency.dnsName};

            location / {
              proxy_pass  http://webapp${toString dependency.port};
            }
          }
        '') webapps}
      }
    '';
  };
}

The above Nix expression's funtion header defines, in addition to the package dependencies, process configuration parameters that make it possible to configure the TCP port that Nginx listens to (port 80 by default) and to which web applications it should forward requests based on their virtual host property.

In the body, these properties are used to generate a nginx.conf file that defines virtualhosts for each web application process. It forwards incoming requests to the appropriate web application instance. To connect to a web application instance, it uses the port number that the webapp instance configuration provides.

Similar to ordinary Nix expressions, Nix expressions for processes also need to be composed, by passing the appropriate function parameters. This can be done in a process composition expression that has the following structure:

{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
}:

let
  createSystemVInitScript = import ./create-sysvinit-script.nix {
    inherit (pkgs) stdenv writeTextFile daemon;
    inherit runtimeDir tmpDir;

    createCredentials = import ./create-credentials.nix {
      inherit (pkgs) stdenv;
    };

    initFunctions = import ./init-functions.nix {
      basePackages = [
        pkgs.coreutils
        pkgs.gnused
        pkgs.inetutils
        pkgs.gnugrep
        pkgs.sysvinit
      ];
      inherit (pkgs) stdenv;
      inherit runtimeDir;
    };
  };
in
rec {
  webapp = rec {
    port = 5000;
    dnsName = "webapp.local";

    pkg = import ./webapp.nix {
      inherit createSystemVInitScript port;
    };
  };

  nginxReverseProxy = rec {
    port = 80;

    pkg = import ./nginx-reverse-proxy.nix {
      inherit createSystemVInitScript;
      inherit stateDir logDir runtimeDir port;
      inherit (pkgs) stdenv writeTextFile nginx;
      webapps = [ webapp ];
    };
  };
}

The above expression (processes.nix) has the following structure:

The expression defines a function in which the function parameters allow common properties that apply to all processes to be configured: pkgs refers to the set of Nixpkgs that contains a big collection of free and open source packages, system refers to the system architecture to build packages for, and stateDir to the directory where processes should store their state (which is /var according to the LSB standard).

The remaining parameters specify the runtime, log and temp directories, that are typically sub directories in the state directory.
In the let block, we compose our createSystemVInitScript function using the relevant state directory parameters, base packages and utility functions.
In the body, we construct an attribute set in which every name represents a process name and every value an attribute set that contains process properties.
One reserved process property of a process attribute set is the pkg property that refers to a package providing the sysvinit script.
The remaining process properties can be freely chosen and can be consumed by any process that has a dependency on it.

For example, the nginxReverseProxy service uses the port and dnsName properties of the webapp process to configure nginx to forward requests to the provided DNS host name (webapp.local) to the web application process listening on the specified TCP port (5000).

Using the above composition Nix expression for processes and the following command-line instruction, we can build the sysvinit script for the web application process:

$ nix-build processes.nix -A webapp

We can start the web application process by using the generated sysvinit script, as follows:

$ ./result/bin/etc/rc.d/init.d/webapp start

and stop it as follows:

$ ./result/bin/etc/rc.d/init.d/webapp stop

We can also build the nginx reverse proxy in a similar way, but to properly activate it, we must make sure that the webapp process is activated first.

To reliably manage a set of processes and activate them in the right order, we can also generate a Nix profile that contains all init.d scripts and rc.d symlinks for stopping and starting:

{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
}:

let
  buildSystemVInitEnv = import ./build-sysvinit-env.nix {
    inherit (pkgs) buildEnv;
  };
in
buildSystemVInitEnv {
  processes = import ./processes.nix {
    inherit pkgs system;
  };
}

The above expression imports the process composition expression shown earlier, and invokes the buildSystemVInitEnv to compose a Nix profile out of it. We can build this environment as follows:

$ nix-build profile.nix

Visually, the content of the Nix profile can presented as follows:

In the above diagram the ovals denote processes and the arrows denote process dependency relationships. The arrow indicates that the webapp process needs to be activated before the nginxReverseProxy.

We can use the system's rc script manage the starting and stopping the processes when runlevels are switched. Runlevels 1-5 make it possible to start the processes on startup and 0 and 6 to stop them on shutdown or reboot.

In addition to the system's rc script, we can also directly control the processes in a Nix profile -- I have created a utility script called: rcswitch that makes it possible to manually start all processes in a profile:

$ rcswitch ./result/etc/rc.d/rc3.d

we can also use the rcswitch command to do an upgrade from one set of processes to another:

$ rcswitch ./result/etc/rc.d/rc.3 ./oldresult/etc/rc.d/rc3.d

The above command checks which of the sysvinit scripts exist in both profiles and will only deactivate obsolete processes and activate new processes.

With the rcrunactivity command it is possible to run arbitrary activities on all processes in a profile. For example, the following command will show all statuses:

$ rcactivity status ./result/etc/rc.d/rc3.d

Deploying services as an unprivileged user

The process composition expression shown earlier is also a Nix function that takes various kinds of state properties as parameters.

By default, it has been configured in such a way that it facilitates production deployments. For example, it stores the state of all services in the global /var directory. Only the super user has the permissions to alter the structure of the global /var directory.

It is also possible to change these configuration parameters in such a way that it becomes possible as an unprivileged user to do process deployment.

For example, by changing the port number of the nginxReverseProxy process to a value higher than 1024, such as 8080 (an unprivileged user is not allowed to bind any services to ports below 1024), and changing the stateDir parameter to a directory in a user's home directory, we can deploy our web application service and Nginx reverse proxy as an unprivileged user:

$ nix-build processes.nix --argstr stateDir /home/sander/var \
  -A nginxReverseProxy

By overriding the stateDir parameter, the resulting Nginx process has been configured to store all state in /home/sander/var as opposed to the global /var that cannot be modified by an unprivileged user.

As an unprivileged user, I should be able to start the Nginx reverse proxy as follows:

$ ./result/etc/rc.d/init.d/nginx start

The above Nginx instance can be reached by opening: http://localhost:8080 in a web browser.

Creating multiple process instances

So far, we have only been deploying single instances of processes. For the Nginx reverse proxy example, it may also be desired to deploy multiple instances of the webapp process so that we can manage forwardings for multiple virtual domains.

We can adjust the Nix expression for the webapp to make it possible to create multiple process instances:

{createSystemVInitScript}:
{port, instanceSuffix ? ""}:

let
  webapp = (import ./webapp {}).package;
  instanceName = "webapp${instanceSuffix}";
in
createSystemVInitScript {
  name = instanceName;
  inherit instanceName;
  process = "${webapp}/lib/node_modules/webapp/app.js";
  processIsDaemon = false;
  runlevels = [ 3 4 5 ];
  environment = {
    PORT = port;
  };
}

The above Nix expression is a modified webapp build recipe that facilitates instantiation:

We have split the Nix expression into two nested functions. The first line: the outer function header defines all dependencies and configurable properties that apply to all services instances.
The inner function header allows all instance specific properties to be configured so that multiple instances can co-exist. An example of such a property is the port parameter -- only one service can bind to a specific TCP port. Configuring an instance to bind to different port allows two instances co-exist.

The instanceSuffix parameter makes it possible to give each webapp process a unique name (e.g. by providing a numeric value).

From the package name and instance suffix a unique instanceName is composed. Propagating the instanceName to the createSystemVInitScript function instructs the daemon command to create a unique PID file (not a PID file that corresponds to the executable name) for each daemon process so that multiple instances can be controlled independently.

Although this may sound as a very uncommon use case, it is also possible to change the Nix expression for the Nginx reverse proxy to support multiple instances.

Typically, for system services, such as web servers and database servers, it is very uncommon to run multiple instances at the same time. Despite the fact that it is uncommon, it is actually possible and quite useful for development and/or experimentation purposes:

{ createSystemVInitScript, stdenv, writeTextFile, nginx
, runtimeDir, stateDir, logDir}:

{port ? 80, webapps ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxStateDir = "${stateDir}/${instanceName}";
in
import ./nginx.nix {
  inherit createSystemVInitScript nginx instanceSuffix;
  stateDir = nginxStateDir;

  dependencies = map (webapp: webapp.pkg) webapps;

  configFile = writeTextFile {
    name = "nginx.conf";
    text = ''
      error_log ${nginxStateDir}/logs/error.log;
      pid ${runtimeDir}/${instanceName}.pid;

      events {
        worker_connections 190000;
      }

      http {
        ${stdenv.lib.concatMapStrings (dependency: ''
          upstream webapp${toString dependency.port} {
            server localhost:${toString dependency.port};
          }
        '') webapps}

        ${stdenv.lib.concatMapStrings (dependency: ''
          server {
            listen ${toString port};
            server_name ${dependency.dnsName};

            location / {
              proxy_pass  http://webapp${toString dependency.port};
            }
          }
        '') webapps}
      }
    '';
  };
}

The code fragment above shows a revised Nginx expression that supports instantiation:

Again, the Nix expression defines a nested function in which the outer function header refers to configuration properties for all services, whereas the inner function header refers to all conflicting parameters that need to be changed so that multiple instances can co-exist.
The port parameter allows the TCP port where Nginx bind to be configured. To have two instances co-existing they both need to bind to unreserved ports.
As with the previous example, the instanceSuffix parameter makes it possible to compose unique names for each Nginx instance. The instanceName variable that is composed from it, is used to create and configure a dedicate state directory, and a unique PID file that does not conflict with other Nginx instances.

With this new convention of nested functions for instantiatable services means that we have to compose these expressions twice. First, we need to pass all parameters that configure properties that apply to all service instances. This can be done in a Nix expression that has the following structure:

{ pkgs
, system
, stateDir
, logDir
, runtimeDir
, tmpDir
}:

let
  createSystemVInitScript = import ./create-sysvinit-script.nix {
    inherit (pkgs) stdenv writeTextFile daemon;
    inherit runtimeDir tmpDir;

    createCredentials = import ./create-credentials.nix {
      inherit (pkgs) stdenv;
    };

    initFunctions = import ./init-functions.nix {
      basePackages = [
        pkgs.coreutils
        pkgs.gnused
        pkgs.inetutils
        pkgs.gnugrep
        pkgs.sysvinit
      ];
      inherit (pkgs) stdenv;
      inherit runtimeDir;
    };
  };
in
{
  webapp = import ./webapp.nix {
    inherit createSystemVInitScript;
  };

  nginxReverseProxy = import ./nginx-reverse-proxy.nix {
    inherit createSystemVInitScript stateDir logDir runtimeDir;
    inherit (pkgs) stdenv writeTextFile nginx;
  };
}

The above Nix expression is something we could call a constructors expression (constructors.nix) that returns an attribute set in which each member refers to a function that allows us to compose a specific process instance.

By using the constructors expression shown above, we can create a processes composition expression that works with multiple instances:

{ pkgs ? import  { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/home/sbu"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs system stateDir runtimeDir logDir tmpDir;
  };
in
rec {
  webapp1 = rec {
    port = 5000;
    dnsName = "webapp1.local";

    pkg = constructors.webapp {
      inherit port;
      instanceSuffix = "1";
    };
  };

  webapp2 = rec {
    port = 5001;
    dnsName = "webapp2.local";

    pkg = constructors.webapp {
      inherit port;
      instanceSuffix = "2";
    };
  };

  webapp3 = rec {
    port = 5002;
    dnsName = "webapp3.local";

    pkg = constructors.webapp {
      inherit port;
      instanceSuffix = "3";
    };
  };

  webapp4 = rec {
    port = 5003;
    dnsName = "webapp4.local";

    pkg = constructors.webapp {
      inherit port;
      instanceSuffix = "4";
    };
  };

  nginxReverseProxy = rec {
    port = 8080;

    pkg = constructors.nginxReverseProxy {
      webapps = [ webapp1 webapp2 webapp3 webapp4 ];
      inherit port;
    };
  };

  webapp5 = rec {
    port = 6002;
    dnsName = "webapp5.local";

    pkg = constructors.webapp {
      inherit port;
      instanceSuffix = "5";
    };
  };

  webapp6 = rec {
    port = 6003;
    dnsName = "webapp6.local";

    pkg = constructors.webapp {
      inherit port;
      instanceSuffix = "6";
    };
  };

  nginxReverseProxy2 = rec {
    port = 8081;

    pkg = constructors.nginxReverseProxy {
      webapps = [ webapp5 webapp6 ];
      inherit port;
      instanceSuffix = "2";
    };
  };
}

In the above expression, we import the constructors expression, as shown earlier. In the body, we construct multiple instances of these processes by using the constructors functions:

We compose six web application instances (webapp1, webapp2, ..., webapp6), each of them listening on a unique TCP port.
We compose two Nginx instances (nginxReverseProxy, nginxReverseProxy2). The first instance listens on TCP port 8080 and redirects the user to any of the first three web application processes, based on the virtual host name. The other Nginx instance listens on TCP port 8081, redirecting the user to the remaining web apps based on the virtual host name.

We can represent the above composition expression visually, as follows:

As with the previous examples, we can deploy each process instance individually:

$ nix-build processes.nix -A webapp3
$ ./result/etc/rc.d/init.d/webapp3 start

Or the the whole set as a Nix profile:

$ nix-build profile.nix
$ rcswitch ./result/etc/rc.d/rc3.d

Again, the rcswitch command will make sure that all processes are activated in the right order. This means that the webapp processes are activated first, followed by the Nginx reverse proxies.

Managing user accounts/state with Dysnomia

Most of the deployment of the processes can be automated in a stateless way -- Nix can deploy the executable as a Nix package and the sysvinit script can manage the lifecycle.

There is another concern, that we may also want to address. Typically, it is not recommended to run processes as a root user, such as essential system services, for security and safety reasons.

In order to run a process as an unprivileged user, an unprivileged group and user account must be created first by some means. Furthermore, when undeploying a process, we may also want to remove the dedicated user and group.

User account management is a feature that the Nix package manager does not support -- Nix only works with files stored in the Nix store and cannot/will not (by design) change any files on the host system, such as /etc/passwd where the user accounts are stored.

I have created a deployment tool for state management (Dysnomia) that can be used for this purpose. It facilitates a plugin system that can manage deployment activities for components that Nix does not support: activating, deactivating, taking snapshots, restoring snapshots etc.

I have created a Dysnomia plugin called: sysvinit-script that can activate or deactivate a process by invoking a sysvinit script. It can also create or discard users and groups from a declarative configuration file that is included with a sysvinit script.

We can revise a process Nix expression to start a process as an unprivileged user:

{createSystemVInitScript}:
{port, instanceSuffix ? ""}:

let
  webapp = (import ./webapp {}).package;
  instanceName = "webapp${instanceSuffix}";
in
createSystemVInitScript {
  name = instanceName;
  inherit instanceName;
  process = "${webapp}/lib/node_modules/webapp/app.js";
  processIsDaemon = false;
  runlevels = [ 3 4 5 ];
  environment = {
    PORT = port;
  };
  user = instanceName;

  credentials = {
    groups = {
      "${instanceName}" = {};
    };
    users = {
      "${instanceName}" = {
        group = instanceName;
        description = "Webapp";
      };
    };
  };
}

The above Nix expression is a revised webapp Nix expression that facilitates user switching:

The user parameter specifies that we want to run the process as an unprivileged user. Because this process can also be instantiated, we have to make sure that it gets a unique name. To facilitate that, we create a user with the same username as the instance name.
The credentials parameter refers to a specification that instructs the sysvinit-script Dysnomia plugin to create an unprivileged user and group on activation, and discard them on deactivation.

For production purposes (e.g. when we deploy processes as the root user), switching to unprivileged users is useful, but for development purposes, such as running a set of processes as an unprivileged user, we cannot switch users because we may not have the permissions to do so.

For convenience purposes, it is also possible to globally disable user switching, which we can do as follows:

{ pkgs
, stateDir
, logDir
, runtimeDir
, tmpDir
, forceDisableUserChange
}:

let
  createSystemVInitScript = import ./create-sysvinit-script.nix {
    inherit (pkgs) stdenv writeTextFile daemon;
    inherit runtimeDir tmpDir forceDisableUserChange;

    createCredentials = import ./create-credentials.nix {
      inherit (pkgs) stdenv;
    };

    initFunctions = import ./init-functions.nix {
      basePackages = [
        pkgs.coreutils
        pkgs.gnused
        pkgs.inetutils
        pkgs.gnugrep
        pkgs.sysvinit
      ];
      inherit (pkgs) stdenv;
      inherit runtimeDir;
    };
  };
in
{
  ...
}

In the above example, the forceDisableUserChange parameter can be used to globally disable user switching for all sysvinit scripts composed in the expression. It invokes a feature of the createSystemVInitScript to ignore any user settings that might have been propagated to it.

With the following command we can deploy a process that does not switch users, despite having user settings configured in the process Nix expressions:

$ nix-build processes.nix --arg forceDisableUserChange true

Distributed process deployment with Disnix

As explained earlier, I have adopted four common Nix package conventions and extended them suit the needs of process management.

This is not the only solution that I have implemented that builds on these four conventions -- the other solution is Disnix, that extends Nix's packaging principles to (distributed) service-oriented systems.

Disnix extends Nix expressions for ordinary packages with another category of dependencies: inter-dependencies that model dependencies on services that may have been deployed to remote machines in a network and require a network connection to work.

In Disnix, a service expression is a nested function in which the outer function header specifies all intra-dependencies (local dependencies, such as build tools and libraries), and the inner function header refers to inter-dependencies.

It is also possible to combine the concepts of process deployment described in this blog post with the service-oriented system concepts of Disnix, such as inter-dependencies -- the example with Nginx reverse proxies and web application processes can be extended to work in a network of machines.

Besides deploying a set processes (that may have dependencies on each other) to a single machine, it is also possible to deploy the web application processes to different machines in the network than the machine where the Nginx reverse proxy is deployed to.

We can configure the reverse proxy in such a way that it will forward requests to the machine where the web application processes may have been deployed to.

{ createSystemVInitScript, stdenv, writeTextFile, nginx
, runtimeDir, stateDir, logDir
}:

{port ? 80, instanceSuffix ? ""}:

interDeps:

let
  instanceName = "nginx${instanceSuffix}";
  nginxStateDir = "${stateDir}/${instanceName}";
in
import ./nginx.nix {
  inherit createSystemVInitScript nginx instanceSuffix;
  stateDir = nginxStateDir;

  dependencies = map (dependencyName: 
    let
      dependency = builtins.getAttr dependencyName interDeps;
    in
    dependency.pkg
   ) dependencies;

  configFile = writeTextFile {
    name = "nginx.conf";
    text = ''
      error_log ${nginxStateDir}/logs/error.log;
      pid ${runtimeDir}/${instanceName}.pid;

      events {
        worker_connections 190000;
      }

      http {
        ${stdenv.lib.concatMapStrings (dependencyName:
          let
            dependency = builtins.getAttr dependencyName interDeps;
          in
          ''
            upstream webapp${toString dependency.port} {
              server ${dependency.target.properties.hostname}:${toString dependency.port};
            }
          '') (builtins.attrNames interDeps)}

        ${stdenv.lib.concatMapStrings (dependencyName:
          let
            dependency = builtins.getAttr dependencyName interDeps;
          in
          ''
            server {
              listen ${toString port};
              server_name ${dependency.dnsName};

              location / {
                proxy_pass  http://webapp${toString dependency.port};
              }
            }
          '') (builtins.attrNames interDeps)}
        }
    '';
  };
}

The above Nix expression is a revised Nginx configuration that also works with inter-dependencies:

The above Nix expression defines three nested functions. The purpose of the outermost function (the first line) is to configure all local dependencies that are common to all process instances. The middle function defines all process instance parameters that are potentially conflicting and need to be configurd with unique values so that multiple instances can co-exist. The third (inner-most) function refers to the inter-dependencies of this process: services that may reside on a different machine in the network and need to be reached with a network connection.
The inter-dependency function header (interDeps:) takes an arbitrary number of dependencies. These inter-dependencies refer to all web application process instances that the Nginx reverse proxy should redirect to.
In the body, we generate an nginx.conf that uses the inter-dependencies to set up the forwardings.

Compared to the previous Nginx reverse proxy example, it will use the dependency.target.properties.hostname property that refers to the hostname of the machine where the web application process is deployed to instead of a forwarding to localhost. This makes it possible to connect to a web application process that may have been deployed to another machine.
The inter-dependencies are also passed to the dependencies function parameter of the Nginx function. This will ensure that if Nginx and a web application process are distributed to the same machine by Disnix, they will also get activated in the right order by the system's rc script on startup.

A with the previous examples, we need to compose the above Disnix expression multiple times. The composition of the constructors can be done in the constructors expression (as shown in the previous examples).

The processes' instance properties and inter-dependencies can be configured in the Disnix services model, that shares many similarities with process composition expression, shown earlier. As a matter of fact, a Disnix services model is a superset of it:

{ pkgs, distribution, invDistribution, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? true
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange;
  };
in
rec {
  webapp = rec {
    name = "webapp";
    port = 5000;
    dnsName = "webapp.local";
    pkg = constructors.webapp {
      inherit port;
    };
    type = "sysvinit-script";
  };

  nginxReverseProxy = rec {
    name = "nginxReverseProxy";
    port = 8080;
    pkg = constructors.nginxReverseProxy {
      inherit port;
    };
    dependsOn = {
      inherit webapp;
    };
    type = "sysvinit-script";
  };
}

The above Disnix services model defines two services (representing processes) that have an inter-dependency on each other, as specified with the dependsOn parameter property of each service.

The sysvinit-script type property instructs Disnix to deploy the services as processes managed by a sysvinit script. In a Disnix-context, services have no specific form or meaning, and can basically represent anything. The type property is used to tell Disnix with what kind of service we are dealing with.

To properly configure remote dependencies we also need to know the target machines where we can deploy to and what their properties are. This is where we can use an infrastructure model for.

For example, a simple infrastructure model of two machines could be:

{
  test1.properties.hostname = "test1";
  test2.properties.hostname = "test2";
}

We must also tell Disnix to which target machines we want to distribute the services. This can be done in a distribution model:

{infrastructure}:

{
  webapp = [ infrastructure.test1 ];
  nginxReverseProxy = [ infrastructure.test2 ];
}

In the above distribution model we distribute the webapp process to the first target machine and the nginxReverseProxy to the second machine. Because both services are deployed to different machines in the network, the nginxReverseProxy uses a network link to forward incoming requests to the web application.

By running the following command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix will deploy the processes to the target machines defined in the distribution model.

The result is the following deployment architecture:

As may be noticed by looking at the above diagram, the process dependency manifest itself as a network link managed as an inter-dependency by Disnix.

Conclusion

In this blog post, I have described a Nix-based functional organization for managing processes based on four simple Nix packaging conventions. This approach offers the following benefits:

Integration with many process managers that manage the lifecycle of a process (in this particular blog post: using sysvinit scripts).
The ability to relocate state to other locations, which is useful to facilitate unprivileged user deployments.
The ability to create multiple instances of processes, by making conflicting properties configurable.
Disabling user switching, which is useful to facilitate unprivileged user deployments.
It can be used on any Linux system that has the Nix package manager installed. It can be used on NixOS, but NixOS is not a requirement.

Related work

Integrating process management with Nix package deployment is not a new subject, nor something that is done for the first time.

Many years ago, there was the "trace" Subversion repository (that was named after the research project TraCE: Transparent Configuration Environments funded by NWO/Jacquard), the repository in which all Nix-related development was done before the transition was made to GitHub (before 2012).

In the trace repository, there was also a services project that could be used to generate sysvinit-like scripts that could be used on any Linux distribution, and several non-Linux systems as well, such as FreeBSD.

Eelco Dolstra's PhD thesis Chapter 9 describes a distributed deployment prototype that extends the init script approach to networks of machines. The prototype facilitates the distribution of init scripts to remote machines and heterogeneous operating systems deployment -- an init script can be built for multiple operating systems, such as Linux and FreeBSD.

Although the prototype shares some concepts with Disnix and the process management described in this blog post support, it also lacks many features -- it has no notion of process dependencies, inter-dependencies, the ability to separate services/processes and infrastructure, and to specify distribution mappings between process and target machines including the deployment of redundant instances.
>
Originally, NixOS used to work with the generated scripts from services sub project in the trace repository, but quite quickly adopted Upstart as its init system. Gradually, the init scripts and upstart jobs got integrated, and eventually replaced by Upstart jobs completely. As a result, it was no longer possible to run services independently of NixOS.

NixOS is a Linux distribution whose static aspects are fully managed by Nix, including user packages, configuration files, the Linux kernel, and kernel modules. NixOS machine configurations are deployed from a single declarative specification.

Although NixOS is an extension of Nix deployment principles to machine-level deployment, a major conceptual difference between NixOS and the Nix packages repository is that NixOS generates a big data structure made out of all potential configuration options that NixOS provides. It uses this (very big) generated data structure as an input for an activation script that will initialize all dynamic system parts, such as populating the state directories (e.g. /var) and loading systemd jobs.

In early incarnations of NixOS, the organization of the repository was quite monolithic -- there was one NixOS file that defines all configuration options for all possible system configuration aspetcts, one file that defines the all the system user accounts, one file that defines all global configuration files in /etc. When it was desired to add a new system service, all these global configuration files need to be modified.

Some time later (mid 2009), the NixOS module system was introduced that makes it possible to isolate all related configuration aspects of, for example, a system service into a separate module. Despite the fact that configuration aspects are isolated, the NixOS module system has the ability (through a concept called fixed points) to refer to properties of the entire configuration. The NixOS module system merges all configuration aspects of all modules into a single configuration data structure.

The NixOS module system is quite powerful. In many ways, it is much more powerful than the process management approach described in this blog post. The NixOS module system allows you to refer, override and adjust any system configuration aspect in any module.

For example, a system service, such as the OpenSSH server, can automatically configure the firewall module in such a way that it will open the SSH port (port 22). With the functional approach described in this blog post, everything has to be made explicit and must be propagated through function arguments. This is probably more memory efficient, but a lot less flexible, and more tedious to write.

There are also certain things that NixOS and the NixOS module system cannot do. For example, with NixOS, it is not possible to create multiple instances of system services which the process management conventions described in this blog post can.

NixOS has another drawback -- evaluating system configurations requires all possible NixOS configuration options to be evaluated. There are actually quite a few of of them.

As a result, evaluating a NixOS configuration is quite slow and memory consuming. For single systems, this is typically not a big problem, but for networked NixOS/NixOps configurations, this may be a problem -- for example, I have an old laptop with 4 GiB of RAM that can no longer deploy a test network of three VirtualBox machines using the latest stable NixOS release (19.09), because the Nix evaluator runs out of memory.

Furthermore, NixOS system services can only be used when you install NixOS as your system's software distribution. It is currently not possible to install Nix on a conventional Linux distribution and use NixOS' system services (systemd services) independently of the entire operating system.

The lack of being able to deploy system services independently is not a limitation of the NixOS module system -- there is also an external project called nix-darwin that uses the NixOS module system to generate launchd services, that can be run on top of macOS, that is not managed by the Nix package manager.

The idea to have a separate function header for creating instances of processes is also not entirely new -- a couple of years ago I have revised the internal deployment model of Disnix to support multiple container instances.

In a Disnix-context, containers can represent anything that can host multiple service instances, such as a process manager, application container, or database management system. I was already using the convention to have a separate function header that makes it possible to create multiple instances of services. In this blog post, I have extended this formalism specifically for managing processes.

Discussion

In this blog post, I have picked sysvinit scripts for process management. The reason why I have picked an old-fashioned solution is not that I consider this to be the best process management facility, or that systemd, the init system that NixOS uses, is a bad solution.

My first reason to choose sysvinit scripts is because it is more universally supported than systemd.

The second reason is that I want to emphasize the value that a functional organization can provide, independent of the process management solution.

Using sysvinit scripts for managing process have all kinds of drawbacks and IMO there is a legitimate reason why alternatives exist, such as systemd (but also other solutions).

For example, controlling daemonized processes is difficult and fragile -- the convention that daemons should follow is to create PID files, but it is not a hard guarantee daemons will comply and that nothing will go wrong. As a result, a daemonized process may escape control of the process manager. systemd, for example, puts all processes that it needs to control in a cgroup and as a result, cannot escape systemd's control.

Furthermore, you may also want to use the more advanced features of the Linux kernel, such as namespaces and cgroups to prevent process from interfering with other processes on the system and the available system resources that a system provides. Namespaces and cgroups are a first class feature in systemd.

If you do not like sysvinit scripts: the functional organization described in this blog post is not specifically designed for sysvinit -- it is actually process manager agnostic. I have also implemented a function called: createSystemdService that makes it possible to construct systemd services.

The following Nix expression composes a systemd service for the web application process, shown earlier:


{stdenv, createSystemdService}:
{port, instanceSuffix ? ""}:

let
  webapp = (import ./webapp {}).package;
  instanceName = "webapp${instanceSuffix}";
in
createSystemdService {
  name = instanceName;

  environment = {
    PORT = port;
  };

  Unit = {
    Description = "Example web application";
    Documentation = http://example.com;
  };

  Service = {
    ExecStart = "${webapp}/lib/node_modules/webapp/app.js";
  };
}

I also tried supervisord -- we can write the following Nix expression to compose a supervisord program configuration file for the web application process:


{stdenv, createSupervisordProgram}:
{port, instanceSuffix ? ""}:

let
  webapp = (import ./webapp {}).package;
  instanceName = "webapp${instanceSuffix}";
in
createSupervisordProgram {
  name = instanceName;

  command = "${webapp}/lib/node_modules/webapp/app.js";
  environment = {
    PORT = port;
  };
}

Switching process managers retains our ability to benefit from the facilities that the functional configuration framework provides -- we can use it manage process dependencies, configure state directories, disable user management and when we use Disnix: manage inter-dependencies and bind it to services that are not processes.

Despite the fact that sysvinit scripts are primitive, there are also two advantages that I see over more "modern alternatives", such as systemd:

Systemd and supervisord require the presence of a deamon that manages processes (i.e. the systemd and supervisord deamons). sysvinit scripts are self-contained from a process management perspective -- the Nix package manager provides the package dependencies that the sysvinit scripts needs (e.g. basic shell utilities, sysvinit commands), but other than that, it does not require anything else.
We can also easily deploy sysvinit scripts to any Linux distribution that has the Nix package manager installed. There are no additional requirements. Systemd services, for example, require the presence of the systemd daemon. Furthermore, we also have to interfere with the host system's systemd service that may also be used to manage essential system services.
We can also easily use sysvinit scripts to deploy processes as an unprivileged user to a machine that has a single-user Nix installation -- the sysvinit script infrastructure does not require any tools or daemons that require super user privileges.

Acknowledgements

I have borrowed the init-functions script from the LFS Bootscripts package of the Linux from Scratch project to get an implementation of the utility functions that the LSB standard describes.

Availability and future work

The functionality described in this blog post is still a work in progress and only a first milestone in a bigger objective.

The latest implementation of the process management framework can be found in my experimental Nix process management repository. The sysvinit-script Dysnomia plugin resides in an experimental branch of the Dysnomia repository.

In the next blog post, I will introduce another interesting concept that we can integrate into the functional process management framework.

Hercules Labs: Launching Hercules CI

Tue, 22 Oct 2019 00:00:00 +0000

In March 2018 we set ourselves a mission to provide seamless infrastructure to teams using Nix -in day-to-day software development.

+ Graham Christensen: ZFS Datasets for NixOS + http://grahamc.com//blog/nixos-on-zfs + http://grahamc.com/blog/nixos-on-zfs +

The outdated and historical nature of the Filesystem Hierarchy +Standard means traditional Linux distributions have to go to great +lengths to separate “user data” from “system data.”

+ +

NixOS’s filesystem architecture does cleanly separate user data from +system data, and has a much easier job to do.

+ +

Traditional Linuxes

+ +

Because FHS mixes these two concerns across the entire hierarchy, +splitting these concerns requires identifying every point across +dozens of directories where the data is the system’s or the user’s. +When adding ZFS to the mix, the installers typically have to create +over a dozen datasets to accomplish this.

+ +

For example, Ubuntu’s upcoming ZFS support creates 16 datasets:

+ +

rpool/
+├── ROOT
+│   └── ubuntu_lwmk7c
+│       ├── log
+│       ├── mail
+│       ├── snap
+│       ├── spool
+│       ├── srv
+│       ├── usr
+│       │   └── local
+│       ├── var
+│       │   ├── games
+│       │   └── lib
+│       │       ├── AccountServices
+│       │       ├── apt
+│       │       ├── dpkg
+│       │       └── NetworkManager
+│       └── www
+└── USERDATA
+

+ +

Going through the great pains of separating this data comes with +significant advantages: a recursive snapshot at any point in the tree +will create an atomic, point-in-time snapshot of every dataset below.

+ +

This means in order to create a consistent snapshot of the system +data, an administrator would only need to take a recursive snapshot +at ROOT. The same is true for user data: take a recursive snapshot of +USERDATA and all user data is saved.

+ +

NixOS

+ +

Because Nix stores all of its build products in /nix/store, NixOS +doesn’t mingle these two concerns. NixOS’s runtime system, installed +packages, and rollback targets are all stored in /nix.

+ +

User data is not.

+ +

This removes the entire complicated tree of datasets to facilitate +FHS, and leaves us with only a few needed datasets.

+ +

Datasets

+ +

Design for the atomic, recursive snapshots when laying out the +datasets.

In June 2018 we released a solution for developers to easily share binary caches, -trusted today by over a thousand developers.

In particular, I don’t back up the /nix directory. This entire +directory can always be rebuilt later from the system’s +configuration.nix, and isn’t worth the space.

In Octobter 2018 we showed the very first demo of Hercules CI at NixCon 2018.

One way to model this might be splitting up the data into three +top-level datasets:

In March 2019 we added added support for private binary caches.

tank/
+├── local
+│   └── nix
+├── system
+│   └── root
+└── user
+    └── home
+

+ +

In tank/local, I would store datasets that should almost never be +snapshotted or backed up. tank/system would store data that I would +want periodic snapshots for. Most importantly, tank/user would +contain data I want regular snapshots and backups for, with a long +retention policy.

+ +

From here, you could add a ZFS dataset per user:

+ +

tank/
+├── local
+│   └── nix
+├── system
+│   └── root
+└── user
+    └── home
+        ├── grahamc
+        └── gustav
+

Since April 2019 we have been gradually giving out early access to the preview release with over a hundred participating developers.

Or a separate dataset for /var:

Today

tank/
+├── local
+│   └── nix
+├── system
+│   ├── var
+│   └── root
+└── user
+

We are announcing general availability of continuous integration specialized for Nix projects.

Importantly, this gives you three buckets for independent and +regular snapshots.

Check out the landing page to get started.

The important part is having /nix under its own top-level dataset. +This makes it a “cousin” to the data you do want backup coverage on, +making it easier to take deep, recursive snapshots atomically.

Going forward

Properties

In the coming months we’re going to work closely with customers to polish the experience and continue to save developer’s time.

Enable compression with compression=on. Specifying on instead of +lz4 or another specific algorithm will always pick the best +available compression algorithm.
The dataset containing journald’s logs (where /var lives) should +have xattr=sa and acltype=posixacl set to allow regular users to +read their journal.
Nix doesn’t use atime, so atime=off on the /nix dataset is +fine.
NixOS requires (as of 2020-04-11) mountpoint=legacy for all +datasets. NixOS does not yet have tooling to require implicitly +created ZFS mounts to settle before booting, and mountpoint=legacy +plus explicit mount points in hardware-configuration.nix will +ensure all your datasets are mounted at the right time.

+ +

I don’t know how to pick ashift, and usually just allow ZFS to guess +on my behalf.

+ +

Partitioning

+ +

I only create two partitions:

+ +

/boot formatted vfat for EFI, or ext4 for BIOS
The ZFS dataset partition.

For support (with getting started and other questions), -contact me at domen@hercules-ci.com so we can set you up -and make sure you get the most out of our CI.

There are spooky articles saying only give ZFS entire disks. The +truth is, you shouldn’t split a disk into two active partitions. +Splitting the disk this way is just fine, since /boot is rarely +read or written.

Subscribe to @hercules_ci for updates.

+
Note: If you do partition the disk, make sure you set the disk’s +scheduler to none. ZFS takes this step automatically if it does +control the entire disk.
-
+
On NixOS, you an set your scheduler to none via:
-
What we do
+
{ boot.kernelParams = [ "elevator=none" ]; }
+
+

Automated hosted infrastructure for Nix, reliable and reproducible developer tooling, -to speed up adoption and lower integration cost. We offer -Continuous Integration and Binary Caches.

Matthew Bauer: Improved performance in Nixpkgs

support@nixbuild.net (nixbuild.net) — Fri, 27 Mar 2020 00:00:00 +0000

1 Avoiding subshells

+ nixbuild.net: New nixbuild.net Resources + https://blog.nixbuild.net/posts/2020-03-27-nixbuild-net-beta.html + https://blog.nixbuild.net/posts/2020-03-27-nixbuild-net-beta.html +

On the support side of the nixbuild.net service, two new resources have been published:

docs.nixbuild.net, collecting all available documentation for nixbuild.net users.
The nixbuild.net feedback repository on GitHub, providing a way to report issues or ask questions related to the service.

These resources are mainly useful for nixbuild.net beta users, but they are open to anyone. And anyone is of course welcome to request a free beta account for evaluating nixbuild.net, by just sending me an email.

Matthew Bauer: Announcing Nixiosk

Sun, 13 Oct 2019 00:00:00 +0000

+Today I’m announcing a project I’ve been working on for the last few +weeks. I’m calling it Nixiosk which is kind of a smashing together of +the words NixOS and Kiosk. The idea is to have an easy way to make +locked down, declarative systems +

-A common complain in using Nixpkgs is that things can become slow when -you have lots of dependencies. Processing of build inputs is processed -in Bash which tends to be pretty hard to make performant. Bash doesn’t -give us any way to loop through dependencies in parallel, so we end up -with pretty slow Bash. Luckily, someone has found some ways to speed -this up with some clever tricks in the setup.sh script. +My main application of this is my two Raspberry Pi systems that I own. +Quite a few people have installed NixOS on these systems, but usually +they are starting from some prebuilt image. A major goal of this +project is to make it easy to build these images yourself. For this to +work, I’ve had to make lots of changes to NixOS cross-compilation +ecosystem, but the results seem to be very positive. I also want the +system to be locked down so that no user can login directly on the +machine. Instead, all administration is done on a remote machine, and +deployed through SSH and Nix remote builders.

- -

1.1 Pull request

-Albert Safin (@xzfc on GitHub) made an excellent PR that promises to -improve performance for all users of Nixpkgs. The PR is available at -PR #69131. The basic idea is to avoid invoking “subshells” in Bash. A -subshell is basically anything that uses $(cmd ...). Each subshell -requires forking a new process which has a constant time cost that -ends up being ~2ms. This isn’t much in isolation, but adds up in big -loops. +Right now, I have RetroArch (a frontend for a bunch of emulators) on +my Raspberry Pi 4, and Epiphany (a web browser) on my Raspberry Pi 0. +Both systems seem to be working pretty well.

-Subshells are usually used in Bash because they are convenient and -easy to reason about. It’s easy to understand how a subshell works as -it’s just substituting the result of one command into another’s -arguments. We don’t usually care about the performance cost of -subshells. In the hot path of Nixpkgs’ setup.sh, however, it’s -pretty important to squeeze every bit of performance we can. +GitHub: https://github.com/matthewbauer/nixiosk

1 Deploying

1.1 Install Nix

-A few interesting changes were required to make this work. I’ll go -through and document what there are. More information can be found at -the Bash manual. +If you haven’t already, you need to install Nix. This can be done +through the installer:

diff --git a/pkgs/stdenv/generic/setup.sh b/pkgs/stdenv/generic/setup.sh
-index 326a60676a26..60067a4051de 100644
---- a/pkgs/stdenv/generic/setup.sh
-+++ b/pkgs/stdenv/generic/setup.sh
-@@ -98,7 +98,7 @@ _callImplicitHook() {
- # hooks exits the hook, not the caller. Also will only pass args if
- # command can take them
- _eval() {
--    if [ "$(type -t "$1")" = function ]; then
-+    if declare -F "$1" > /dev/null 2>&1; then
-         set +u
-         "$@" # including args
-     else
-
+$ bash <(curl -L https://nixos.org/nix/install)

1.2 Cache

-The first change is pretty easy to understand. It just replaces the -type call with a declare call, utilizing an exit code in place of -stdout. Unfortunately, declare is a Bashism which is not available -in all POSIX shells. It’s been ill defined whether Bashisms can be -used in Nixpkgs, but we now will require Nixpkgs to be sourced with -Bash 4+. +To speed things up, you should setup a binary cache for nixiosk. This +can be done easily through Cachix. First, install Cachix:

diff --git a/pkgs/stdenv/generic/setup.sh b/pkgs/stdenv/generic/setup.sh
-index 60067a4051de..7e7f8739845b 100644
---- a/pkgs/stdenv/generic/setup.sh
-+++ b/pkgs/stdenv/generic/setup.sh
-@@ -403,6 +403,7 @@ findInputs() {
-     # The current package's host and target offset together
-     # provide a <=-preserving homomorphism from the relative
-     # offsets to current offset
-+    local -i mapOffsetResult
-     function mapOffset() {
-         local -ri inputOffset="$1"
-         if (( "$inputOffset" <= 0 )); then
-@@ -410,7 +411,7 @@ findInputs() {
-         else
-             local -ri outputOffset="$inputOffset - 1 + $targetOffset"
-         fi
--        echo "$outputOffset"
-+        mapOffsetResult="$outputOffset"
-     }
-
-     # Host offset relative to that of the package whose immediate
-@@ -422,8 +423,8 @@ findInputs() {
-
-         # Host offset relative to the package currently being
-         # built---as absolute an offset as will be used.
--        local -i hostOffsetNext
--        hostOffsetNext="$(mapOffset relHostOffset)"
-+        mapOffset relHostOffset
-+        local -i hostOffsetNext="$mapOffsetResult"
-
-         # Ensure we're in bounds relative to the package currently
-         # being built.
-@@ -441,8 +442,8 @@ findInputs() {
-
-             # Target offset relative to the package currently being
-             # built.
--            local -i targetOffsetNext
--            targetOffsetNext="$(mapOffset relTargetOffset)"
-+            mapOffset relTargetOffset
-+            local -i targetOffsetNext="$mapOffsetResult"
-
-             # Once again, ensure we're in bounds relative to the
-             # package currently being built.
-
+$ nix-env -iA cachix -f https://cachix.org/api/v1/install

-Similarly, this change makes mapOffset set to it’s result to -mapOffsetResult instead of printing it to stdout, avoiding the -subshell. Less functional, but more performant! +Then, use the nixiosk cache:

diff --git a/pkgs/stdenv/generic/setup.sh b/pkgs/stdenv/generic/setup.sh
-index 7e7f8739845b..e25ea735a93c 100644
---- a/pkgs/stdenv/generic/setup.sh
-+++ b/pkgs/stdenv/generic/setup.sh
-@@ -73,21 +73,18 @@ _callImplicitHook() {
-     set -u
-     local def="$1"
-     local hookName="$2"
--    case "$(type -t "$hookName")" in
--        (function|alias|builtin)
--            set +u
--            "$hookName";;
--        (file)
--            set +u
--            source "$hookName";;
--        (keyword) :;;
--        (*) if [ -z "${!hookName:-}" ]; then
--                return "$def";
--            else
--                set +u
--                eval "${!hookName}"
--            fi;;
--    esac
-+    if declare -F "$hookName" > /dev/null; then
-+        set +u
-+        "$hookName"
-+    elif type -p "$hookName" > /dev/null; then
-+        set +u
-+        source "$hookName"
-+    elif [ -n "${!hookName:-}" ]; then
-+        set +u
-+        eval "${!hookName}"
-+    else
-+        return "$def"
-+    fi
-     # `_eval` expects hook to need nounset disable and leave it
-     # disabled anyways, so Ok to to delegate. The alternative of a
-     # return trap is no good because it would affect nested returns.
+$ cachix use nixiosk
+
+

+ +

1.3 Configuration

+To make things simple, it just reads from an ad-hoc JSON file that +describe the hardware plus some other customizations. It looks like +this: +

{
+    "hostName": "nixiosk",
+    "hardware": "raspberryPi4",
+    "authorizedKeys": [],
+    "program": {
+        "package": "epiphany",
+        "executable": "/bin/epiphany",
+        "args": ["https://en.wikipedia.org/"]
+    },
+    "networks": {
+        "my-router": "0000000000000000000000000000000000000000000000000000000000000000",
+    },
+    "locale": {
+        "timeZone": "America/New_York",
+        "regDom": "US",
+        "lang": "en_US.UTF-8"
+    },
+    "localSystem": {
+        "system": "x86_64-linux",
+        "sshUser": "me",
+        "hostName": "my-laptop-host",
+    }
+}

-This change replaces the type -t command with calls to specific Bash -builtins. declare -F tells us if the hook is a function, type -p -tells us if hookName is a file, and otherwise -n tells us if the -hook is non-empty. Again, this introduces a Bashism. +Here’s a basic idea of what each of these fields do:

hostName: Name of the host to use. If mDNS is configured on your +network, this can be used to identify the IP address of the device +via “<hostName>.local”.
hardware: A string describing what hardware we are using. Valid +values currently are “raspberryPi0”, “raspberryPi1”, “raspberryPi2”, +“raspberryPi3”, “raspberryPi4”.
authorizedKeys: A list of SSH public keys that are authorized to +make changes to your device. Note this is required because no +passwords will be set for this system.
program: What to do in the kiosk. This should be a Nixpkgs attribute +(package), an executable in that package, and a list of args.
networks: This is a name/value pairing of SSIDs to PSK passphrases. +This can be found with the wpa_passphrase(8) command from +wpa_supplicant.
locale: This provides some information of what localizations to use. +You can set regulation domain, language, time zone via “regDom”, +“lang”, and “timeZone”. If unspecified, defaults to US / English / +New York.
localSystem: Information on system to use for remote builder. +Optional.

+ +

1.4 Initial deployment

-In the worst case, this does replace one case with multiple if -branches. But since most hooks are functions, most of the time this -ends up being a single if. +The deployment is pretty easy provided you have Nix installed. Here +are some steps:

diff --git a/pkgs/stdenv/generic/setup.sh b/pkgs/stdenv/generic/setup.sh
-index e25ea735a93c..ea550a6d534b 100644
---- a/pkgs/stdenv/generic/setup.sh
-+++ b/pkgs/stdenv/generic/setup.sh
-@@ -449,7 +449,8 @@ findInputs() {
-             [[ -f "$pkg/nix-support/$file" ]] || continue
-
-             local pkgNext
--            for pkgNext in $(< "$pkg/nix-support/$file"); do
-+            read -r -d '' pkgNext < "$pkg/nix-support/$file" || true
-+            for pkgNext in $pkgNext; do
-                 findInputs "$pkgNext" "$hostOffsetNext" "$targetOffsetNext"
-             done
-         done
-
+$ git clone https://github.com/matthewbauer/nixiosk.git
+$ cd nixiosk/
+$ cp nixiosk.json.sample nixiosk.json

-This change replaces the $(< ) call with a read call. This is a -little surprising since read is using an empty delimiter '' -instead of a new line. This replaces one Bashsism $(< ) with another -in -d. And, the result, gets rid of a remaining subshell usage. +Now you need to make some changes to nixiosk.json to reflect what you +want your system to do. The important ones are ‘authorizedKeys’ and +‘networks’ so that your systems can startup and you can connect to it. +

+ +

+If you have an SSH key setup, you can get its value with:

diff --git a/pkgs/build-support/bintools-wrapper/setup-hook.sh b/pkgs/build-support/bintools-wrapper/setup-hook.sh
-index f65b792485a0..27d3e6ad5120 100644
---- a/pkgs/build-support/bintools-wrapper/setup-hook.sh
-+++ b/pkgs/build-support/bintools-wrapper/setup-hook.sh
-@@ -61,9 +61,8 @@ do
-     if
-         PATH=$_PATH type -p "@targetPrefix@${cmd}" > /dev/null
-     then
--        upper_case="$(echo "$cmd" | tr "[:lower:]" "[:upper:]")"
--        export "${role_pre}${upper_case}=@targetPrefix@${cmd}";
--        export "${upper_case}${role_post}=@targetPrefix@${cmd}";
-+        export "${role_pre}${cmd^^}=@targetPrefix@${cmd}";
-+        export "${cmd^^}${role_post}=@targetPrefix@${cmd}";
-     fi
- done
+$ cat $HOME/.ssh/id_rsa.pub
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC050iPG8ckY/dj2O3ol20G2lTdr7ERFz4LD3R4yqoT5W0THjNFdCqavvduCIAtF1Xx/OmTISblnGKf10rYLNzDdyMMFy7tUSiC7/T37EW0s+EFGhS9yOcjCVvHYwgnGZCF4ec33toE8Htq2UKBVgtE0PMwPAyCGYhFxFLYN8J8/xnMNGqNE6iTGbK5qb4yg3rwyrKMXLNGVNsPVcMfdyk3xqUilDp4U7HHQpqX0wKrUvrBZ87LnO9z3X/QIRVQhS5GqnIjRYe4L9yxZtTjW5HdwIq1jcvZc/1Uu7bkMh3gkCwbrpmudSGpdUlyEreaHOJf3XH4psr6IMGVJvxnGiV9 mbauer@dellbook
+
+

+ +

+which will give you a line for “authorizedKeys” like: +

"authorizedKeys": ["ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC050iPG8ckY/dj2O3ol20G2lTdr7ERFz4LD3R4yqoT5W0THjNFdCqavvduCIAtF1Xx/OmTISblnGKf10rYLNzDdyMMFy7tUSiC7/T37EW0s+EFGhS9yOcjCVvHYwgnGZCF4ec33toE8Htq2UKBVgtE0PMwPAyCGYhFxFLYN8J8/xnMNGqNE6iTGbK5qb4yg3rwyrKMXLNGVNsPVcMfdyk3xqUilDp4U7HHQpqX0wKrUvrBZ87LnO9z3X/QIRVQhS5GqnIjRYe4L9yxZtTjW5HdwIq1jcvZc/1Uu7bkMh3gkCwbrpmudSGpdUlyEreaHOJf3XH4psr6IMGVJvxnGiV9 mbauer@dellbook"],

-This replace a call to tr with a usage of the ^^. -${parameter^^pattern} is a Bash 4 feature and allows you to -upper-case a string without calling out to tr. +and you can get a PSK value for your WiFi network with:

diff --git a/pkgs/build-support/bintools-wrapper/setup-hook.sh b/pkgs/build-support/bintools-wrapper/setup-hook.sh
-index 27d3e6ad5120..2e15fa95c794 100644
---- a/pkgs/build-support/bintools-wrapper/setup-hook.sh
-+++ b/pkgs/build-support/bintools-wrapper/setup-hook.sh
-@@ -24,7 +24,8 @@ bintoolsWrapper_addLDVars () {
-         # Python and Haskell packages often only have directories like $out/lib/ghc-8.4.3/ or
-         # $out/lib/python3.6/, so having them in LDFLAGS just makes the linker search unnecessary
-         # directories and bloats the size of the environment variable space.
--        if [[ -n "$(echo $1/lib/lib*)" ]]; then
-+        local -a glob=( $1/lib/lib* )
-+        if [ "${#glob[*]}" -gt 0 ]; then
-             export NIX_${role_pre}LDFLAGS+=" -L$1/lib"
-         fi
-     fi
+$ nix run nixpkgs.wpa_supplicant -c wpa_passphrase my-network
+network={
+        ssid="my-network"
+        #psk="abcdefgh"
+        psk=17e76a6490ac112dbeba996caa7cd1387c6ebf6ce721ef704f92b681bb2e9000
+}

-Here, we are checking for whether any files exist in /lib/lib* using -a glob. It originally used a subshell to check if the result was -empty, but this change replaces it with the Bash ${#parameter} -length operation. +so your .json file looks like:

diff --git a/pkgs/stdenv/generic/setup.sh b/pkgs/stdenv/generic/setup.sh
-index 311292169ecd..326a60676a26 100644
---- a/pkgs/stdenv/generic/setup.sh
-+++ b/pkgs/stdenv/generic/setup.sh
-@@ -17,7 +17,8 @@ fi
- # code). The hooks for <hookName> are the shell function or variable
- # <hookName>, and the values of the shell array ‘<hookName>Hooks’.
- runHook() {
--    local oldOpts="$(shopt -po nounset)"
-+    local oldOpts="-u"
-+    shopt -qo nounset || oldOpts="+u"
-     set -u # May be called from elsewhere, so do `set -u`.
-
-     local hookName="$1"
-@@ -32,7 +33,7 @@ runHook() {
-         set -u # To balance `_eval`
-     done
-
--    eval "${oldOpts}"
-+    set "$oldOpts"
-     return 0
- }
-
-@@ -40,7 +41,8 @@ runHook() {
- # Run all hooks with the specified name, until one succeeds (returns a
- # zero exit code). If none succeed, return a non-zero exit code.
- runOneHook() {
--    local oldOpts="$(shopt -po nounset)"
-+    local oldOpts="-u"
-+    shopt -qo nounset || oldOpts="+u"
-     set -u # May be called from elsewhere, so do `set -u`.
-
-     local hookName="$1"
-@@ -57,7 +59,7 @@ runOneHook() {
-         set -u # To balance `_eval`
-     done
-
--    eval "${oldOpts}"
-+    set "$oldOpts"
-     return "$ret"
- }
-
-@@ -500,10 +502,11 @@ activatePackage() {
-     (( "$hostOffset" <= "$targetOffset" )) || exit -1
-
-     if [ -f "$pkg" ]; then
--        local oldOpts="$(shopt -po nounset)"
-+        local oldOpts="-u"
-+        shopt -qo nounset || oldOpts="+u"
-         set +u
-         source "$pkg"
--        eval "$oldOpts"
-+        set "$oldOpts"
-     fi
-
-     # Only dependencies whose host platform is guaranteed to match the
-@@ -522,10 +525,11 @@ activatePackage() {
-     fi
-
-     if [[ -f "$pkg/nix-support/setup-hook" ]]; then
--        local oldOpts="$(shopt -po nounset)"
-+        local oldOpts="-u"
-+        shopt -qo nounset || oldOpts="+u"
-         set +u
-         source "$pkg/nix-support/setup-hook"
--        eval "$oldOpts"
-+        set "$oldOpts"
-     fi
- }
-
-@@ -1273,17 +1277,19 @@ showPhaseHeader() {
-
- genericBuild() {
-     if [ -f "${buildCommandPath:-}" ]; then
--        local oldOpts="$(shopt -po nounset)"
-+        local oldOpts="-u"
-+        shopt -qo nounset || oldOpts="+u"
-         set +u
-         source "$buildCommandPath"
--        eval "$oldOpts"
-+        set "$oldOpts"
-         return
-     fi
-     if [ -n "${buildCommand:-}" ]; then
--        local oldOpts="$(shopt -po nounset)"
-+        local oldOpts="-u"
-+        shopt -qo nounset || oldOpts="+u"
-         set +u
-         eval "$buildCommand"
--        eval "$oldOpts"
-+        set "$oldOpts"
-         return
-     fi
-
-@@ -1313,10 +1319,11 @@ genericBuild() {
-
-         # Evaluate the variable named $curPhase if it exists, otherwise the
-         # function named $curPhase.
--        local oldOpts="$(shopt -po nounset)"
-+        local oldOpts="-u"
-+        shopt -qo nounset || oldOpts="+u"
-         set +u
-         eval "${!curPhase:-$curPhase}"
--        eval "$oldOpts"
-+        set "$oldOpts"
-
-         if [ "$curPhase" = unpackPhase ]; then
-             cd "${sourceRoot:-.}"
+"networks": {
+  "my-network": "17e76a6490ac112dbeba996caa7cd1387c6ebf6ce721ef704f92b681bb2e9000",
+},
+
+

+Now, after inserting your Raspberry Pi SD card into the primary slot, +you can deploy to it with: +

+ +

$ ./deploy.sh /dev/mmcblk0

-This last change is maybe the trickiest. $(shopt -po nounset) is -used to get the old value of nounset. The nounset setting tells -Bash to treat unset variables as an error. This is used temporarily -for phases and hooks to enforce this property. It will be reset to its -previous value after we finish evaling the current phase or hook. To -avoid the subshell here, the stdout provided in shopt -po is -replaced with an exit code provided in shopt -qo nounset. If the -shopt -qo nounset fails, we set oldOpts to +u, otherwise it is -assumed that it is -u. +You can now eject your SD card and insert it into your Raspberry Pi. +It will boot immediately to an Epiphany browser, loading +en.wikipedia.org.

-This commit was first merged in on September 20, but it takes a while -for it to hit master. Today, it was finally merged into master -(October 13) in 4e6826a so we can finally can see the benefits from -it! +Troubleshooting steps can be found in the README.

1.2 Benchmarking

1.5 Redeployments

-Hyperfine makes it easy to compare differences in timings. You can -install it locally with: +You can pretty easily make changes to a running system given you have +SSH access. This is as easy as cloning the running config:

$ nix-env -iA nixpkgs.hyperfine
+$ git clone ssh://root@nixiosk.local/etc/nixos/configuration.git nixiosk-configuration
+$ cd nixiosk-configuration

-Here are some of the results: +Then, make some changes in your repo. After your done, you can just +run ‘git push’ to redeploy.

$ hyperfine --warmup 3 \
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p stdenv --run :' \
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p stdenv --run :'
-Benchmark #1: nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p stdenv --run :
-  Time (mean ± σ):     436.4 ms ±   8.5 ms    [User: 324.7 ms, System: 107.8 ms]
-  Range (min … max):   430.8 ms … 459.6 ms    10 runs
-
-Benchmark #2: nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p stdenv --run :
-  Time (mean ± σ):     244.5 ms ±   2.3 ms    [User: 190.7 ms, System: 34.2 ms]
-  Range (min … max):   241.8 ms … 248.3 ms    12 runs
-
-Summary
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p stdenv --run :' ran
-    1.79 ± 0.04 times faster than 'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p stdenv --run :'
+$ git add .
+$ git commit
+$ git push

$ hyperfine --warmup 3 \
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p i3.buildInputs --run :' \
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p i3.buildInputs --run :'
-Benchmark #1: nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p i3.buildInputs --run :
-  Time (mean ± σ):      3.428 s ±  0.015 s    [User: 2.489 s, System: 1.081 s]
-  Range (min … max):    3.404 s …  3.453 s    10 runs
-
-Benchmark #2: nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p i3.buildInputs --run :
-  Time (mean ± σ):     873.4 ms ±  12.2 ms    [User: 714.7 ms, System: 89.3 ms]
-  Range (min … max):   861.5 ms … 906.4 ms    10 runs
-
-Summary
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p i3.buildInputs --run :' ran
-    3.92 ± 0.06 times faster than 'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p i3.buildInputs --run :'
-

+You’ll see the NixOS switch-to-configuration log in your command +output. If all is successful, the system should immediately reflect +your changes. If not, the output of Git should explain what went +wrong. +

$ hyperfine --warmup 3 \
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p inkscape.buildInputs --run :' \
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p inkscape.buildInputs --run :'
-Benchmark #1: nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p inkscape.buildInputs --run :
-  Time (mean ± σ):      4.380 s ±  0.024 s    [User: 3.155 s, System: 1.443 s]
-  Range (min … max):    4.339 s …  4.409 s    10 runs
-
-Benchmark #2: nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p inkscape.buildInputs --run :
-  Time (mean ± σ):      1.007 s ±  0.011 s    [User: 826.7 ms, System: 114.2 ms]
-  Range (min … max):    0.995 s …  1.026 s    10 runs
-
-Summary
-  'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/4e6826a.tar.gz -p inkscape.buildInputs --run :' ran
-    4.35 ± 0.05 times faster than 'nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/33366cc.tar.gz -p inkscape.buildInputs --run :'
-

+Note, that some versions of the Raspberry Pi like the 0 and the 1 are +not big enough to redeploy the whole system. You will probably need to +setup remote builders. This is described in the README. +

+ +

2 Technology

+Here are some of the pieces that make the Kiosk system possible: +

+ +

Cage / Wayland: Cage is a Wayland compositor that allows only one +application to display at a time. This makes the system a true +Kiosk.
NixOS - A Linux distro built on top of functional package management.
Basalt: A tool to manage NixOS directly from Git. This allows doing +push-to-deploy directly to NixOS.
Plymouth: Nice graphical boot animations. Right now, it uses the +NixOS logo but in the future this should be configurable so that you +can include your own branding.
OpenSSH: Since no direct login is available, SSH is required for +remote administration.
Avahi: Configures mDNS registration for the system, allowing you to +remember host names instead of IP addresses.

-Try running these commands yourself, and compare the results. +I would also like to include some more tools to make administration +easier:

+ +

ddclient / miniupnp: Allow registering external IP address with a +DNS provider. This would enable administration outside of the +device’s immediate network.

1.3 Results

3 Project

-Avoiding subshells leads to a decrease in up to 4x of the time it used -to take. That multiplier is going to depend on precisely how many -inputs we are processing. It’s a pretty impressive improvement, and it -comes with no added cost. These kind of easy wins in performance are -pretty rare, and worth celebrating! +You can try it out right now if you have an Raspberry Pi system. Other +hardware is probably not too hard, but may require tweaking. The +project page is available at https://github.com/matthewbauer/nixiosk +and issues and pull requests are welcomed.

Hercules Labs: Agent 0.5.0 with Terraform support and simpler configuration

support@cachix.org (Domen Kožar) — Mon, 07 Oct 2019 00:00:00 +0000

Last week, we’ve released agent version 0.5.0. The main theme for the release is ease of installation. Running an agent should be as simple as possible, so we made:

- -

simplifications to the binary cache configuration
terraform modules and an example

- -

Follow getting started guide to set up your first agent.

- -

If you have and you’re using the module (NixOS, NixOps, nix-darwin) the update is entirely self-explanatory. Otherwise, check the notes.

- -

Trusted-user

- -

The agent now relies on being a trusted-user to the Nix daemon. The agent does not allow projects to execute arbitrary Nix store operations anyway. It may improve security since it simplifies configuration and secrets handling.

- -

The security model for the agent is simple at this point: only build git refs from your repository. This way, third-party contributors can not run arbitrary code on your agent system; only contributors with write access to the repo can.

- -

Talking about trust, we’ll share some details about securely doing CI for Open Source with Bors soon!

Craige McWhirter: Installing LineageOS 16 on a Samsung SM-T710 (gts28wifi)

Thu, 03 Oct 2019 23:04:41 +0000

Check the prerequisites
Backup any files you want to keep
Download LineageOS ROM and optional GAPPS package
Copy LineageOS image & additional packages to the SM-T710
Boot into recovery mode
Wipe the existing installation.
Format the device
Install LineageOS ROM and other optional ROMs.

- + Flying Circus: Our new NixOS 19.03 Platform Is Ready for Production + http://blog.flyingcircus.io/?p=5295 + https://blog.flyingcircus.io/2020/02/28/our-new-nixos-19-03-platform-is-ready-for-production/ +

We have developed our third-generation platform which is now based on NixOS 19.03. All provided components have been ported to the new platform and VMs are already running in production.

0 - Check the Prerequisites

The device already has the latest TWRP -installed.
Android debugging is enabled on the device
ADB is installed on -your workstation.
You have a suitably configured SD card as a back up handy.

- - -

I use this android.nix -to ensure my NixOS environment has the prerequisites -install and configured for it's side of the process.

- -

1 - Backup any Files You Want to Keep

- -

I like to use adb to pull the files from the device. There are also other -methods available too.

- -

$ adb pull /sdcard/MyFolder ./Downloads/MyDevice/
-

Usage of adb is documented at Android Debug Bridge

Most of our development work is done for the new platform and new features will be available only for it. We pull in security updates from upstream regularly and will follow new NixOS releases more quickly in the future. The old NixOS 15.09 platform still receives critical security and bug fixes.

2 - Download LineageOS ROM and optional GAPPS package

I downloaded -lineage-16.0-20191001-UNOFFICIAL-gts28wifi.zip from -gts28wifi.

I also downloaded Open GApps ARM, nano to enable -Google Apps.

Effective March 6, VMs created via customer self-service will use the 19.03 platform.

I could have also downloaded and installed LineageOS -addonsu and -addonsu-remove -but opted not to at this point.

3 - Copy LineageOS image & additional packages to the SM-T710

I use adb to copy the files files across:

You can find the documentation for the new platform here:

$ adb push ./lineage-16.0-20191001-UNOFFICIAL-gts28wifi.zip /sdcard/
-./lineage-16.0-20191001-UNOFFICIAL-gts28wifi.zip: 1 file pushed. 12.1 MB/s (408677035 bytes in 32.263s)
-$ adb push ./open_gapps-arm-9.0-nano-20190405.zip /sdcard/
-./open_gapps-arm-9.0-nano-20190405.zip: 1 file pushed. 11.1 MB/s (185790181 bytes in 15.948s)
-

I also copy both to the SD card at this point as the SM-T710 is an awful device -to work with and in many random cases will not work with ADB. When this -happens, I fall back to the SD card.

4 - Boot into recovery mode

https://flyingcircus.io/doc/guide/platform_nixos_2/index.html

I power the device off, then power it back into recovery mode by holding down -[home]+[volume up]+[power].

5 - Wipe the existing installation

Press Wipe then Advanced Wipe.

We recommend user profiles (done with buildEnv) in case your application needs specific packages in its environment:

Select:

Dalvik / Art Cache
System
Data
Cache

https://flyingcircus.io/doc/guide/platform_nixos_2/user_profile.html

Swipe Swipe to Wipe at the bottom of the screen.

Press Back to return to the Advanced Wipe screen.

Press the triangular "back" button once to return to the Wipe screen.

Upgrading 15.09 Machines

6 - Format the device

Press Format Data.

Type yes and press blue check mark at the bottom-right corner to commence -the format process.

Upgrading existing VMs online is supported and we have already done that for a number of VMs.
Sometimes however, it can be better to create new NixOS VMs in parallel and set up your applications there.

Press Back to return to the Advanced Wipe screen.

Press the triangular "back" button twice to return to the main screen.

7 - Install LineageOS ROM and other optional ROMs

Most managed components will just work after the upgrade. We are working on instructions for specific things that should be done before or after the upgrade.

Press Install, select the images you wish to install and swipe make it go.

Reboot when it's completed and you should be off and running wtth a brand new -LineageOS 16 on this tablet.

Hercules Labs: Post-mortem on recent Cachix downtime

Fri, 28 Feb 2020 14:16:49 +0000

On 6th of September, Cachix experienced 3 hours of downtime.

We’d like to let you know exactly what happened and what measures we have taken to prevent such an event from happening in the future.

If you’re a customer with a support contract in the “Guided” or “Managed” service classes
then we’ll approach you directly and discuss when and how to upgrade VMs in the coming months.

Timeline (UTC)

2019-09-06 17:15:05: cachix.org down alert triggered
2019-09-06 20:06:00: Domen gets out of MuniHac dinner in the basement and receives the alert
2019-09-06 20:19:00: Domen restarts server process
2019-09-06 20:19:38: cachix.org is back up

Observations

If you’re a customer in the “Hosted” service class then we recommend contacting our support team to discuss the upgrade.

The backend logs were full of:

Sep 06 17:02:34 cachix-production.cachix cachix-server[6488]: Network.Socket.recvBuf: resource vanished (Connection reset by peer)
-

And:

If you have questions …

(ConnectionFailure Network.BSD.getProtocolByName: does not exist (no such protocol name: tcp)))
-

Most importantly, there were no logs after downtime was triggered and until the restart:

Sep 06 17:15:48 cachix-production.cachix cachix-server[6488]: Network.Socket.recvBuf: resource vanished (Connection reset by peer)
-Sep 06 20:19:26 cachix-production.cachix systemd[1]: Stopping cachix server service...
-

As always: if you have any questions or comments then let us know and send us an email to support@flyingcircus.io.

nixbuild.net: Introducing nixbuild.net

support@nixbuild.net (nixbuild.net) — Tue, 18 Feb 2020 00:00:00 +0000

Our monitoring revealed an increased number of nginx connections and file handles (the time are in CEST - UTC+2):

Why nixbuild.net?

However, just when you’ve tasted those sweet distributed builds you very likely run into the issue of scaling.

What if you need a really big server to run your builds, but only really need it once or twice per day? You’ll be wasting a lot of money keeping that build server available.

Who are We?

nixbuild.net is developed and operated by me (Rickard Nilsson) and my colleague David Waern. We both have extensive experience in building Nix-based solutions, for ourselves and for various clients.

What does nixbuild.net Look Like?

To the end-user, a person or team using Nix for building software, nixbuild.net behaves just like any other remote build machine. As such, you can add it as an entry in your /etc/nix/machines file:

beta.nixbuild.net x86_64-linux - 100 1 big-parallel,benchmark

That’s all there is to it, now we can run builds using nixbuild.net!

Let’s try building the following silly build, just so we can see some action:

let pkgs = import <nixpkgs> { system = "x86_64-linux"; };
 
-
+in pkgs.runCommand "silly" {} ''
+  n=0
+  while (($n < 12)); do
+    date | tee -a $out
+    sleep 10
+    n=$(($n + 1))
+  done
+''

This build will run for 2 minutes and output the current date every ten seconds:

$ nix-build silly.nix
+these derivations will be built:
+  /nix/store/cy14fc13d3nzl65qp0sywvbjnnl48jf8-silly.drv
+building '/nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv' on 'ssh://beta.nixbuild.net'...
+Mon Feb 17 20:53:47 UTC 2020
+Mon Feb 17 20:53:57 UTC 2020
+Mon Feb 17 20:54:07 UTC 2020

You can see that Nix is telling us that the build is running on nixbuild.net!

The nixbuild.net Shell

nixbuild.net supports a simple shell interface that you can access through SSH. This shell allows you to retrieve information about your builds on the service.

For example, we can list the currently running builds:

$ ssh beta.nixbuild.net shell
+nixbuild.net> list builds --running
+10524 2020-02-17 21:05:20Z [40.95s] [Running]
+      /nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv

We can also get information about any derivation or nix store path that has been built:

nixbuild.net> show drv /nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv
+Derivation
+  path = /nix/store/khvphdj3q7nyim46jk97fjp174damrik-silly.drv
+  builds = 1
+  successful builds = 1
 
-Conclusions
+Outputs
+  out -> /nix/store/8c7sndr3npwmskj9zzp4347cnqh5p8q0-silly
 
+Builds
+  10524 2020-02-17 21:05:20Z [02:01] [Built]

This shell is under development, and new features are added continuously. A web-based frontend will also be implemented.

The Road Ahead

To finish up this short introduction to nixbuild.net, let’s talk a bit about our long-term goals for the service.

There are a number of features that basically just are nice side-effects of the design of nixbuild.net:

-
The main cause for downtime was hanged backend. The underlying cause was not identified -due to lack of information.
-
-
The backend was failing some requests due to reaching the limit of 1024 file descriptors.
-
-
The duration of the downtime was due to the absence of a telephone signal.
-
Building a large number of variants of the same derivation (a build matrix or some sort of parameter sweep) will take the same time as running a single build, since nixbuild.net can run all builds in parallel.
Running repeated builds to find issues related to non-determinism/reproducability will not take longer than running a single build.
A whole team/company can share the same account in nixbuild.net letting builds be shared in a cost-effective way. If everyone in a team delegates builds to nixbuild.net, the same derivation will never have to be built twice. This is similar to having a shared Nix cache, but avoids having to configure a cache and perform network uploads for each build artifact. Of course, nixbuild.net can be combined with a Nix cache too, if desired.

- -

What we’ve already done

Sander van der Burg: A declarative process manager-agnostic deployment framework based on Nix tooling

noreply@blogger.com (Sander van der Burg) — Sat, 15 Feb 2020 20:07:00 +0000

In a previous blog post written two months ago, I have introduced a new experimental Nix-based process framework, that provides the following features:

It uses the Nix expression language for configuring running process instances, including their dependencies. The configuration process is based on only a few simple concepts: function definitions to define constructors that generate process manager configurations, function invocations to compose running process instances, and Nix profiles to make collections of process configurations accessible from a single location.
The Nix package manager delivers all packages and configuration files and isolates them in the Nix store, so that they never conflict with other running processes and packages.
It identifies process dependencies, so that a process manager can ensure that processes are activated and deactivated in the right order.
The ability to deploy multiple instances of the same process, by making conflicting resources configurable.
Deploying processes/services as an unprivileged user.
Advanced concepts and features, such as namespaces and cgroups, are not required.


{createSystemVInitScript, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSystemVInitScript {
  name = instanceName;
  description = "Nginx";
  activities = {
    start = ''
      mkdir -p ${nginxLogDir}
      log_info_msg "Starting Nginx..."
      loadproc ${nginx}/bin/nginx -c ${configFile} -p ${stateDir}
      evaluate_retval
    '';
    stop = ''
      log_info_msg "Stopping Nginx..."
      killproc ${nginx}/bin/nginx
      evaluate_retval
    '';
    reload = ''
      log_info_msg "Reloading Nginx..."
      killproc ${nginx}/bin/nginx -HUP
      evaluate_retval
    '';
    restart = ''
      $0 stop
      sleep 1
      $0 start
    '';
    status = "statusproc ${nginx}/bin/nginx";
  };
  runlevels = [ 3 4 5 ];

  inherit dependencies instanceName;
}

In the above Nix expression, we specify five activities to manage the life-cycle of Nginx, a free/open source web server:

The start activity initializes the state of Nginx and starts the process (as a daemon that runs in the background).
stop stops the Nginx daemon.
reload instructs Nginx to reload its configuration
restart restarts the process
status shows whether the process is running or not.


{createSystemVInitScript, nginx,  stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSystemVInitScript {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile "-p" stateDir ];
  runlevels = [ 3 4 5 ];

  inherit dependencies instanceName;
}

Writing declarative deployment specifications for managed running processes


{createSupervisordProgram, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSupervisordProgram {
  name = instanceName;
  command = "mkdir -p ${nginxLogDir}; "+
    "${nginx}/bin/nginx -c ${configFile} -p ${stateDir}";
  inherit dependencies;
}


{createSystemdService, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createSystemdService {
  name = instanceName;
  Unit = {
    Description = "Nginx";
  };
  Service = {
    ExecStartPre = "+mkdir -p ${nginxLogDir}";
    ExecStart = "${nginx}/bin/nginx -c ${configFile} -p ${stateDir}";
    Type = "simple";
  };

  inherit dependencies;
}


{createManagedProcess, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createManagedProcess {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile" -p" "${stateDir}/${instanceName}" ];

  inherit dependencies instanceName;
}

sysvinit scripts, also known as LSB Init compliant scripts.
supervisord programs
systemd services
launchd services
BSD rc scripts
Windows services (via Cygwin's cygrunsrv)


{createManagedProcess, nginx, stateDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createManagedProcess {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-c" configFile" -p" "${stateDir}/${instanceName}" ];

  inherit dependencies instanceName;

  overrides = {
    sysvinit = {
      runlevels = [ 3 4 5 ];
    };
  };
}

Simulating process manager-agnostic concepts with no direct equivalents

For some process manager-agnostic concepts, process managers do not always have direct equivalents. In such cases, there is still the possibility to apply non-trivial simulation strategies.

Foreground processes or daemons


{createManagedProcess, nginx, stateDir, runtimeDir}:
{configFile, dependencies ? [], instanceSuffix ? ""}:

let
  instanceName = "nginx${instanceSuffix}";
  nginxLogDir = "${stateDir}/${instanceName}/logs";
in
createManagedProcess {
  name = instanceName;
  description = "Nginx";
  initialize = ''
    mkdir -p ${nginxLogDir}
  '';
  process = "${nginx}/bin/nginx";
  args = [ "-p" "${stateDir}/${instanceName}" "-c" configFile ];
  foregroundProcessExtraArgs = [ "-g" "daemon off;" ];
  daemonExtraArgs = [ "-g" "pid ${runtimeDir}/${instanceName}.pid;" ];

  inherit dependencies instanceName;

  overrides = {
    sysvinit = {
      runlevels = [ 3 4 5 ];
    };
  };
}

The process parameter specifies the process that needs to be started both in foreground mode and as a daemon. The args parameter specifies common command-line arguments that both the foreground and daemon process will use.
The foregroundProcessExtraArgs parameter specifies additional command-line arguments that are only used when the process is started in foreground mode. In the above example, it is used to provide Nginx the global directive that disables the daemon setting.
The daemonExtraArgs parameter specifies additional command-line arguments that are only used when the process is started as a daemon. In the above example, it used to provide Nginx a global directive with a PID file path that uniquely identifies the process instance.


$ daemon -U -i myforegroundprocess


export _TOP_PID=$$

# Handle to SIGTERM and SIGINT signals and forward them to the daemon process
_term()
{
    trap "exit 0" TERM
    kill -TERM "$pid"
    kill $_TOP_PID
}

_interrupt()
{
    kill -INT "$pid"
}

trap _term SIGTERM
trap _interrupt SIGINT

# Start process in the background as a daemon
${executable} "$@"

# Wait for the PID file to become available.
# Useful to work with daemons that don't behave well enough.
count=0

while [ ! -f "${_pidFile}" ]
do
    if [ $count -eq 10 ]
    then
        echo "It does not seem that there isn't any pid file! Giving up!"
        exit 1
    fi

    echo "Waiting for ${_pidFile} to become available..."
    sleep 1

    ((count=count++))
done

# Determine the daemon's PID by using the PID file
pid=$(cat ${_pidFile})

# Wait in the background for the PID to terminate
${if stdenv.isDarwin then ''
  lsof -p $pid +r 3 &>/dev/null &
'' else if stdenv.isLinux || stdenv.isCygwin then ''
  tail --pid=$pid -f /dev/null &
 '' else if stdenv.isBSD || stdenv.isSunOS then ''
   pwait $pid &
 '' else
   throw "Don't know how to wait for process completion on system: ${stdenv.system}"}

# Wait for the blocker process to complete.
# We use wait, so that bash can still
# handle the SIGTERM and SIGINT signals that may be sent to it by
# a process manager
blocker_pid=$!
wait $blocker_pid

In the beginning of the script we configure signal handlers for the TERM and INT signals so that the process manager can terminate the daemon process.
We must start the daemon and wait for it to become available. Although the parent process of a well-behaving daemon should only terminate when the initialization is done, this turns out not be a hard guarantee -- to make the process a bit more robust, we deliberately wait for the PID file to become available, before we attempt to wait for the termination of the daemon.
Then we wait for the PID to terminate. The bash shell has an internal wait command that can be used to wait for a background process to terminate, but this only works with processes in the same process group as the shell. Daemons are in a new session (with different process groups), so they cannot be monitored by the shell by using the wait command.

From this Stackoverflow article, I learned that we can use the tail command of GNU Coreutils, or lsof on macOS/Darwin, and pwait on BSDs and Solaris/SunOS to monitor processes in other process groups.
When a command is being executed by a shell script (e.g. in this particular case: tail, lsof or pwait), the shell script can no longer respond to signals until the command completes. To still allow the script to respond to signals while it is waiting for the daemon process to terminate, we must run the previous command in background mode, and we use the wait instruction to block the script. While a wait command is running, the shell can respond to signals.

State initialization

Process dependencies

Changing user privileges

Deploying process manager-agnostic configurations

With a processes Nix expression, we can define which process instances we want to run (and how they can be constructed from source code and their dependencies):


{ pkgs ? import  { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange processManager;
  };                                                                                                                                                                                               
in                                                                                                                                                                                                 
rec {                                                                                                                                                                                              
  webapp = rec {                                                                                                                                                                                   
    port = 5000;                                                                                                                                                                                   
    dnsName = "webapp.local";                                                                                                                                                                      
                                                                                                                                                                                                   
    pkg = constructors.webapp {                                                                                                                                                                    
      inherit port;                                                                                                                                                                                
    };                                                                                                                                                                                             
  };                                                                                                                                                                                               
                                                                                                                                                                                                   
  nginxReverseProxy = rec {
    port = 8080;

    pkg = constructors.nginxReverseProxy {
      webapps = [ webapp ];
      inherit port;
    } {};
  };
}

In the above Nix expression, we compose two running process instances:

webapp is a trivial web application process that will simply return a static HTML page by using the HTTP protocol.
nginxReverseProxy is a Nginx server configured as a reverse proxy server. It will forward incoming HTTP requests to the appropriate web application instance, based on the virtual host name. If a virtual host name is webapp.local, then Nginx forwards the request to the webapp instance.


$ nixproc-build --process-manager sysvinit \
  --state-dir /home/sander/var \
  --force-disable-user-change processes.nix


$ nixproc-build --process-manager systemd \
  --state-dir /home/sander/var \
  --force-disable-user-change processes.nix

The following command will automatically build and deploy all processes, using sysvinit as a process manager:


$ nixproc-sysvinit-switch --state-dir /home/sander/var \
  --force-disable-user-change processes.nix

We can also run a life-cycle management activity on all previously deployed processes. For example, to retrieve the statuses of all processes, we can run:


$ nixproc-sysvinit-runactivity status

We can also traverse the processes in reverse dependency order. This is particularly useful to reliably stop all processes, without breaking any process dependencies:


$ nixproc-sysvinit-runactivity -r stop

Similarly, there are command-line tools to use the other supported process managers. For example, to deploy systemd units instead of sysvinit scripts, you can run:


$ nixproc-systemd-switch processes.nix

Distributed process manager-agnostic deployment with Disnix


{ pkgs, distribution, invDistribution, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "sysvinit"
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange processManager;
  };

  processType =
    if processManager == "sysvinit" then "sysvinit-script"
    else if processManager == "systemd" then "systemd-unit"
    else if processManager == "supervisord" then "supervisord-program"
    else if processManager == "bsdrc" then "bsdrc-script"
    else if processManager == "cygrunsrv" then "cygrunsrv-service"
    else throw "Unknown process manager: ${processManager}";
in
rec {
  webapp = rec {
    name = "webapp";
    port = 5000;
    dnsName = "webapp.local";
    pkg = constructors.webapp {
      inherit port;
    };
    type = processType;
  };

  nginxReverseProxy = rec {
    name = "nginxReverseProxy";
    port = 8080;
    pkg = constructors.nginxReverseProxy {
      inherit port;
    };
    dependsOn = {
      inherit webapp;
    };
    type = processType;
  };
}


{ pkgs, distribution, invDistribution, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? null
}:

let
  constructors = import ./constructors.nix {
    inherit pkgs stateDir runtimeDir logDir tmpDir;
    inherit forceDisableUserChange processManager;
  };
in
rec {
  webapp = rec {
    name = "webapp";
    port = 5000;
    dnsName = "webapp.local";
    pkg = constructors.webapp {
      inherit port;
    };
    type = "managed-process";
  };

  nginxReverseProxy = rec {
    name = "nginxReverseProxy";
    port = 8080;
    pkg = constructors.nginxReverseProxy {
      inherit port;
    };
    dependsOn = {
      inherit webapp;
    };
    type = "managed-process";
  };
}


{
  test1 = {
    properties = {
      hostname = "test1";
    };
    containers = {
      managed-process = {
        processManager = "sysvinit";
      };
    };
  };

  test2 = {
    properties = {
      hostname = "test2";
    };
    containers = {
      managed-process = {
        processManager = "systemd";
      };
    };
  };
}


{infrastructure}:

{
  webapp = [ infrastructure.test1 ];
  nginxReverseProxy = [ infrastructure.test2 ];
}

and the deploy the system as follows:


$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Discussion

Target six different process managers on four different kinds of operating systems.
Implement various kinds of deployment scenarios: production deployments, test deployments as an unprivileged user.
Construct multiple instances of processes.

Limitations

The following table, summarizes the differences between the process manager solutions that I have investigated:

	sysvinit	bsdrc	supervisord	systemd	launchd	cygrunsrv
Process type	daemon	daemon	foreground	foreground daemon	foreground	foreground
Process control method	PID files	PID files	Process PID	cgroups	Process PID	Process PID
Scripting support	yes	yes	yes	yes	no	no
Process dependency management	Numeric ordering	Dependency-based	Numeric ordering	Dependency-based + dependency loading	None	Dependency-based + dependency loading
User changing capabilities	user	user	user and group	user and group	user and group	user
Unprivileged user deployments	yes*	yes*	yes	yes*	no	no
Operating system support	Linux	FreeBSD >OpenBSD NetBSD	Many UNIX-like: Linux macOS FreeBSD Solaris	Linux (+glibc) only	macOS (Darwin)	Windows (Cygwin)

Related work

Some remaining thoughts

Availability

Cachix: CDN and double storage size

support@cachix.org (Domen Kožar) — Wed, 29 Jan 2020 08:00:00 +0000

Cachix - Nix binary cache hosting, has grown quite a bit in recent months in terms of day to day usage and that was mostly noticable on bandwidth. +Over 3000 GB were served in December 2019. +CDN by CloudFlare Increased usage prompted a few backend machine instance upgrades to handle concurrent upload/downloads, but it became clear it’s time to abandon single machine infrastructure. +As of today, all binary caches are served by CloudFlare CDN.

Mayflower: __structuredAttrs in Nix

Mon, 20 Jan 2020 12:00:00 +0000

In Nix 2 a new parameter to the derivation primitive was added. It changes how information is passed to the derivation builder. +Current State In order to show how it changes the handling of parameters to derivation, the first example will show the current state with __structuredAttrs set to false and the stdenv.mkDerivation wrapper around derivation. All parameters are passed to the builder as environment variables, canonicalised by Nix in imitation of shell script conventions:

Hercules Labs: Hercules CI & Cachix split up

Tue, 14 Jan 2020 00:00:00 +0000

After careful consideration of how to balance between the two products, we’ve decided to split up. Each of the two products will be a separate entity:

-
To avoid any hangs in the future, we have configured systemd watchdog -which automatically restarts the service if the backend doesn’t respond for 3 seconds. -Doing so we released warp-systemd Haskell library to integrate Warp (Haskell web server) -with systemd, such as socket activation and watchdog features.
-
-
We’ve increased file descriptors limit to 8192.
-
-
We’ve set up Cachix status page so that you can check the state of the service.
-
-
For a better visibility into errors like file handles, we’ve configured sentry.io -error reporting. -Doing so we released katip-raven for seamless Sentry integration -of structured logging which we also use to log Warp (Haskell web server) exceptions.
-
-
Robert is now fully onboarded to be able to resolve any Cachix issues
-
-
We’ve made a number of improvements for the performance of Cachix. Just tuning GHC RTS settings -shows 15% speed up in common usage.
-
Hercules CI becomes part of Robert Hensing’s Ensius B.V.
Cachix becomes part of Domen Kožar’s Enlambda OÜ

Future work

For customers there will be no changes, except for the point of contact in support requests.

-
Enable debugging builds for production. This would allow systemd watchdog to send signal SIGQUIT and get an execution stack in which program hanged.
- -
We opened nixpkgs pull request to lay the ground work -to be able to compile debugging builds.
- -
However there’s a GHC bug opened showing debugging builds alter the performance of programs, so we need to asses our impact first.
-
-
Upgrade network library to 3.0 fixing unneeded file handle usage and a possible candidate for a deadlock.
- -
Stackage just included network-3.* in latest snapshot -so it’s a matter of weeks.
-
-
Improve load testing tooling to be able to reason about performance implications.
-

Domen & Robert

Mayflower: Windows-on-NixOS, part 1: Migrating bare-metal to a VM

Wed, 27 Nov 2019 06:00:00 +0000

This is part 1 of a series of blog posts explaining how we took an existing Windows installation on hardware and moved it into a VM running on top of NixOS. +Background We have a decently-equipped desktop PC sitting in our office, which is designated for data experiments using TensorFlow and such. During off-hours, it’s also used for games, and for that purpose it has Windows installed on it. We decided to try moving Windows into a VM within NixOS so that we could run both operating systems in parallel.

Craige McWhirter: Deploying and Configuring Vim on NixOS

Mon, 30 Sep 2019 00:00:00 +0000

Summary

I had a need to deploy vim and my particular preferred +configuration both system-wide and across multiple systems (via +NixOps).

We’re confident such issues shouldn’t affect the production anymore and since availability of -Cachix is our utmost priority, we are going to make sure to complete the rest of the work in a timely manner.

I started by creating a file named vim.nix that would be imported into either +/etc/nixos/configuration.nix or an appropriate NixOps Nix file. This example +is a stub that shows a number of common configuration items:

vim.nix:

What we do

with import <nixpkgs> {};
 
-Automated hosted infrastructure for Nix, reliable and reproducible developer tooling,
-to speed up adoption and lower integration cost. We offer
-Continuous Integration and Binary Caches.



Craige McWhirter: Setting Up Wireless Networking with NixOS
Thu, 26 Sep 2019 21:38:34 +0000

+vim_configurable.customize {
+  name = "vim";   # Specifies the vim binary name.
+  # Below you can specify what usually goes into `~/.vimrc`
+  vimrcConfig.customRC = ''
+    " Preferred global default settings:
+    set number                    " Enable line numbers by default
+    set background=dark           " Set the default background to dark or light
+    set smartindent               " Automatically insert extra level of indentation
+    set tabstop=4                 " Default tabstop
+    set shiftwidth=4              " Default indent spacing
+    set expandtab                 " Expand [TABS] to spaces
+    syntax enable                 " Enable syntax highlighting
+    colorscheme solarized         " Set the default colour scheme
+    set t_Co=256                  " use 265 colors in vim
+    set spell spelllang=en_au     " Default spell checking language
+    hi clear SpellBad             " Clear any unwanted default settings
+    hi SpellBad cterm=underline   " Set the spell checking highlight style
+    hi SpellBad ctermbg=NONE      " Set the spell checking highlight background
+    match ErrorMsg '\s\+$'        "
 
-The current NixOS Manual is a little sparse
-on details for different options to configure wireless
-networking. The
-version in
-master
-is a little better but still ambiguous. I've made a pull
-request to resolve
-this but in the interim, this documents how to configure a number of wireless
-scenarios with NixOS.
-
-If you're going to use NetworkManager, this is not for you. This is for those
-of us who want reproducible configurations.
-
-To enable a wireless connection with no spaces or special characters in the
-name that uses a pre-shared key, you first need to generate the raw PSK:
-
-$ wpa_passphrase exampleSSID abcd1234
-network={
-        ssid="exampleSSID"
-        #psk="abcd1234"
-        psk=46c25aa68ccb90945621c1f1adbe93683f884f5f31c6e2d524eb6b446642762d
-}
-
+    let g:airline_powerline_fonts = 1   " Use powerline fonts
+    let g:airline_theme='solarized'     " Set the airline theme
 
-Now you can add the following stanza to your configuration.nix to enable
-wireless networking and this specific wireless connection:
+    set laststatus=2   " Set up the status line so it's coloured and always on
 
-networking.wireless = {
-  enable = true;
-  userControlled.enable = true;
-  networks = {
-    exampleSSID = {
-      pskRaw = "46c25aa68ccb90945621c1f1adbe93683f884f5f31c6e2d524eb6b446642762d";
-    };
+    " Add more settings below
+  '';
+  # store your plugins in Vim packages
+  vimrcConfig.packages.myVimPackage = with pkgs.vimPlugins; {
+    start = [               # Plugins loaded on launch
+      airline               # Lean & mean status/tabline for vim that's light as air
+      solarized             # Solarized colours for Vim
+      vim-airline-themes    # Collection of themes for airlin
+      vim-nix               # Support for writing Nix expressions in vim
+    ];
+    # manually loadable by calling `:packadd $plugin-name`
+    # opt = [ phpCompletion elm-vim ];
+    # To automatically load a plugin when opening a filetype, add vimrc lines like:
+    # autocmd FileType php :packadd phpCompletion
   };
-};
+}
 
 
-If you had another WiFi connection that had spaces and/or special characters in the name, you would configure it like this:
+I then needed to import this file into my system packages stanza:
 
-networking.wireless = {
-  enable = true;
-  userControlled.enable = true;
-  networks = {
-    "example's SSID" = {
-      pskRaw = "46c25aa68ccb90945621c1f1adbe93683f884f5f31c6e2d524eb6b446642762d";
-    };
+  environment = {
+    systemPackages = with pkgs; [
+      someOtherPackages   # Normal package listing
+      (
+        import ./vim.nix
+      )
+    ];
   };
-};
 
 
-If you need to connect to a hidden network, you would do it like this:
+This will then install and configure Vim as you've defined it.
 
-networking.wireless = {
-  enable = true;
-  userControlled.enable = true;
-  networks = {
-    myHiddenSSID = {
-      hidden = true;
-      pskRaw = "46c25aa68ccb90945621c1f1adbe93683f884f5f31c6e2d524eb6b446642762d";
-    };
-  };
-};
+If you'd like to give this build a run in a non-production space, I've written vim_vm.nix with which you can build a VM, ssh into afterwards and test the Vim configuration:
+
+$ nix-build '<nixpkgs/nixos>' -A vm --arg configuration ./vim_vm.nix
+...
+$ export QEMU_OPTS="-m 4192"
+$ export QEMU_NET_OPTS="hostfwd=tcp::18080-:80,hostfwd=tcp::10022-:22"
+$ ./result/bin/run-vim-vm-vm
 
 
-The final scenario that I have, is connecting to open SSIDs that have some kind
-of secondary method (like a login in web page) for authentication of
-connections:
+Then, from a another terminal:
 
-networking.wireless = {
-  enable = true;
-  userControlled.enable = true;
-  networks = {
-    FreeWiFi = {};
-  };
-};
+$ ssh nixos@localhost -p 10022
 
 
-This is all fairly straight forward but was non-trivial to find the answers
-too.