Proposal for using the Nix build system

[therishidesai]

Transitioning to Nix

The following is a proposal on why we should move to Nix, how we should move to Nix, and what are some of the unknowns if we move to Nix.

Why Nix?

Nix is a package management and build system that allows for reproducible builds. For ILLIXR, we have code from many different areas that have a wide variety of dependencies. Currently, we apt-install or build from source all of the dependencies on to the users host machine. This makes a lot of assumptions as to what their underlying system has and there is a chance we may ruin the user's apt repositories and other projects with the compiler changes that we make. Finally, this install process isn't reliably portable to other systems and as we want to move to ARM systems in the future this will be important. Nix solves almost all of these issues because of its design. When you build a package in Nix it is an entirely indpenedent build so it doesn't pollute the rest of the system. In addition, Nix has clean cross compilation support which allow us to easily port to ARM. Nix also has a smart cacheing system that will speed up many of our builds. Finally, we will be able to use Nix to move away from git submodules which continue to cause us a lot of issues.

How we use Nix?

The first step in moving to Nix is writting a Nix derivation (Nix script with build instructions) for each one of our components. For components like OpenVINS this will be fairly easy since they are standard C++ project. We also need to write a Nix derivation for our ILLIXR common files. The first challenge that comes with Nix is how to use OpenGL and CUDA on the users machine. This becomes a challenge because of the machine specifics with Nvidia and mesa drivers. The solution in the Nix ecosystem is to use something called nixGL. What nixGL does is find the Nvidia/mesa driver that is installed locally on the user's machine and pulls the same driver in to the Nix environment. The one challenge with nixGL is the user needs to have the most up to date Nvidia driver since Nvidia has a habit of removing the older driver repositories.
The final thing with Nix is how we transition the repository setup. We will remove all of the submodules in our main ILLIXR root repository. Then we will have the root repository contain all of the nix derivations for each component and a src.nix file that points to the git repo for the individual component and information on which branch/commit we want to build from:

ILLIXR
  -- components
     -- open_vins
        -- default.nix (Nix build derivation)
        -- src.nix

With this repository setup, all of the code changes will be in their individual repos and then the master repo will have a PR where a contributor just bumps the commit version in the src.nix file.

Unkowns with Nix

The biggest unknown is how Monado will work with the nixGL setup. I can confidently say we will have our whole setup working with gldemo but Monado and godot are two pieces that may not work.

[charmoniumQ]

Should common/ be a header-only nix-package?

[charmoniumQ]

There are several header-only libraries we use, such as concurrentqueue and sqlite3pp. I assume we should make these nix packages to. Does nix handle header-only packages well?

[therishidesai]

There are several header-only libraries we use, such as concurrentqueue and sqlite3pp. I assume we should make these nix packages to. Does nix handle header-only packages well?

Yes, it is pretty easy to handle header only packages in nix. sqlite3 is already packaged in nix upstream so we can use that. Then we can make our own concurrentqueue package and common package.

[therishidesai]

Closing this dicussion as have decided to not use Nix due to complexities with OpenGL, Vulkan, Nvidia and Monado.

[charmoniumQ]

:'(

[e3m3]

Open questions:

• What mature alternatives to Nix are there?
• Does Monado build on Nix / NixOS?
• What encapsulation or virtualization happens using just nix-shell?
• What is the overhead of qemu?
• What do we need to passthrough if using qemu?

[astro-friedel]

The upcoming CMake build system changes make this discussion moot.