Make list-packages command runnable in a Nix build environment

[thomashoneyman]

Summary

Some Spago commands cannot be run in a Nix build because of restrictions on the environment (namely, builds have restricted permissions and can mutate their temporary build directory, but can't write files outside it and can't access files outside it unless explicitly given a path to them, and network access is disallowed during the build). Some of these commands seem like they ought to be runnable in this environment but aren't.

Attempting to run a command like spago list-packages will fail, for example, with this error on Mac OS:

Failed to list packages:
spago: security: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

and this error on Linux:

spago:
Error: Remote host not found

URL: https://github.com/purescript/package-sets/releases/download/psc-0.13.6-20200123/packages.dhall

I am working on support for import from derivation for @justinwoo's spago2nix tool, but this has been a blocker.

I'd like to explore how to make at least the list-packages commands runnable in a pure build environment. The build command is already runnable in a pure environment given the --no-install flag. I don't believe any other commands will be necessary for import-from-derivation, but if there are then I can add them as comments on this issue.

Context

I use Spago with @justinwoo's excellent spago2nix tool. This tool takes a project's spago.dhall and packages.dhall files and generates the equivalent Nix expressions necessary to build each dependency.

Right now, the workflow of spago2nix is to:

1. Ensure you have Spago installed, a packages.dhall file, and a spago.dhall file.
2. Run spago2nix generate to generate a new spago-packages.nix file which describes how to build the dependencies
3. In the Nix expression which describes how to build your project, import the spago-packages.nix file to get all the sources
4. When describing the build steps, either use spago2nix build or spago build --no-install or call to the compiler directly with purs compile "src/**/*.purs" ${spagoPackages.compilePaths}

This has a key drawback: steps 2 and 3 really ought to be a single step. In other words, spago2nix should follow in the footsteps of yarn2nix and merge generating the Nix file and importing its result. This is called import from derivation and it carries some important benefits:

1. You don't have to check the generated Nix file in to source code
2. Developers don't have to remember to run spago2nix after installing a new dependency

Current issues

The list-packages command is not runnable in a Nix build environment, which is required in order to get import-from-derivation working with spago2nix without implementing a separate parser for the packages.dhall and spago.dhall files.

Next Steps

I am not exactly sure what Spago does in this command that is not simply a filesystem read and parsing these files; if it is possible to update it or provide a flag to disable some behavior (like yarn's --offline flag) then I'd like to explore how to make that possible.

My ideal result is that the reproduction below can be run without errors.

Reproduction

If you have Nix installed then you can reproduce a typical Nix build environment with a spago.dhall, packages.dhall and src/Main.purs with the below default.nix file and this command:

# In the same directory which contains the below default.nix file
nix-build

let
  # Imports the Nix package set released in September 2019
  pkgs = import (builtins.fetchTarball {
    name = "nixos-19.09";
    url = "https://github.com/nixos/nixpkgs-channels/archive/dae3575cee5b88de966d06b11861c602975cb23a.tar.gz";
    sha256 = "0zv948jsw313fmwz3rbwna92hijq8ji3dqlzhliwkvxl1adh8pbg";
  }) {};

  # Imports the most recent `easy-purescript-nix` tooling
  easy-purescript = import (pkgs.fetchFromGitHub {
    owner = "justinwoo";
    repo = "easy-purescript-nix";
    rev = "6cb5825430ab44719139f28b93d50c5810891366";
    sha256 = "1awsywpw92xr4jmkwfj2s89wih74iw4ppaifc97n9li4pyds56h4";
  }) { inherit pkgs; };

  # With the Nix package set and easy-purescript in scope, we can
  # pull out the few tools needed for the reproduction

  # Next, we'll write the bare minimum files necessary for a Spago project
  # into the Nix store. Then we can copy them into the build environment
  # for testing purposes.

  spagoDhall = pkgs.writeText "spago.dhall" ''
    { name = "test"
    , dependencies = [ "effect", "console", "psci-support" ]
    , packages = ./packages.dhall
    , sources = [ "src/**/*.purs" ]
    }
  '';

  packagesDhall = pkgs.writeText "packages.dhall" ''
    let upstream =
          https://github.com/purescript/package-sets/releases/download/psc-0.13.6-20200123/packages.dhall sha256:687bb9a2d38f2026a89772c47390d02939340b01e31aaa22de9247eadd64af05

    in  upstream
  '';

  sourceCode = pkgs.writeTextDir "src/Main.purs" ''
    module Main where

    import Prelude

    import Effect (Effect)
    import Effect.Console (log)

    main :: Effect Unit
    main = do
      log "🍝"
  '';

in

# This command will run the script below, and allows you to copy build
# artifacts to the $out directory to place them in the Nix store.
#
# In this case we can run various Spago or spago2nix commands to see
# if they work in the restricted environment or not.

pkgs.runCommand
  "test"
  { nativeBuildInputs = with easy-purescript; [ spago spago2nix purs ]; } ''

  # Setup: ensure the source code, spago.dhall file, and packages.dhall
  # file are in the current directory.
  cp -r ${sourceCode}/src .
  cat ${spagoDhall} > spago.dhall
  cat ${packagesDhall} > packages.dhall

  # Test: run various spago commands to ensure they succeed
  echo "Running spago list-packages"
  spago list-packages -f transitive -j > $out
  spago2nix generate # same as the above command
''

[thomashoneyman]

spago2nix needs to know the package name, GitHub URL, and revision (the Spago 'version' field), which can be listed as JSON with spago list-packages -f transitive -j.

In order for this command to work offline it seems like the package metadata for all packages listed in spago.dhall would have to be available locally. Fortunately, the upstream field is populated with both a URL and a sha256, which means that we can fetch it in Nix no problem.

Given a packages.dhall file like:

let upstream =
    https://github.com/purescript/package-sets/releases/download/psc-0.13.6-20200123/packages.dhall sha256:687bb9a2d38f2026a89772c47390d02939340b01e31aaa22de9247eadd64af05

in upstream

then we can re-use the URL and sha256 in Nix:

# The contents of `fetchedPackages` are the contents of the `packages.dhall` file
fetchedPackages = builtins.fetchurl {
  url = "https://github.com/purescript/package-sets/releases/download/psc-0.13.6-20200123/packages.dhall";

  # The sha256 produced by Spago in `spago init`
  sha256 = "687bb9a2d38f2026a89772c47390d02939340b01e31aaa22de9247eadd64af05"

  # The sha256 that Nix actually expects; I'm not sure how to resolve the difference
  sha256 = "1zzccm2qmbg673h6rx1lyznl78j97z9cx9v2vyn5bbj8pyjsn47f";
};

This would allow us to copy an upstream packages.dhall file locally as part of the build, which Spago could then use offline.

buildPhase = ''
  cat ${fetchedPackages} > upstream.dhall
  ...
'';

However, I understand that this could be a brittle solution as I'm not familiar with all the different ways a packages.dhall file could be configured (I'm only used to the upstream // additions // overrides format, but I'm aware you could have several upstreams and merge them together, for example).

It may not be possible to get the package metadata information that spago2nix would require without significant changes to Spago. In that case I'm happy to close this issue until I have time to explore the code base and put together a serious proposal towards that goal.

[cdepillabout]

The sha256 that Nix actually expects; I'm not sure how to resolve the difference

I'm not sure if this is the error you're seeing, but the nix tools generally use a base-32 hash, not a base-16 hash.

Here's a couple links explaining the situation a little (although nothing super informative):

• sha256 - Nothing makes sense NixOS/nix#806
• How to manually replicate/reproduce/obtain the sha256 hash specified in Nix with fetchgit or fetchFromGitHub? NixOS/nixpkgs#191128
• https://discourse.nixos.org/t/how-to-calculate-the-sha256-parameter-of-the-pullimage-function/250

(Sorry if you already knew this and the problem you had is actually something else completely...)

[f-f]

@thomashoneyman one of the reasons why we have the hashes on the package set import in packages.dhall is to avoid refetching it every time we Dhall-evaluate that file.
The reason why we don't have to refetch it is that the first time we evaluate the packages.dhall file then the file at that URL is fetched, hashed, and if the hash matches with the one in the file it gets stored in the Dhall cache.

I.e. if you run this with an empty cache:

dhall <<< 'https://github.com/purescript/package-sets/releases/download/psc-0.13.6-20200123/packages.dhall sha256:687bb9a2d38f2026a89772c47390d02939340b01e31aaa22de9247eadd64af05'

..then the result will be cached (and reused the next time you evaluate that) in this file: ~/.cache/dhall/1220687bb9a2d38f2026a89772c47390d02939340b01e31aaa22de9247eadd64af05
(note: you can affect the location by changing the value of the XDG_CACHE_HOME envvar)

Does this help? I guess you could resolve the file once, and then copy the cache over in every derivation you need that.
Another option is to render the config and just use that. I.e. you run dhall <<< './spago.dhall' > spago-rendered.dhall, and then run all the spago commands with -x spago-rendered.dhall. Makes sense?

[f-f]

Re having an --offline flag: yes, this has been on my radar for a while (I had to use Spago while offline some time ago, and it was terrible), but I didn't have time to look at it yet, so everything is welcome on this side of things 🙂

[justinwoo]

I think there's information missing in here that the yarn2nix uses the yarn.lock file checked in as a source of information, since a package definition in yarn.lock looks like this:

"@babel/cli@^7.1.5":
  version "7.8.3"
  resolved "https://registry.yarnpkg.com/@babel/cli/-/cli-7.8.3.tgz#121beb7c273e0521eb2feeb3883a2b7435d12328"
  integrity sha512-K2UXPZCKMv7KwWy9Bl4sa6+jTNP7JyDiHKzoOiUUygaEDbC60vaargZDnO9oFMvlq8pIKOOyUUgeMYrsaN9djA==
  dependencies:
    commander "^4.0.1"
    convert-source-map "^1.1.0"
    fs-readdir-recursive "^1.1.0"
    glob "^7.0.0"
    lodash "^4.17.13"
    make-dir "^2.1.0"
    slash "^2.0.0"
    source-map "^0.5.0"
  optionalDependencies:
    chokidar "^2.1.8"

So this then allows for generating nix expressions at runtime from this file using the "integrity" field. Otherwise yarn2nix would not work, unless the derivation were jailbroken to allow for network requests.

[justinwoo]

"Could not resolve host" just comes from the derivation environment being sandboxed. Given default.nix:

{ pkgs ? import <nixpkgs> {} }:

pkgs.runCommand "fail" {
  buildInputs = [ pkgs.curl ];
} ''
  >$out curl https://google.com
''

You will get the following when you try to build it:

$ nix-build
these derivations will be built:
  /nix/store/a3ywj8a0rzzp7gwdka99hl2b57775ijv-fail.drv
building '/nix/store/a3ywj8a0rzzp7gwdka99hl2b57775ijv-fail.drv'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: google.com
builder for '/nix/store/a3ywj8a0rzzp7gwdka99hl2b57775ijv-fail.drv' failed with exit code 6
error: build of '/nix/store/a3ywj8a0rzzp7gwdka99hl2b57775ijv-fail.drv' failed

curl: (6) Could not resolve host: google.com

[thomashoneyman]

@justinwoo I was just looking at that, and that’s a good point; since we don’t record the sha256 for the packages themselves I would still have to make network requests to calculate those (nix-prefetch-git). So unless the package sets themselves included a hash for each package to verify its contents I don’t believe this is possible.

@f-f Thanks for explaining the cache, that’s helpful and does make sense.

@cdepillabout Yes, I think you’ve pointed out the problem wrt the hashes. I would expect there’s not a way to convert from base-16 to base-32, so that may not be possible to solve here.

[justinwoo]

Sorry, I will unsubscribe from this thread now. Hope you figure this out, or please have someone from your company contact me. Thanks.

[thomashoneyman]

While I still think it would be nice to have a way to list packages -- with their sha256 hashes -- offline, this issue is requesting a specific piece of functionality which would require quite a bit more thought to explore and implement properly with consequences beyond the narrow use case I've described here. With that in mind, I'm going to go ahead and close this, though we can continue chatting here as needed. Thanks everyone for the feedback!

[f-f]

You're welcome 🙂

[jamesdbrock]

Thanks for your workflow writeup @thomashoneyman , I copied it onto the spago2nix README https://github.com/justinwoo/spago2nix#workflow