callCabal2nix doesn't seem to work with recent Hydra / restricted eval mode

[dalaing]

Issue description

We updated our NixOS machine that runs Hydra recently, and all of our jobs that use callCabal2nix started failing with an error about accessing store paths in restricted mode.

Steps to reproduce

As far as I can tell: use callCabal2nix in a job that is being run by Hydra.

I asked about this on IRC and it seems to be a known problem with the intersection of callCabal2nix and Hydra, although I can't find an open issue about it at the moment.

Technical details

The error looks like this in the logs:

hydra-eval-jobs returned exit code 1:
error: access to path '/nix/store/lzb72kqip2q3g5dnx32m0sfpcz1wzd8j-cabal2nix-hedgehog' is forbidden in restricted mode
(use '--show-trace' to show detailed location information)

and turns up as an evaluation error on the jobsets page.

[ElvishJerricco]

Just got hit by this. Is there a workaround for this anywhere?

[dalaing]

I have something dodgy as a workaround the time being:

let
  nixpkgs-hydra = pkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "3e96cee147d1ab9ee790f5b47871472a48e71a8d";
    sha256 = "14j46l2cfp4ckvny459fq7ndnkj2rl0w6vv27kzz9k3rfqr05jhr";
  };
  old = import nixpkgs-hydra {};
in {
...
  services.hydra = {
    ...
    package = old.hydra;
    ...
  };
...

[ElvishJerricco]

@dalaing Gross. Thanks. Do you know which commit broke hydra here?

[dalaing]

I don't know which commit was involved. The versions of hydra and nix-unstable in the tree are closely intertwined, so I think having a whole separate checkout of nixpkgs is the easy way forward, rather than trying to roll back hydra, nix-unstable, and all of the various things they may have been depending on any given point in time. It looks like a tough problem - I'm pretty happy I've got the option to grab a local version of nixpkgs for this issue and move forwards.

My rough understanding is that callCabal2nix uses import-from-derivation, and apparently that is bad, and the new restricted eval mode rules it out. I'm not sure if it really counts as a hydra breakage instead of perhaps indicating the need for a new way of doing callCabal2nix.

[ElvishJerricco]

@dalaing I think there is inherently no way to do callCabal2nix other than IFD. At least not without a major new Nix feature.

I think there just needs to be a way to manually turn off this restriction in Hydra.

[shlevy]

This should work in 2.0... NixOS/nix@43f8ef7

[ElvishJerricco]

@shlevy Hm... I tried @dalaing's workaround but with this checkout instead, and I'm still seeing the same error.

[shlevy]

@ElvishJerricco What's your nix version on the hydra evaluator?

[ElvishJerricco]

@shlevy The machine is on 1.11.16 (NixOS 17.09). But I thought having Hydra built with Nix 2.0 would be enough. I guess that's not the case?

[shlevy]

It's whatever nix your hydra-evaluator is linked to.

[shlevy]

Note that it has to actually be 2.0, I don't think any of the pre-releases have that fix.

[ElvishJerricco]

@shlevy My hydra UI says: Hydra 2017-11-21 (using nix-2.0pre5968_a6c0b773).

So I think that's not the official release. But it does appear to have the commit you referenced as an ancestor:

$ git clone git@github.com:NixOS/nix
$ cd nix
$ (git merge-base --is-ancestor 43f8ef73 a6c0b773 && echo yes) || echo no
yes

So it should have that commit.

[ElvishJerricco]

I suppose my nix-daemon is still 1.11.16... That would probably break it, right?

[shlevy]

No, restricted mode is a function of the evaluator. Can you make a minimal repro?

[shlevy]

Ah never mind I'll try it tomorrow

[ElvishJerricco]

Ok, can now confirm that Hydra 2017-11-21 (using nix-2.0) (and nix-daemon 2.0 for good measure) also does exhibit this issue. I've bumped to the latest nixos-17.09, set nix.package to the new pkgs.nixStable2, and set services.hydra.package = pkgs.hydra.override { nixUnstable = pkgs.nixStable2; };.

However, in my search for a minimal repro, I discovered that IFD works fine. Having Hydra build this works:

{ nixpkgs }:

let pkgs = import nixpkgs {};
in {
  foo = import (pkgs.runCommand "default.nix" {} ''
    echo 'runCommand: runCommand "foo" {} "touch $out"' > $out
  '') pkgs.runCommand;
}

But this does not:

{ nixpkgs }:

let pkgs = import nixpkgs {};
in {
  foo = pkgs.haskellPackages.callCabal2nix "mtl" (pkgs.fetchFromGitHub {
    owner = "haskell";
    repo = "mtl";
    rev = "c7d396732bd45e409478bd4df1d0ca95d6f39356";
    sha256 = "1ama43hhhlnsil9c6vf6lm554cm0q2z1jr93r502mclzh9swhih0";
  }) {};
}

So IFD works fine I think. It's something else about callCabal2nix.

---

To reproduce the Hydra server in a VM, clone simple-hydra and use nixos-rebuild build-vm, with the latest nixos-17.09 for nixpkgs:

# Use this with `nixos-rebuild build-vm` to build a VM for testing Hydra.
#
#   $ nixos-rebuild -I nixos-config=`pwd`/example.nix build-vm
#   $ ./result/bin/run-nixos-vm

{ pkgs, config, ... }:

{
  imports = [./simple-hydra <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix>];

  users.users.root.initialPassword = "test";

  virtualisation = {
    graphics = false;
    memorySize = 8000; # M
    diskSize = 50000; # M
    writableStoreUseTmpfs = false;
  };

  simple-hydra.enable = true;
  simple-hydra.hostName = "hydra.example.org";
  simple-hydra.useNginx = false;

  services.hydra.package = pkgs.hydra.override { nixUnstable = pkgs.nixStable2; };

  networking.firewall.allowedTCPPorts = [ 3000 ];
}

[shlevy]

@ElvishJerricco Can you test this patch against nixpkgs? http://ix.io/QAO Let me know if it works and I'll make a complete fix in nixpkgs.

[ElvishJerricco]

@shlevy Where am I supposed to apply that? It doesn't seem to apply to nixos-17.09, nixos-unstable, or master. EDIT: Whoops, that was my fault. Patch applies just fine.

[ElvishJerricco]

@shlevy That seems to have worked! Though it requires pkgs.hydra.override { nixUnstable = pkgs.nixStable2; };, even when nixUnstable is nix-2.0pre5889_c287d731.

[dalaing]

Thanks heaps for that @shlevy!