Unlink error when building Oni

[johnchildren]

Re: NixOS/nixpkgs#38231

I attempted to build Oni using yarn2nix, but with this expression:

{ yarn2nix, fetchFromGitHub }:
with yarn2nix; mkYarnPackage rec {
  name = "oni-vim-${version}";
  version = "v0.3.1";
  src = fetchFromGitHub {
    owner = "onivim";
    repo = "oni";
    rev = "adf404a32c585b498b57d9f52fabeaef635e087d";
    sha256 = "1h4xlvm2h6lln2w2f5flxycf3175w9cb19f2g5v3d7ksnzm9x2wp";
  };
}

I had this error

builder for '/nix/store/lhi77k72zi84415gcl4iz81ngimjlwdw-oni-modules-0.3.1.drv' failed with exit code 1; last 10 log lines:
  building
  yarn config v1.5.1
  success Set "yarn-offline-mirror" to "/nix/store/1gvl75025qh6608fnq9r7q7iz13s136c-offline".
  Done in 0.10s.
  yarn install v1.5.1
  [1/4] Resolving packages...
  [2/4] Fetching packages...
  error An unexpected error occurred: "EACCES: permission denied, unlink '/nix/store/1gvl75025qh6608fnq9r7q7iz13s136c-offline/shelljs-0.7.7.tgz'".
  info If you think this is a bug, please open a bug report with the information provided in "/tmp/nix-build-oni-modules-0.3.1.drv-0/yarn-error.log".
  info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
cannot build derivation '/nix/store/jaqwiq29ba2zv8yyh9ir63xsjwiqq3gi-oni-vim-v0.3.1.drv': 1 dependencies couldn't be built
[349 built (1 failed), 70 copied (6.6 MiB), 2.9 MiB DL]

I don't really know enough about yarn to know the root cause of this, but I suspect it is something to do with the unlink command attempting to interfere with nixpkgs symlinks? Will attempt to investigate further later in the week.

[wizzup]

I've got similar EACCES error when try to build https://github.com/jmerle/cg-local-ext

[wizzup@ cg-local-ext](master)$ cat default.nix 
with (import <nixpkgs> {});

rec {
  cg-local-ext = mkYarnPackage {
    name = "cg-local-ext";
    src = ./.;
    packageJson = ./package.json;
    yarnLock = ./yarn.lock;
    # NOTE: this is optional and generated dynamically if omitted
    yarnNix = ./yarn.nix;
  };
}

[wizzup@ cg-local-ext](master)$ nix-build                                                                                                                                                             
these derivations will be built:                                                                                                                                                                      
  /nix/store/q99j1pfiim8iwr4a38sk0cdll0jimhpx-cg-local-ext-modules-1.0.1.drv                                                                                                                          
  /nix/store/bc55psm4g94f2l7hlwcrlxg184dqi37r-cg-local-ext.drv
building '/nix/store/q99j1pfiim8iwr4a38sk0cdll0jimhpx-cg-local-ext-modules-1.0.1.drv'...                                                                                                              
configuring                                                                                                                                                                                           
building                                                                                                                                                                                              
yarn config v1.5.1                                                                                                                                                                                    
success Set "yarn-offline-mirror" to "/nix/store/d1rkhaqawfyk2w5alc3j98ff7i1drymf-offline".                                                                                                           
Done in 0.16s.                                                                                                                                                                                        
yarn install v1.5.1                                                                                                                                                                                   
[1/4] Resolving packages...                                                                                                                                                                           
[2/4] Fetching packages...                                                                                                                                                                            
error An unexpected error occurred: "EACCES: permission denied, unlink '/nix/store/d1rkhaqawfyk2w5alc3j98ff7i1drymf-offline/strip-bom-3.0.0.tgz'".                                                    
info If you think this is a bug, please open a bug report with the information provided in "/tmp/nix-build-cg-local-ext-modules-1.0.1.drv-0/yarn-error.log".                                          
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.                                                                                                              
builder for '/nix/store/q99j1pfiim8iwr4a38sk0cdll0jimhpx-cg-local-ext-modules-1.0.1.drv' failed with exit code 1                                                                                      
cannot build derivation '/nix/store/bc55psm4g94f2l7hlwcrlxg184dqi37r-cg-local-ext.drv': 1 dependencies couldn't be built
error: build of '/nix/store/bc55psm4g94f2l7hlwcrlxg184dqi37r-cg-local-ext.drv' failed           

while there is no no problem when running yarn

[wizzup@ cg-local-ext](master)$ nix-shell -p yarn yarn2nix nodejs-9_x                                                                                                                                 
                                                                                                                                                                                                      
[nix-shell:/data/works/coding_challenges/codingame/extensions/cg-local/cg-local-ext]$ yarn                                                                                                            
yarn install v1.5.1                                                                                                                                                                                   
[1/4] Resolving packages...                                                                                                                                                                           
success Already up-to-date.                                                                                                                                                                           
Done in 1.30s.  

[pmiddend]

I also encounter this error: "EACCES: permission denied, unlink '/nix/store/vsy3ykxz5xaimxifkz3w9n2c4ynfr4ki-offline/serve-static-1.13.2.tgz'"

[domenkozar]

Same for heroku in NixOS/nixpkgs#42869

[domenkozar]

I've debugged this for an hour and couldn't find a way to prevent yarn from unlinking. I think the next step should be to figure out what line actually does unlinking.

Unfortunately error we get from node includes no traceback, so I've opened yarnpkg/yarn#6058

[phryneas]

I have a similar error and maybe a clue.

First, what I did:

• start a build, let it fail

cd /tmp/nix-build-code-oss-dev-modules-1.27.2.drv-6
sudo -s -u nixbld1
source ./env-vars
node --inspect-brk  $(which yarn) install --offline --frozen-lockfile --ignore-engines --ignore-scripts

• start chrome, go to chrome://inspect/
• enable "pause on caught exceptions", start stepping with F8 until I encountered something fishy in TarballFetcher.prototype.createExtractor

What I found:

A mismatching hash. The yarn.lock contains resolved "minimist-1.2.0.tgz#a35008b20f41383eec1fb914f4cd5df79a264284", but:

% sha1sum /nix/store/i22zcaqriz4wr5xbfq4i138lhywrmqsi-offline/minimist-1.2.0.tgz 
69a23a3ad29caf0097f06eda59b361ee2f0639f6  /nix/store/i22zcaqriz4wr5xbfq4i138lhywrmqsi-offline/minimist-1.2.0.tgz

what I guess is happening after that:

After the TarballFetcher fails, https://github.com/yarnpkg/yarn/blob/master/src/package-fetcher.js#L47 kicks in, trying to delete the .tgz with the wrong hash.

So now I know (at least in my case) how this error is triggered, but I'm not much wiser on how to react in the case of a wrong hash.

Make yarn2nix validate all hashes and throw an exception earlier in the build? Add an option to yarn2nix to force-override all hashes?

[phryneas]

Oh, for reproductions sake, here's a derivation with problems:

{ mkYarnPackage, ...}: 
mkYarnPackage rec {
  name = "vscode";
  version = "1.27.2";
  src = builtins.fetchTarball { 
    url = "https://github.com/Microsoft/vscode/archive/${version}.tar.gz" ;
    sha256 = "0b42c2xa2cpmhxmazpcwgxay2n4jmv74rwxqizh38cg8390jpvp2";
  };
}

[phryneas]

Okay, one more weird thing: when I download "https://registry.yarnpkg.com/minimist/-/minimist-1.2.0.tgz" by hand, it has the expected sha1sum of a35008b20f41383eec1fb914f4cd5df79a264284

This might actually be an error on the side of yarn2nix, not a corrupted yarn.lock

[phryneas]

Yup, it's with yarn2nix. It mixes up those two:

minimist@1.2.0, minimist@^1.1.0, minimist@^1.1.3, minimist@^1.2.0:
  version "1.2.0"
  resolved "https://registry.yarnpkg.com/minimist/-/minimist-1.2.0.tgz#a35008b20f41383eec1fb914f4cd5df79a264284"


"@types/minimist@1.2.0":
  version "1.2.0"
  resolved "https://registry.yarnpkg.com/@types/minimist/-/minimist-1.2.0.tgz#69a23a3ad29caf0097f06eda59b361ee2f0639f6"

[domenkozar]

This issue is fixed in master and should be closed, but yarn2nix in nixpkgs is very outdated: NixOS/nixpkgs#20637 (comment)