fetch-bower fails fetching components when run under nix-build

[abuibrahim]

Issue description

fetch-bower produces truncated nix-store entry when run under nix-build:

$ nix-hash --base32 --type sha256 /nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0
1kgli0afc7zvnny3xk2hjp19zn4brs7manjn0xq25ndjnngb0fcv

Looking into the retrieved jquery in the nix store you can see jquery is actually missing:

$ find /nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0/packages
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0/registry
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0/registry/bower.herokuapp.com
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0/registry/bower.herokuapp.com/search
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0/registry/bower.herokuapp.com/lookup
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0/registry/bower.herokuapp.com/lookup/jquery_d223e
/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0/registry/bower.herokuapp.com/list

It works fine when trying to manually run it as a regular user:

$ fetch-bower -q --out out jquery '*' '3.1.0' 
$ nix-hash --base32 --type sha256 out
1rh20xgppsm0vz3w1fgxqghdjlplnyw7hj0ak4i81df4xsdh1dka

Steps to reproduce

bower.json:

{
  "name": "test",
  "version": "0.0.0",
  "dependencies": {
    "jquery": "*"
  }
}

bower-packages.nix generated by bower2nix bower.json bower-packages.nix:

{ fetchbower, buildEnv }:
buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [
  (fetchbower "jquery" "3.1.0" "*" "1rh20xgppsm0vz3w1fgxqghdjlplnyw7hj0ak4i81df4xsdh1dka")
]; }

default.nix:

{ myApp ? { outPath = ./.; name = "MyApp"; }
, pkgs ? import <nixpkgs> {}
}:

pkgs.stdenv.mkDerivation {
  name = "MyApp";
  src = myApp;

  bowerComponents = pkgs.buildBowerComponents {
    name = "MyApp";
    generated = ./bower-packages.nix;
    src = myApp;
  };
}

run nix-build

these derivations will be built:
  /nix/store/arzazlvdgj62d4z96hjqi0fgci3nyr25-jquery-3.1.0.drv
  /nix/store/f1g8216xi09fm1abvxkaza3s89111l8z-bower-env.drv
  /nix/store/59araajhcziv0binyp3xqh3jc8vxnvn8-bower_components-MyApp.drv
  /nix/store/mm6hwjpwd6b3g60c7jqcrfjh1cwk2mz3-MyApp.drv
these paths will be fetched (1.94 MiB download, 15.36 MiB unpacked):
  /nix/store/ymwnm3jjm4hdv91qa2b5cqa2xrrpbp6y-node-bower-1.7.9
fetching path ‘/nix/store/ymwnm3jjm4hdv91qa2b5cqa2xrrpbp6y-node-bower-1.7.9’...
building path(s) ‘/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0’

*** Downloading ‘https://cache.nixos.org/nar/1lwnwkvxdd0wc26d580p9jkjc3jz5lhszfwv9clgfpb3w9w5w49z.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/ymwnm3jjm4hdv91qa2b5cqa2xrrpbp6y-node-bower-1.7.9’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 1983k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0output path ‘/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0’ has r:sha256 hash ‘1kgli0afc7zvnny3xk2hjp19zn4brs7manjn0xq25ndjnngb0fcv’ when ‘1rh20xgppsm0vz3w1fgxqghdjlplnyw7hj0ak4i81df4xsdh1dka’ was expected
cannot build derivation ‘/nix/store/f1g8216xi09fm1abvxkaza3s89111l8z-bower-env.drv’: 1 dependencies couldn't be built
killing process 3665
cannot build derivation ‘/nix/store/59araajhcziv0binyp3xqh3jc8vxnvn8-bower_components-MyApp.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/mm6hwjpwd6b3g60c7jqcrfjh1cwk2mz3-MyApp.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/mm6hwjpwd6b3g60c7jqcrfjh1cwk2mz3-MyApp.drv’ failed

Technical details

• System: (16.09beta172.e3bdf6b (Flounder))
• Nix version: (nix-env (Nix) 1.11.4)
• Nixpkgs version: (16.09beta172.e3bdf6b)

[abuibrahim]

ping @rvl @shlevy

[rvl]

Sorry for the delay @abuibrahim, I will test and merge today.

[rvl]

Hello @abuibrahim, thanks for your detailed report. I am getting slightly different results to you, so more investigation will be necessary.

$ nix-build
these derivations will be built:
  /nix/store/7jxmpnqgaz1xczd57pq6pqxv4jzjyisw-jquery-3.1.0.drv
  /nix/store/2x2lbxc196d4cac6gp2sj3a531y458si-bower-env.drv
  /nix/store/5n2fb24pxik0mg6rcx7wjxc7f9j3313j-bower_components-MyApp.drv
  /nix/store/j4i5ikjyhzmmxvjlwlzka8nq6hmd2pw5-MyApp.drv
building path(s) ‘/nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0’
building path(s) ‘/nix/store/cx7iz6prix1wzcg8y85n91j6q87q13pq-bower-env’
created 2 symlinks in user environment
building path(s) ‘/nix/store/ir1zjcxdp9cy71mr888kgiyv2pwx71dz-bower_components-MyApp’
bower jquery#*                  cached https://github.com/jquery/jquery-dist.git#3.1.0
bower jquery#*                 install jquery#3.1.0

jquery#3.1.0 bower_components/jquery
building path(s) ‘/nix/store/yqjx91myzmmr5vrsk5k6wxm525np3gvw-MyApp’

jquery#3.1.0 bower_components/jquery
building path(s) ‘/nix/store/yqjx91myzmmr5vrsk5k6wxm525np3gvw-MyApp’
unpacking sources
unpacking source archive /nix/store/7xhlr8iizliwjdh0sp8rnhzyvi7nc6n7-issue-18454
source root is issue-18454
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
install flags: install    
make: *** No rule to make target 'install'.  Stop.
builder for ‘/nix/store/j4i5ikjyhzmmxvjlwlzka8nq6hmd2pw5-MyApp.drv’ failed with exit code 2
error: build of ‘/nix/store/j4i5ikjyhzmmxvjlwlzka8nq6hmd2pw5-MyApp.drv’ failed

In my environment (NixOS 16.09 channel), there was no hash mismatch.

$ nix-hash --base32 --type sha256 /nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0
1rh20xgppsm0vz3w1fgxqghdjlplnyw7hj0ak4i81df4xsdh1dka

Possibly something caused git to fail to download the jquery package.

If it silently fails producing an incorrect result, then that is a bug in bower2nix which could be reported at https://github.com/rvl/bower2nix/issues.

[abuibrahim]

I am not sure this due to bower2nix. Fetch-bower is calling git and should handle the error's accordingly, no?

[abuibrahim]

It consistently fails for me both on the latest unstable and 16.09 channels. Can you try updating your channels?

My suspect was commit 457eddd but I wasn't able to revert it cleanly to try it out.

[rvl]

I am not sure this due to bower2nix. Fetch-bower is calling git and should handle the error's accordingly, no?

By bower2nix, I also mean fetch-bower. I have rolled the two programs into one repo.

You are definitely right, fetch-bower is calling git (indirectly, through the bower API), and should handle the error.

It consistently fails for me both on the latest unstable and 16.09 channels. Can you try updating your channels?

I'm an avid subscriber to 16.09 and especially nixos-unstable, so have tested on those.

Obviously it would be easier to fix the error handling if I can make fetch-bower/git fail myself. Do you have any further info about your setup which might be relevant? E.g. any proxy settings required for git to work. Can I also confirm that you are running nix-store --delete /nix/store/jfhkyajsjmbjdrc7wyrikx8ayr94ald7-jquery-3.1.0 before retrying nix-build?

[abuibrahim]

I don't have any proxy settings. Is it possible GitHub is rate limiting requests from bower? I found this thread interesting componentjs/component#546

And yes I am running nix-store --delete before trying nix-build.

[abuibrahim]

Hi @rvl, are there any additional details I can provide? I am not that familiar with typescript or javascript, but I could try to narrow the issue down with some help.

[rvl]

How to reproduce:

nix-shell --pure -p nodePackages.bower2nix --command "fetch-bower jquery"     

Expected output: bower package info json.
Actual output: nothing, not even an error message.

[edofic]

This problem is back. I can reproduce it by following the steps outlined in the top message. (latest nixpkgs-unstable)

It did work for me a while ago though, so it looks like a regression. Did some digging around but I cannot figure it out.

cc @shlevy @rvl

[domenkozar]

Good catch, I think I've seen the same issue as well, but didn't bother (lack of time). Could be a different cause, but we have to start somewhere :)

[edofic]

Running

nix-shell --pure -p nodePackages.bower2nix --command "fetch-bower -o out jquery"

produces correct result while nix-build-ing the following

let

  pkgs = import <nixpkgs> {};

in

  pkgs.stdenv.mkDerivation {
    name = "jq";
    buildInputs = with pkgs; [ nodePackages.bower2nix ];
    buildCommand = ''
      fetch-bower -o $out jquery
    '';
  }

produces an output without jquery (just meta data)

[edofic]

Figured it out. Git doesn't see SSL certificates. I made it run with this expression

let

  pkgs = import <nixpkgs> {};

in

  pkgs.stdenv.mkDerivation {
    name = "jq";
    src = ./.;
    buildInputs = with pkgs; [ nodePackages.bower2nix ];
    buildCommand = ''
      export SSL_CERT_FILE=${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt
      fetch-bower --out $out jquery
    '';
  }

I'll try to open a PR to fix this in nixpkgs. I'll also try to open a PR to bower2nix because there is a bug in it - missing SSL cert should not result in exit code 0.