New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"ar: command not found" in cross-compiling glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf #33023

Closed
telent opened this Issue Dec 24, 2017 · 8 comments

Comments

3 participants
@telent
Contributor

telent commented Dec 24, 2017

Issue description

When cross-building anything (e.g. GNU Hello) it stops with the message

/nix/store/65l6hr8snf4v823f974k97jc65i7bhvf-bash-4.4-p12/bin/bash: ar: command not found            
make[2]: *** [Makefile:135: /tmp/nix-build-glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf.drv-0/build/sunrpc/librpc_compat_pic.a] Error 127            
make[2]: *** Waiting for unfinished jobs....                                    
make[2]: Leaving directory '/tmp/nix-build-glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf.drv-0/glibc-2.26/sunrpc'                 
make[1]: *** [Makefile:215: sunrpc/subdir_lib] Error 2                          
make[1]: Leaving directory '/tmp/nix-build-glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf.drv-0/glibc-2.26'                        
make: *** [Makefile:9: all] Error 2     
note: keeping build directory ‘/tmp/nix-build-glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf.drv-0’                                
builder for ‘/nix/store/dzjfk7nklc4hbvwc7nq8jkghdz8y3q5x-glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf.drv’ failed with exit code 2                   
cannot build derivation ‘/nix/store/svl5d7mnfw08iif7mvw3qmdw8mdj6ygi-armv6l-unknown-linux-gnueabihf-gcc-cross-wrapper-6.4.0-armv6l-unknown-linux-gnueabihf-stage-final.drv’: 1 dependencies couldn't be built                                   
cannot build derivation ‘/nix/store/5x9lzs3d8gc29za2xc80ckls9ab4nfjv-stdenv.drv’: 1 dependencies couldn't be built      
cannot build derivation ‘/nix/store/xfaldyq4zn097p8vypp77dj80f4v4v2r-hello-2.10-
armv6l-unknown-linux-gnueabihf.drv’: 1 dependencies couldn't be built           

Steps to reproduce

Checkout nixpkgs master (or any version since 8e557ed) and add this file to the top-level nixpkgs directory

$ cat foo.nix
let armHost =  import ./default.nix {
       crossSystem = (import ./lib ).systems.examples.raspberryPi;
    };
    dontCheck = p : s : s.lib.overrideDerivation p (a: {
      doCheck = false ;
    });
in rec {
  armHello = dontCheck armHost.pkgs.hello armHost;
}

Now run nix-build ./foo.nix -A armHello and watch it fail

Technical details

I've done a bit of poking around and in all honesty I'm not sure if the problem is with the packaging or the upstream (or me, even). AR is set correctly in the shell environment:

[dan@loaclhost:/tmp/nix-build-glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf.drv-1]$ ( . env-vars && echo $AR )
armv6l-unknown-linux-gnueabihf-ar

but something in the glibc Makefile maze overrides it

[dan@loaclhost:/tmp/nix-build-glibc-2.26-75-armv6l-unknown-linux-gnueabihf-armv6l-unknown-linux-gnueabihf.drv-1/build]$ grep AR config.make 
AR = ar

I suspect (though am not 100%, it was quite late at night) that glibc may have been using the build ar instead of the host ar for a long time but only recently has it disappeared from the path.

Affected versions

  • git bisect says this behaviour was introduced in 8e557ed "bintools-wrapper: Init"
  • it still exists in 857a71c
  • nix-build (Nix) 1.11.13

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

[dan@loaclhost:~/src/nixwrt-bisecting]$ nix-shell -p nix-info --run "nix-info -m"
error: undefined variable ‘nix-info’ at (string):1:66
(use ‘--show-trace’ to show detailed location information)

?

My understanding of the Nix cross-compilation environment is frankly a bit weak and I might of course just be Doing It All Wrong - any advice to that effect would also be gratefully received.

@telent

This comment has been minimized.

Show comment
Hide comment
@telent

telent Dec 24, 2017

Contributor

Not sure of etiquette for tagging individual developers, but suspect that @Ericson2314 may be well placed to have a look at this

Contributor

telent commented Dec 24, 2017

Not sure of etiquette for tagging individual developers, but suspect that @Ericson2314 may be well placed to have a look at this

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Dec 24, 2017

Member

This is fixed with #26805. Cross compilation is completely unreliable until that lands.

Member

Ericson2314 commented Dec 24, 2017

This is fixed with #26805. Cross compilation is completely unreliable until that lands.

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Dec 24, 2017

Member

And tagging me is perfectly fine.

Member

Ericson2314 commented Dec 24, 2017

And tagging me is perfectly fine.

@telent

This comment has been minimized.

Show comment
Hide comment
@telent

telent Dec 27, 2017

Contributor

Thanks. I rebased/cherry-picked my actual tree (mips32 embedded stuff) onto said branch and it's working much better - and has allowed me to significantly clean up the derivation for the kernel. I look forward to seeing your branch land.

FWIW my work is at obsidiansystems/nixpkgs@02726a2...telent:nixwrt-cross-elegant if you're interested, and contains a few trivial-with-hindsight cross-compilation fixes you might want to look at -

I guess the question is, would you like these as a PR against master, or to cherry-pick them into your branch (perhaps makes it easier for you to avoid conflicts), or should I hold them until after your merge? Let me know.


Next step is working out why my libc has gcc and linux headers as runtime dependency, but that's a separate issue.

[dan@loaclhost:~/src/nixwrt]$ nix-store -q --references /nix/store/cmaafly9mrpvdl2z9cl228k9xnfkmfa6-glibc-2.26-75-mipsel-unknown-linux-gnu-mipsel-unknown-linux-gnu |cat
/nix/store/ibxd54g3h666g2dljp5f1vr2jzs5qc39-gcc-6.4.0-mipsel-unknown-linux-gnu-stage-static
/nix/store/z81llg5qxxfjyf4s5y4hdnbpyrd91sz7-linux-headers-4.4.10-mipsel-unknown-linux-gnu
/nix/store/cmaafly9mrpvdl2z9cl228k9xnfkmfa6-glibc-2.26-75-mipsel-unknown-linux-gnu-mipsel-unknown-linux-gnu
Contributor

telent commented Dec 27, 2017

Thanks. I rebased/cherry-picked my actual tree (mips32 embedded stuff) onto said branch and it's working much better - and has allowed me to significantly clean up the derivation for the kernel. I look forward to seeing your branch land.

FWIW my work is at obsidiansystems/nixpkgs@02726a2...telent:nixwrt-cross-elegant if you're interested, and contains a few trivial-with-hindsight cross-compilation fixes you might want to look at -

I guess the question is, would you like these as a PR against master, or to cherry-pick them into your branch (perhaps makes it easier for you to avoid conflicts), or should I hold them until after your merge? Let me know.


Next step is working out why my libc has gcc and linux headers as runtime dependency, but that's a separate issue.

[dan@loaclhost:~/src/nixwrt]$ nix-store -q --references /nix/store/cmaafly9mrpvdl2z9cl228k9xnfkmfa6-glibc-2.26-75-mipsel-unknown-linux-gnu-mipsel-unknown-linux-gnu |cat
/nix/store/ibxd54g3h666g2dljp5f1vr2jzs5qc39-gcc-6.4.0-mipsel-unknown-linux-gnu-stage-static
/nix/store/z81llg5qxxfjyf4s5y4hdnbpyrd91sz7-linux-headers-4.4.10-mipsel-unknown-linux-gnu
/nix/store/cmaafly9mrpvdl2z9cl228k9xnfkmfa6-glibc-2.26-75-mipsel-unknown-linux-gnu-mipsel-unknown-linux-gnu
@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Dec 27, 2017

Member

Oh sorry I should have also pointed out #30882 which contains all the per-package fixes. (The first PR's misc changes are mainly to keep the native builds from regressin.)

The plan is to land the first PR more or less as is to get the foundation on place, and then land the per-package fixes much more rapidly.

Member

Ericson2314 commented Dec 27, 2017

Oh sorry I should have also pointed out #30882 which contains all the per-package fixes. (The first PR's misc changes are mainly to keep the native builds from regressin.)

The plan is to land the first PR more or less as is to get the foundation on place, and then land the per-package fixes much more rapidly.

@Ericson2314 Ericson2314 reopened this Dec 27, 2017

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Dec 27, 2017

Member

So feel free to rebase your changes on the second one instead, @bgamari and I should be able to encorperate it there ahead of the first being merged.

Member

Ericson2314 commented Dec 27, 2017

So feel free to rebase your changes on the second one instead, @bgamari and I should be able to encorperate it there ahead of the first being merged.

@flokli

This comment has been minimized.

Show comment
Hide comment
@flokli
Contributor

flokli commented Dec 27, 2017

@Ericson2314 Ericson2314 self-assigned this Dec 28, 2017

@Ericson2314 Ericson2314 added this to Solved by big PR in Cross compilation Dec 28, 2017

@Ericson2314

This comment has been minimized.

Show comment
Hide comment
@Ericson2314

Ericson2314 Dec 31, 2017

Member

#26805 has landed in staging

Member

Ericson2314 commented Dec 31, 2017

#26805 has landed in staging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment