New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: chroot: failed to run command '/debootstrap/debootstrap': No such file or directory #6

Closed
ruffsl opened this Issue Feb 27, 2017 · 11 comments

Comments

Projects
None yet
2 participants
@ruffsl
Copy link
Member

ruffsl commented Feb 27, 2017

I'm having issues using the build script to bootstrap some ubuntu images for amr. I'm able to build the i386 images just fine, but when I try to build an images of any arm arch, of any suite, I encounter this error:

osrf/multiarch-docker-image-generation$ ./build_all.py --os ubuntu --suite xenial --arch armhf
...
I: Extracting zlib1g...
+ [[ armhf arm64 =~ armhf ]]
+ '[' armhf == armhf ']'
+ cp qemu-arm-static /var/chroot/ubuntu_armhf_xenial/usr/bin/
+ LC_ALL=C
+ LANGUAGE=C
+ LANG=C
+ chroot /var/chroot/ubuntu_armhf_xenial /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': No such file or directory
failed to process osrf/ubuntu_armhf:xenial
...

I can see the static qemu binary has been placed in the chroot, as so has debootstrap:

/var/chroot$ ls ubuntu_armhf_xenial/debootstrap/
arch  base  debootstrap  debootstrap.log  debpaths  deburis  devices.tar.gz  functions  required  suite  suite-script  variant

/var/chroot$ ls ubuntu_armhf_xenial/usr/bin/ | grep qemu
qemu-arm-static

Is there something I'm forgetting to install, or is there an issue with the static qemu files in the repo for my system?

System info:

$ uname -a
Linux cog-1 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.1 LTS
Release:	16.04
Codename:	xenial

ping @tfoote

@tfoote

This comment has been minimized.

Copy link
Contributor

tfoote commented Feb 27, 2017

@ruffsl I can't reproduce your error:

$ uname -a
Linux snowman 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
LSB Version:	core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.1 LTS
Release:	16.04
Codename:	xenial
@ruffsl

This comment has been minimized.

Copy link
Member

ruffsl commented Feb 27, 2017

Ok, on a new machine I:

sudo apt install debootstrap

otherwise you'd get:

+ set -e
+ '[' ubuntu ']'
+ os=ubuntu
+ '[' armhf ']'
+ arch=armhf
+ '[' xenial ']'
+ suite=xenial
+ chroot_dir=/var/chroot/ubuntu_armhf_xenial
+ docker_image=osrf/ubuntu_armhf:xenial
+ foreign_arches=(armhf arm64)
+ '[' ubuntu == ubuntu ']'
+ [[ armhf arm64 =~ armhf ]]
+ apt_mirror=http://ports.ubuntu.com
+ export DEBIAN_FRONTEND=noninteractive
+ DEBIAN_FRONTEND=noninteractive
+ foreign_arg=
+ [[ armhf arm64 =~ armhf ]]
+ foreign_arg=--foreign
+ debootstrap --foreign --variant=minbase --arch=armhf xenial /var/chroot/ubuntu_armhf_xenial http://ports.ubuntu.com
./build-image.sh: line 58: debootstrap: command not found

But after debootstrap is installed, I still get the same error:

I: Extracting zlib1g...
+ [[ armhf arm64 =~ armhf ]]
+ '[' armhf == armhf ']'
+ cp qemu-arm-static /var/chroot/ubuntu_armhf_xenial/usr/bin/
+ LC_ALL=C
+ LANGUAGE=C
+ LANG=C
+ chroot /var/chroot/ubuntu_armhf_xenial /debootstrap/debootstrap --second-stage
chroot: failed to run command '/debootstrap/debootstrap': Exec format error
failed to process osrf/ubuntu_armhf:xenial

Could you try it on a new machine, I'm thinking your workstation has something set that my two do not.

$ sudo dpkg -l | grep 'docker\|debootstrap'
ii  debootstrap                                  1.0.78+nmu1ubuntu1.2                                        all          Bootstrap a basic Debian system
ii  docker-engine                                1.13.1-0~ubuntu-xenial                                      amd64        Docker: the open-source application container engine
@tfoote

This comment has been minimized.

Copy link
Contributor

tfoote commented Feb 27, 2017

I'm surprised to see the exec format error. debootstrap is an sh script not an elf binary.

$ file /var/chroot/ubuntu_armhf_xenial/debootstrap/debootstrap
/var/chroot/ubuntu_armhf_xenial/debootstrap/debootstrap: POSIX shell script, ASCII text executable
$ head /var/chroot/ubuntu_armhf_xenial/debootstrap/debootstrap
#!/bin/sh
set -e

VERSION='1.0.78+nmu1ubuntu1.2'

unset TMP TEMP TMPDIR || true

# might not be exported if we're running from init=/bin/sh or similar
export PATH

$ dpkg -l debootstrap
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  debootstrap    1.0.78+nmu1u all          Bootstrap a basic Debian system

@ruffsl

This comment has been minimized.

Copy link
Member

ruffsl commented Feb 28, 2017

That's what I noticed when looking up some stack overflow issues:
http://unix.stackexchange.com/questions/128046/chroot-failed-to-run-command-bin-bash-no-such-file-or-directory
http://unix.stackexchange.com/questions/76490/no-such-file-or-directory-on-an-executable-yet-file-exists-and-ldd-reports-al

I think it has something to do with /bin/sh not being emulated using the emulation with qemu-arm-static in the chroot. Do you have qemu installed on the host system? I tried that, but didn't effect anything.

Can't seem to run bash even:

$ LC_ALL=C LANGUAGE=C LANG=C sudo chroot /var/chroot/ubuntu_armhf_xenial /bin/bash -s ls
chroot: failed to run command '/bin/bash': Exec format error
@tfoote

This comment has been minimized.

Copy link
Contributor

tfoote commented Feb 28, 2017

I have qemu-arm-static version 1:2.5+dfsg-5ubuntu10.7

$ dpkg -l qemu-user-static
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version             Architecture        Description
+++-=============================-===================-===================-===============================================================
ii  qemu-user-static              1:2.5+dfsg-5ubuntu1 amd64               QEMU user mode emulation binaries (static version)

It looks like there's a patch update available to build 10.9

@ruffsl

This comment has been minimized.

Copy link
Member

ruffsl commented Feb 28, 2017

Ah, ok I had to install qemu-user-static on my host to "bootstrap" the deboostrap script that uses the armhf bash binary in the chroot.

sudo apt install qemu-user-static

Now I can start bash:

$ LC_ALL=C LANGUAGE=C LANG=C sudo chroot /var/chroot/ubuntu_armhf_xenial /bin/bash -s ls
/usr/bin/groups: cannot find name for group ID 0
I have no name!@dox:/# exit
exit

And it looks like trusty and xenial are building for armhf. Mind if I send a PR to note about that dependency on the readme?

Would the docker images work for docker hosts that don't have qemu-user-static installed? I'm thinking no as it looks like I get this error on the machine I haven't yet installed qemu-user-static on:

$ docker run -it osrf/ubuntu_armhf:trusty bash
standard_init_linux.go:178: exec user process caused "no such file or directory"

How is the /usr/bin/qemu-arm-static file being used in the docker image, i.e. how does a deriving container system environment made aware to utilize the emulator, and not that of the host's?

@tfoote

This comment has been minimized.

Copy link
Contributor

tfoote commented Feb 28, 2017

A pr for documenting the dependency would be great. I thought that the internal copy meant that the external one didn't need it. The included one is newer and I guess ends up on the path before the external one. Maybe it's a bootstrapping issue needing qemu to bootstrap it, but once it's running it's self contained?

@ruffsl

This comment has been minimized.

Copy link
Member

ruffsl commented Feb 28, 2017

I think installing qemu-user-static (and binfmt-support qemu or via update-binfmts --enable qemu-arm?) makes sure the binfmt entries were successfully registered for arm [1]. This would be needed for the method the script looks to be following, the "QEMU/debootstrap approach" as listed in the CrossDebootstrap wiki for EmDebian [2].

In this case the host's kernel still needs to be configured, as when the binfmt_misc module is loaded, we don't have to prefix every command with /usr/bin/qemu-arm-static. When Docker attempts to run /bin/sh, the kernel will automatically detect it is an ARM executable and invoke QEMU. Basicly we are coping over the qemu-arm-static file as "the qemu emulator for the target CPU needs to be accessible from inside the chroot jail", so we don't have to mount it from a host later.

We could run a armhf binary our current images on a unconfigured docker host like so:

docker run -it --entrypoint /usr/bin/qemu-arm-static osrf/ubuntu_armhf:trusty /bin/bash
root@13bf18a5f903:/# /usr/bin/qemu-arm-static /bin/echo Hello from ARM container

Although this would drop you into a shell, you wouldn't be able to start anything else with the shell, as "QEMU will happily pass this on to the kernel, but the kernel has no idea what to do with this file since /bin/echo is an ARM binary!". Unless, again, we'd have to prefix it with qemu.

For "Dropping the kernel dependency" [3], we could follow the example used in Resin's armv7hf-debian-qemu repo [4]. You'd think there'd be a better way than playing switcheroo with a shim and /bin/sh/?

Looks like that's what some folks have been doing [5].

[1] https://wiki.debian.org/QemuUserEmulation
[2] https://wiki.debian.org/EmDebian/CrossDebootstrap#QEMU.2Fdebootstrap_approach
[3] https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/#droppingthekerneldependency
[4] https://github.com/resin-io-projects/armv7hf-debian-qemu
[5] https://github.com/sdt/docker-raspberry-pi-cross-compiler/blob/441846d48081601b5ade0144ec8ea13527ad8a9d/Dockerfile#L40

@ruffsl

This comment has been minimized.

Copy link
Member

ruffsl commented Mar 2, 2017

Looks like there is qemu-debootstrap for these situations that I first encountered:
https://wiki.debian.org/ArmHardFloatChroot

@tfoote

This comment has been minimized.

Copy link
Contributor

tfoote commented Aug 16, 2017

@ruffsl Is there a follow up needed for this? I see that qemu-debootstrap is part of qemu-user-static

ruffsl added a commit to ruffsl/ubuntu_32bit-docker that referenced this issue Aug 16, 2017

@ruffsl

This comment has been minimized.

Copy link
Member

ruffsl commented Aug 16, 2017

@tfoote , see #8

@ruffsl ruffsl closed this Aug 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment