New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop at stage 2 with "Illegal instruction" error #205

Closed
hongquan opened this Issue Sep 19, 2018 · 15 comments

Comments

Projects
None yet
2 participants
@hongquan

hongquan commented Sep 19, 2018

Hi. I'm trying pi-gen on an ARM bare metal server (from Scaleway). The script stopped at the middle with error:

[09:22:16] Begin /home/quan/RaspberryPi/pi-gen/stage2
[09:22:16] Begin /home/quan/RaspberryPi/pi-gen/stage2/prerun.sh
[09:22:16] End /home/quan/RaspberryPi/pi-gen/stage2/prerun.sh
[09:22:16] Begin /home/quan/RaspberryPi/pi-gen/stage2/00-copies-and-fills
[09:22:16] Begin /home/quan/RaspberryPi/pi-gen/stage2/00-copies-and-fills/01-packages
/home/quan/RaspberryPi/pi-gen/scripts/common: line 71: 26057 Illegal instruction     capsh --drop=cap_setfcap "--chroot=${ROOTFS_DIR}/" -- "$@"

Please help.

@XECDesign

This comment has been minimized.

Contributor

XECDesign commented Sep 19, 2018

Unfortunately their servers use Cavium ThunderX processors which only have Aarch64 instructions, so they can't be used for this.

@hongquan

This comment has been minimized.

hongquan commented Sep 19, 2018

Thank you. I thought they use ARM 32bit processors (I can build BeagleBone image successfully on it).

@hongquan hongquan closed this Sep 19, 2018

@hongquan

This comment has been minimized.

hongquan commented Sep 19, 2018

@XECDesign Can you review again, I check here https://www.scaleway.com/faq/servers/ and they say that ARM64 is for different package - Virtual Cloud Servers (which is already no longer offered). I'm using the C1 which is based on ARMv7.

@hongquan hongquan reopened this Sep 19, 2018

@XECDesign

This comment has been minimized.

Contributor

XECDesign commented Sep 19, 2018

Yes, you're right. It should work on C1. Have you made any changes to the image build? What's the host os? Does it work if you remove the copies-and-fills substage?

@hongquan

This comment has been minimized.

hongquan commented Sep 20, 2018

I made this change:

  • In stage0/01-locale/00-debconf, I change default locale to en_US.UTF-8.
    The OS of the server is:
$ uname -a
Linux meomeo 4.9.93-mainline-rev1 #1 SMP Tue Apr 10 09:42:40 UTC 2018 armv7l GNU/Linux

The weird thing is that, the boot kernel is reported to be version 4.9.93, but it doesn't exist on disk:

$ ls /boot/
config-4.9.0-8-armmp  grub  initrd.img-4.9.0-8-armmp  System.map-4.9.0-8-armmp  vmlinuz-4.9.0-8-armmp

$ dpkg -l | grep linux
ii  firmware-linux-free            3.4                                  all          Binary firmware for various drivers in the Linux kernel
ii  libselinux1:armhf              2.6-3+b3                             armhf        SELinux runtime shared libraries
ii  linux-base                     4.5                                  all          Linux image base package
ii  linux-image-4.9.0-8-armmp      4.9.110-3+deb9u4                     armhf        Linux 4.9 for ARMv7 multiplatform compatible SoCs
ii  linux-image-armmp              4.9+80+deb9u6                        armhf        Linux for ARMv7 multiplatform compatible SoCs (meta-package)
ii  linux-libc-dev:armhf           4.9.110-3+deb9u4                     armhf        Linux support headers for userspace development
ii  util-linux                     2.29.2-1+deb9u1                      armhf        miscellaneous system utilities

I haven't tried skipping copies-and-fills substage yet.

@hongquan

This comment has been minimized.

hongquan commented Sep 20, 2018

I skipped the stage2/00-copies-and-fills/ substage by putting a SKIP file. Now the script stops at the next substage with the same error.

[03:55:42] Begin /home/quan/RaspberryPi/pi-gen/stage2/01-sys-tweaks
[03:55:42] Begin /home/quan/RaspberryPi/pi-gen/stage2/01-sys-tweaks/00-debconf
/home/quan/RaspberryPi/pi-gen/scripts/common: line 71:  9159 Illegal instruction     capsh --drop=cap_setfcap "--chroot=${ROOTFS_DIR}/" -- "$@"
@hongquan

This comment has been minimized.

hongquan commented Sep 21, 2018

The Debian image of Scaleway contains foreign binary, for example:

$ file /usr/bin/qemu-arm-static
/usr/bin/qemu-arm-static: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=90042da8e1c0256ca8d131471c06769cb054e3f9, stripped, too many notes (256)

I think it is the cause of "Illegal instruction" error. I'm looking for where that file comes from...

@hongquan

This comment has been minimized.

hongquan commented Sep 22, 2018

@XECDesign I has fixed the Scaleway server and made the build script run successfully. I would like to contribute to Wiki about issues with Scaleway server. Do you want that?

@hongquan hongquan closed this Sep 22, 2018

@XECDesign

This comment has been minimized.

Contributor

XECDesign commented Sep 24, 2018

I think it's worthwhile figuring out exactly why the issue happened and fixing it within pi-gen. My guess is that we check whether we need qemu-arm-static by checking whether we're running on armhf and scaleway uses an aarch64 kernel? What does uname -m say?

@hongquan

This comment has been minimized.

hongquan commented Sep 24, 2018

@XECDesign:

$ uname -a
Linux meomeo 4.9.93-mainline-rev1 #1 SMP Tue Apr 10 09:42:40 UTC 2018 armv7l GNU/Linux

$ uname -m
armv7l

$ file /usr/bin/file
/usr/bin/file: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=88a61f41f497572fba7d4d2a62fcf8e0b00c3b38, stripped

The C1 server is actually ARM 32 bit.

After I delete the pre-included /usr/bin/qemu-arm-static file and reinstall qemu-user-static, now I get:

$ file /usr/bin/qemu-arm-static
/usr/bin/qemu-arm-static: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=5c15a15a6f96e3a528a1017487e2560a6edad857, stripped

I wonder why even I run pi-gen on an ARM machine, it still need to execute qemu?

@XECDesign

This comment has been minimized.

Contributor

XECDesign commented Sep 24, 2018

It shouldn't. There's some logic (or lack therof) which needs to be fixed here:
The logic here (or lack thereof) should be fixed https://github.com/RPi-Distro/pi-gen/blob/master/export-image/00-allow-rerun/00-run.sh
But that wouldn't result in the behaviour you saw.

I thought the problem was here:
https://github.com/RPi-Distro/pi-gen/blob/master/scripts/common#L12
That seems to mostly do the right thing, but without aarch64 support.

I guess scaleway have '/usr/bin/qemu-arm-static' because they left it there after preparing their images, or they use it to chroot into their disks whenever they need to. I'm not sure how it would've ended up in your pi-gen chroot at that stage. There are no other references to it.

@hongquan

This comment has been minimized.

hongquan commented Sep 24, 2018

The logic at https://github.com/RPi-Distro/pi-gen/blob/master/scripts/common#L12 worked as expected, because it executed debootstrap, not qemu-debootstrap in my ARM server.

But there is another problem, also with Scaleway, is that Scaleway pre-included a debootstrap of different version. The debootstrap of Debian Stretch is 1.0.89, but some hidden Scaleway software added an APT source file scaleway-ubuntu-stable-bionic.list which brought debootstrap 1.0.109 and caused pi-gen failing.

I can manage to make pi-gen run successfully by removing debootstrap, delete that APT source file, then install debootstrap again. But after a while, debootstrap suddenly become 1.0.109 and that APT source file appears again, like a myth!

@XECDesign

This comment has been minimized.

Contributor

XECDesign commented Sep 24, 2018

Ah, then pi-gen probably can't account for haunted servers. The spirits of Scaleway seem to demand a special version of debootstrap, which adds qemu-arm-static, I guess.

@hongquan

This comment has been minimized.

hongquan commented Sep 27, 2018

Today pi-gen fails again at stage2/01-sys-tweaks/00-debconf, with the same "Illegal instruction" error, even though the file /usr/bin/qemu-arm-static is already correct.

@XECDesign

This comment has been minimized.

Contributor

XECDesign commented Sep 28, 2018

Something else bringing in x64 contamination? Sorry, I don't use scaleway, so I really can't look into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment