This repository has been archived by the owner. It is now read-only.

[titan] Cannot boot kernel compiled with pmOS #126

Closed
PabloCastellano opened this Issue Jul 2, 2017 · 12 comments

Comments

Projects
None yet
5 participants
@PabloCastellano
Member

PabloCastellano commented Jul 2, 2017

It can happen as well to other devices.

LineageOS boot image is 8MB and I can flash it but my pmOS boot image is 12MB, of which 8MB is the kernel.

We need to find some way to make the kernel smaller. I had to disable CONFIG_KERNEL_XZ otherwise the kernel would not boot. With CONFIG_KERNEL_LZMA it won't boot either.

Things to investigate:

  • Why just enabling XZ kernel compression doesn't work (this works for hammerhead)
  • How much we can achieve without the gzip wrapper (code)
  • CONFIG_CC_OPTIMIZE_FOR_SIZE=y
  • Add a gzip layer around the already compressed vmlinuz (and repack boot.img) (*)
  • Only compress initrd with XZ
  • Load modules instead
  • How can we know that is the size limit beforehand?
  • Can we change this limit? Is it safe?
  • Is this limit the same for all the devices?

(*) MartijnBraam said on IRC:

you can try compressing the xz kernel image with gzip so the bootloader unpacks only one layer
Your bootloader might be patched to always gunzip
Littlekernel should detect the gzip header and unpack before booting

This is the output of fastboot:

$ sudo fastboot flash boot boot.img 
target reported max download size of 536870912 bytes
sending 'boot' (11920 KB)...
OKAY [  0.461s]
writing 'boot'...
(bootloader) Image size exeeded partition limits
(bootloader) Preflash validation failed
FAILED (remote failure)
finished. total time: 0.501s

With LineageOS boot.img:

sudo fastboot flash boot boot.img 
target reported max download size of 536870912 bytes
sending 'boot' (8192 KB)...
OKAY [  0.352s]
writing 'boot'...
OKAY [  0.459s]
finished. total time: 0.811s
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 2, 2017

Member

pmbootstrap uses a hack to make gzip always use the fastest compression method, instead of the best compressing one (trading speed for file size). That hack gets installed to /usr/local/bin/gzip and it impacts both the size of the initramfs, and the kernel (when the kernel gets compressed with gzip). These are side-effects, the original idea of the hack is to reduce the size of the apk packages, and therefore the build time.

What you could do now:

  • Remove /usr/local/bin/gzip in all your chroots, and build the kernel image and initramfs again, it should be smaller. When this is all, that's needed to fix this issue for you, I think we should provide this as an option (or even better, upstream a compression parameter to apk, so we don't need the hack at all)
  • Choose another compression algorithm in the kernel config (example)
Member

ollieparanoid commented Jul 2, 2017

pmbootstrap uses a hack to make gzip always use the fastest compression method, instead of the best compressing one (trading speed for file size). That hack gets installed to /usr/local/bin/gzip and it impacts both the size of the initramfs, and the kernel (when the kernel gets compressed with gzip). These are side-effects, the original idea of the hack is to reduce the size of the apk packages, and therefore the build time.

What you could do now:

  • Remove /usr/local/bin/gzip in all your chroots, and build the kernel image and initramfs again, it should be smaller. When this is all, that's needed to fix this issue for you, I think we should provide this as an option (or even better, upstream a compression parameter to apk, so we don't need the hack at all)
  • Choose another compression algorithm in the kernel config (example)
@PabloCastellano

This comment has been minimized.

Show comment
Hide comment
@PabloCastellano

PabloCastellano Jul 2, 2017

Member

Some numbers:

  • Current initramfs is 3MB (being /lib/libcrypto.so.41 with 1.3mb is the biggest file)
  • With GZIP compression, kernel is 8,5MB and boot.img is 12MB
  • With GZIP compression and CONFIG_CC_OPTIMIZE_FOR_SIZE=y, kernel is 8,2MB
  • With LZMA compression, kernel is 6,8MB and boot.img is 9,8MB
  • With XZ compression, kernel is 6,5MB and boot.img is 9,5MB
  • With XZ compression, and gzipped vmlinuz kernel is 4,8MB and boot.img is 7,8MB (*)
  • I found no difference in the size with XZ compression and the GZIP wrapper removed

(*) But it doesn't boot either - it just reboots

Member

PabloCastellano commented Jul 2, 2017

Some numbers:

  • Current initramfs is 3MB (being /lib/libcrypto.so.41 with 1.3mb is the biggest file)
  • With GZIP compression, kernel is 8,5MB and boot.img is 12MB
  • With GZIP compression and CONFIG_CC_OPTIMIZE_FOR_SIZE=y, kernel is 8,2MB
  • With LZMA compression, kernel is 6,8MB and boot.img is 9,8MB
  • With XZ compression, kernel is 6,5MB and boot.img is 9,5MB
  • With XZ compression, and gzipped vmlinuz kernel is 4,8MB and boot.img is 7,8MB (*)
  • I found no difference in the size with XZ compression and the GZIP wrapper removed

(*) But it doesn't boot either - it just reboots

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 20, 2017

Member

Let me quote @drebrez, who made a very relevant comment in #127 (which is about including new programs in the initramfs to make automatic resizing of the system partition possible).

Another good solution would be to save all the additional tools (cryptsetup, parted, resize2fs, ...) that we need, inside the boot partition (or a dedicated one) of the pmos system image.
In this way we will reduce the initramfs size and it might help solving the #126.

@ollieparanoid @PabloCastellano what do you think?

Here some sizes based on what we include in mkinitfs.sh#L103:

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx"

3'075 KiB

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx /sbin/e2fsck /usr/sbin/resize2fs"

3'771 KiB

BINARIES="/bin/busybox /bin/busybox-extras /usr/sbin/telnetd /sbin/kpartx"

1'567 KiB

Member

ollieparanoid commented Jul 20, 2017

Let me quote @drebrez, who made a very relevant comment in #127 (which is about including new programs in the initramfs to make automatic resizing of the system partition possible).

Another good solution would be to save all the additional tools (cryptsetup, parted, resize2fs, ...) that we need, inside the boot partition (or a dedicated one) of the pmos system image.
In this way we will reduce the initramfs size and it might help solving the #126.

@ollieparanoid @PabloCastellano what do you think?

Here some sizes based on what we include in mkinitfs.sh#L103:

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx"

3'075 KiB

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx /sbin/e2fsck /usr/sbin/resize2fs"

3'771 KiB

BINARIES="/bin/busybox /bin/busybox-extras /usr/sbin/telnetd /sbin/kpartx"

1'567 KiB

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 20, 2017

Member

I think it's a good idea to put extra files in the boot partition - but I would put them in an extra file (e.g. initramfsbig-samsung-i9100?).

What we would need to keep in the real initramfs is:

  • busybox
  • kpartx (to detect the boot partition, if it is a subpartition of the system partition)

Dependencies:

/ # lddtree $(which kpartx)
kpartx => /sbin/kpartx (interpreter => /lib/ld-musl-armhf.so.1)
    libdevmapper.so.1.02 => /lib/libdevmapper.so.1.02
    libc.musl-armhf.so.1 => /lib/libc.musl-armhf.so.1
/ # lddtree $(which busybox)
busybox => /bin/busybox (interpreter => /lib/ld-musl-armhf.so.1)
    libc.musl-armhf.so.1 => /lib/libc.musl-armhf.so.1

Total size:

/ # du -h -c /sbin/kpartx /lib/libdevmapper.so.1.02 /lib/ld-musl-armhf.so.1 /bin/busybox
40.0K   /sbin/kpartx
288.0K  /lib/libdevmapper.so.1.02
536.0K  /lib/ld-musl-armhf.so.1
920.0K  /bin/busybox
1.7M    total

Remarks:

  • that size is uncompressed
  • we could make it even smaller by compiling only what we need from busybox to find/mount the boot partition and extract the big initramfs image. The busybox-extras binary (which contains telnetd among very few others) is 88kb in size for example (!) (we could have our own busybox-initfs package or something like that)
  • we could make it smaller by compiling statically against musl. that way only the parts of musl, that are actually used, are kept.
Member

ollieparanoid commented Jul 20, 2017

I think it's a good idea to put extra files in the boot partition - but I would put them in an extra file (e.g. initramfsbig-samsung-i9100?).

What we would need to keep in the real initramfs is:

  • busybox
  • kpartx (to detect the boot partition, if it is a subpartition of the system partition)

Dependencies:

/ # lddtree $(which kpartx)
kpartx => /sbin/kpartx (interpreter => /lib/ld-musl-armhf.so.1)
    libdevmapper.so.1.02 => /lib/libdevmapper.so.1.02
    libc.musl-armhf.so.1 => /lib/libc.musl-armhf.so.1
/ # lddtree $(which busybox)
busybox => /bin/busybox (interpreter => /lib/ld-musl-armhf.so.1)
    libc.musl-armhf.so.1 => /lib/libc.musl-armhf.so.1

Total size:

/ # du -h -c /sbin/kpartx /lib/libdevmapper.so.1.02 /lib/ld-musl-armhf.so.1 /bin/busybox
40.0K   /sbin/kpartx
288.0K  /lib/libdevmapper.so.1.02
536.0K  /lib/ld-musl-armhf.so.1
920.0K  /bin/busybox
1.7M    total

Remarks:

  • that size is uncompressed
  • we could make it even smaller by compiling only what we need from busybox to find/mount the boot partition and extract the big initramfs image. The busybox-extras binary (which contains telnetd among very few others) is 88kb in size for example (!) (we could have our own busybox-initfs package or something like that)
  • we could make it smaller by compiling statically against musl. that way only the parts of musl, that are actually used, are kept.

@ollieparanoid ollieparanoid referenced this issue Jul 20, 2017

Closed

"The Mainline Kernel" #91

5 of 5 tasks complete
@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Jul 20, 2017

Member

I've created a new issue for the initramfs split solution: #211

Member

ollieparanoid commented Jul 20, 2017

I've created a new issue for the initramfs split solution: #211

@PabloCastellano

This comment has been minimized.

Show comment
Hide comment
@PabloCastellano

PabloCastellano Jul 28, 2017

Member

After #257 has been merged, I'm still unable to flash the boot partition. However the error message is different. I've not yet investigated:

$ pmb flasher flash_kernel
[21:47:06] (rootfs_motorola-titan) mkinitfs motorola-titan
[21:47:18] (native) flash kernel motorola-titan
(bootloader) has-slot:boot: not found
target reported max download size of 536870912 bytes
sending 'boot' (10616 KB)...
OKAY [  0.408s]
writing 'boot'...
(bootloader) Image size exeeded partition limits
(bootloader) Preflash validation failed
FAILED (remote failure)
finished. total time: 0.456s
[21:52:12] ERROR: Command failed: (native) % fastboot flash boot /mnt/rootfs_motorola-titan/boot/boot.img-motorola-titan
[21:52:12] Run 'pmbootstrap log' for details.
[21:52:12] See also: <https://postmarketos.org/troubleshooting>
Member

PabloCastellano commented Jul 28, 2017

After #257 has been merged, I'm still unable to flash the boot partition. However the error message is different. I've not yet investigated:

$ pmb flasher flash_kernel
[21:47:06] (rootfs_motorola-titan) mkinitfs motorola-titan
[21:47:18] (native) flash kernel motorola-titan
(bootloader) has-slot:boot: not found
target reported max download size of 536870912 bytes
sending 'boot' (10616 KB)...
OKAY [  0.408s]
writing 'boot'...
(bootloader) Image size exeeded partition limits
(bootloader) Preflash validation failed
FAILED (remote failure)
finished. total time: 0.456s
[21:52:12] ERROR: Command failed: (native) % fastboot flash boot /mnt/rootfs_motorola-titan/boot/boot.img-motorola-titan
[21:52:12] Run 'pmbootstrap log' for details.
[21:52:12] See also: <https://postmarketos.org/troubleshooting>
@PabloCastellano

This comment has been minimized.

Show comment
Hide comment
@PabloCastellano

PabloCastellano Jul 28, 2017

Member

More news!

Enabling CONFIG_KERNEL_LZMA now the boot.img is 8MB and I can flash it \o/.
However I'm still having the same problem of not being able to boot with XZ compression (I forgot that it also happened with LZMA). But at least it is flashable :)

If you don't mind I'd like to keep the issue opened so that I don't forget about investigating and documenting some of the findings

Member

PabloCastellano commented Jul 28, 2017

More news!

Enabling CONFIG_KERNEL_LZMA now the boot.img is 8MB and I can flash it \o/.
However I'm still having the same problem of not being able to boot with XZ compression (I forgot that it also happened with LZMA). But at least it is flashable :)

If you don't mind I'd like to keep the issue opened so that I don't forget about investigating and documenting some of the findings

@ollieparanoid ollieparanoid changed the title from [titan] Cannot flash boot.img because it is too big to [titan] Cannot boot kernel compiled with pmOS Jul 28, 2017

@drebrez

This comment has been minimized.

Show comment
Hide comment
@drebrez

drebrez Sep 13, 2017

Member

@PabloCastellano you can try some of these kernel options to reduce the size => http://elinux.org/Kernel_Size_Tuning_Guide#Kernel_Configuration_Options

Member

drebrez commented Sep 13, 2017

@PabloCastellano you can try some of these kernel options to reduce the size => http://elinux.org/Kernel_Size_Tuning_Guide#Kernel_Configuration_Options

craftyguy added a commit that referenced this issue Sep 20, 2017

Improve copy in mkinitfs
'install' does not properly handle symlinks, it copies the entire file
instead of a symlink to it. This PR uses `cp -a` to preserve symlinks.

For example, with `install`, the files libc.musl-armhf.so.1 and
ld-musl-armhf.so.1 are the same size, despite the fact that in rootfs
libc.musl-armhf.so.1 is actually a symlink to ld-musl-armhf.so.1.

From @drebrez:
```
initramfs:
old size => 1 567 930
new size => 1 168 591
```

This might also help with #126.

This PR is from a commit in the 'add osk' branch that I made, but it's
useful enough to have it here since it reduces the initramfs size by
copying any symlinks over as symlinks. For osk-sdl's dependencies, this
change dramatically reduces the size of initramfs-extras since binaries
are not copied twice.
@craftyguy

This comment has been minimized.

Show comment
Hide comment
@craftyguy

craftyguy Sep 20, 2017

Member

@PabloCastellano

See if #599 helps reduce the initramfs size for you.

Member

craftyguy commented Sep 20, 2017

@PabloCastellano

See if #599 helps reduce the initramfs size for you.

@walidham

This comment has been minimized.

Show comment
Hide comment
@walidham

walidham Oct 12, 2017

In arch/arm/mach-msm/Makefile.boot remove or commented thea dts files (if you want titan), like this:

MSM8226 Motorola Devices
ifeq ($(CONFIG_MMI_TITAN_DTB),y)
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4b.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4c.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4d.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4e.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4f.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p1a.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p1c.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p2.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p3.dtb

The size will be reduce, i have same issue in ubuntu touch, i got big kernel size and i can't flash it, but after this change, boot.img work well and ubuntu touch also

walidham commented Oct 12, 2017

In arch/arm/mach-msm/Makefile.boot remove or commented thea dts files (if you want titan), like this:

MSM8226 Motorola Devices
ifeq ($(CONFIG_MMI_TITAN_DTB),y)
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4b.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4c.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4d.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4e.dtb
dtb-$(CONFIG_ARCH_MSM8226)	+= msm8226-titan-4f.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p1a.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p1c.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p2.dtb
#dtb-$(CONFIG_ARCH_MSM8226)	+= msm8926-thea-p3.dtb

The size will be reduce, i have same issue in ubuntu touch, i got big kernel size and i can't flash it, but after this change, boot.img work well and ubuntu touch also

@PabloCastellano

This comment has been minimized.

Show comment
Hide comment
@PabloCastellano

PabloCastellano Nov 18, 2017

Member

So finally I found the issue with XZ compression in my kernel 🎉

It turns out that the kernel makefile is broken because it uses a very weak method to detect if you are using the non-gnu version fo the stat command and it fails if you are using busybox's stat, which is the case in Alpine by default. The message in the log is cryptic and hard to spot because it doesn't fail hardly and the zImage probably remains corrupt somehow:

  LD      vmlinux.o
  MODPOST vmlinux.o
  GEN     .version
  CHK     include/generated/compile.h
  LD      .tmp_vmlinux1
  KSYM    .tmp_kallsyms1.S
  AS      .tmp_kallsyms1.o
  LD      .tmp_vmlinux2
  KSYM    .tmp_kallsyms2.S
  AS      .tmp_kallsyms2.o
  LD      vmlinux
  SYSMAP  System.map
  SYSMAP  .tmp_System.map
  OBJCOPY arch/arm/boot/Image
  Kernel: arch/arm/boot/Image is ready
  AS      arch/arm/boot/compressed/head.o
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
  XZKERN  arch/arm/boot/compressed/piggy.xzkern
  CC      arch/arm/boot/compressed/misc.o
  CC      arch/arm/boot/compressed/decompress.o
  CC      arch/arm/boot/compressed/string.o
  SHIPPED arch/arm/boot/compressed/lib1funcs.S
  SHIPPED arch/arm/boot/compressed/ashldi3.S
  AS      arch/arm/boot/compressed/lib1funcs.o
  AS      arch/arm/boot/compressed/ashldi3.o
  AS      arch/arm/boot/compressed/piggy.xzkern.o
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
  Kernel: arch/arm/boot/zImage is ready
  CAT     arch/arm/boot/zImage-dtb
  Kernel: arch/arm/boot/zImage-dtb is ready

I solved it by adding coreutils to makedepends= in the kernel and now I am also able to flash the kernel.

As a last note, I suspect that my device admits both appended dtb to zImage and to the boot image.

Member

PabloCastellano commented Nov 18, 2017

So finally I found the issue with XZ compression in my kernel 🎉

It turns out that the kernel makefile is broken because it uses a very weak method to detect if you are using the non-gnu version fo the stat command and it fails if you are using busybox's stat, which is the case in Alpine by default. The message in the log is cryptic and hard to spot because it doesn't fail hardly and the zImage probably remains corrupt somehow:

  LD      vmlinux.o
  MODPOST vmlinux.o
  GEN     .version
  CHK     include/generated/compile.h
  LD      .tmp_vmlinux1
  KSYM    .tmp_kallsyms1.S
  AS      .tmp_kallsyms1.o
  LD      .tmp_vmlinux2
  KSYM    .tmp_kallsyms2.S
  AS      .tmp_kallsyms2.o
  LD      vmlinux
  SYSMAP  System.map
  SYSMAP  .tmp_System.map
  OBJCOPY arch/arm/boot/Image
  Kernel: arch/arm/boot/Image is ready
  AS      arch/arm/boot/compressed/head.o
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
  XZKERN  arch/arm/boot/compressed/piggy.xzkern
  CC      arch/arm/boot/compressed/misc.o
  CC      arch/arm/boot/compressed/decompress.o
  CC      arch/arm/boot/compressed/string.o
  SHIPPED arch/arm/boot/compressed/lib1funcs.S
  SHIPPED arch/arm/boot/compressed/ashldi3.S
  AS      arch/arm/boot/compressed/lib1funcs.o
  AS      arch/arm/boot/compressed/ashldi3.o
  AS      arch/arm/boot/compressed/piggy.xzkern.o
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
  Kernel: arch/arm/boot/zImage is ready
  CAT     arch/arm/boot/zImage-dtb
  Kernel: arch/arm/boot/zImage-dtb is ready

I solved it by adding coreutils to makedepends= in the kernel and now I am also able to flash the kernel.

As a last note, I suspect that my device admits both appended dtb to zImage and to the boot image.

@ollieparanoid

This comment has been minimized.

Show comment
Hide comment
@ollieparanoid

ollieparanoid Nov 18, 2017

Member

Awesome news! \o/

It might be enough to depend on the xz packages though, I guess coreutils pulls that in (altough I couldn't see it directly), because xz is only provided by xz (and busybox, but that isn't listed as it installs the links dynamically).

Member

ollieparanoid commented Nov 18, 2017

Awesome news! \o/

It might be enough to depend on the xz packages though, I guess coreutils pulls that in (altough I couldn't see it directly), because xz is only provided by xz (and busybox, but that isn't listed as it installs the links dynamically).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.