RPI2 does not boot anymore (using latest 4.4 or 4.1)

[drtyhlpr]

Hi,
thanks for this great kernel github, thanks for your time and work!

I created a debian-jessie rpi2 bootstrapping script that was running fine (on latest script release date) - 22 days ago. Yesterday I found out that the created Images (with compiled kernel) do not boot anymore. I tried to investigate whats the problem - but failed so far.

Uncompressing :Linux... done, booting the kernel.

After the "uncompressing kernel" message I don't get any more messages and I struggle to find out more about the problem. Is there any other debugging option except using JTAG to debug this early boot step? (the only general debugging options I found dtdebug=1 needs userland to see debugging logs)

The kernel compilation works fine without errors. I am using default bcm2709_defconfig (classic kernel make and install (zImage modules dtbs - modules_install, firmware_install, headers_install), finally the copy ./boot/dts and overlays to /boot/).

Of course I am using the latest boot binaries from firmware github - dts and overlays look very similar.

Is there any major change inside the /boot directory or the dtbs, overlays structure - that I miss - or I am to blind to see?

here is the bootstrapping script (I am not posting this as PR, lol):
https://github.com/drtyhlpr/rpi2-gen-image
(-> https://github.com/drtyhlpr/rpi2-gen-image/blob/master/bootstrap.d/13-kernel.sh)

Listing of my generated boot directory (compiled kernel, copied dtbs and overlays, latest rpi firmware files):

/mnt/bcm2709-rpi-2-b.dtb
/mnt/bcm2710-rpi-3-b.dtb
/mnt/bootcode.bin
/mnt/cmdline.txt
/mnt/config.txt
/mnt/fixup_cd.dat
/mnt/fixup.dat
/mnt/fixup_x.dat
/mnt/kernel7.img
/mnt/start_cd.elf
/mnt/start.elf
/mnt/start_x.elf

/mnt/overlays:
ads7846.dtbo
akkordion-iqdacplus.dtbo
at86rf233.dtbo
bmp085_i2c-sensor.dtbo
boomberry-dac.dtbo
boomberry-digi.dtbo
dht11.dtbo
dpi24.dtbo
dwc2.dtbo
dwc-otg.dtbo
enc28j60.dtbo
gpio-ir.dtbo
gpio-poweroff.dtbo
hifiberry-amp.dtbo
hifiberry-dac.dtbo
hifiberry-dacplus.dtbo
hifiberry-digi.dtbo
hy28a.dtbo
hy28b.dtbo
i2c0-bcm2708.dtbo
i2c1-bcm2708.dtbo
i2c-gpio.dtbo
i2c-mux-pca9548a.dtbo
i2c-pwm-pca9685a.dtbo
i2c-rtc.dtbo
i2s-mmap.dtbo
iqaudio-dac.dtbo
iqaudio-dacplus.dtbo
lirc-rpi.dtbo
mcp2515-can0.dtbo
mcp2515-can1.dtbo
mmc.dtbo
mz61581.dtbo
pi3-act-led.dtbo
pi3-disable-bt.dtbo
pi3-miniuart-bt.dtbo
piscreen2r.dtbo
piscreen.dtbo
pitft28-capacitive.dtbo
pitft28-resistive.dtbo
pps-gpio.dtbo
pwm-2chan.dtbo
pwm.dtbo
qca7000.dtbo
raspidac3.dtbo
README
rpi-backlight.dtbo
rpi-dac.dtbo
rpi-display.dtbo
rpi-ft5406.dtbo
rpi-proto.dtbo
rpi-sense.dtbo
sdhost.dtbo
sdio.dtbo
sdtweak.dtbo
smi-dev.dtbo
smi.dtbo
smi-nand.dtbo
spi1-1cs.dtbo
spi1-2cs.dtbo
spi1-3cs.dtbo
spi2-1cs.dtbo
spi2-2cs.dtbo
spi2-3cs.dtbo
spi-gpio35-39.dtbo
tinylcd35.dtbo
uart1.dtbo
vc4-kms-v3d.dtbo
vga666.dtbo
w1-gpio.dtbo
w1-gpio-pullup.dtbo
wittypi.dtbo

thanks for your help, If you need more output or maybe have ideas to generate more debugging output please tell me

ps. I also tried the 4.1.y kernel (fetching -b 4.1.y and using firmware "kernel: bump to 4.1.21") -> same problem

[popcornmix]

Did you build kernel using these instructions: https://www.raspberrypi.org/documentation/linux/kernel/building.md ? Including using mkknlimg?

[drtyhlpr]

yes the kernel was built like shown in the CROSS-COMPILING section of these instructions. I did not use mkknlimg (lately) because the zImage was definitely working fine. I will try to convert it like mentioned- thanks for the hint

[drtyhlpr]

YES! thank you so mutch popcornmix: after using mkknlimg the system boots (again). A little bit strange because RPi2 was booting fine with zImage 22 days ago.

[anyc]

Does someone have an idea what could be wrong if I can boot 4.1.21 but not 4.4.7? I am using Uboot and I can see the device tree in Uboot and also pass it to the kernel.

[popcornmix]

Can you post your config.txt to start with?

[anyc]

grep -v "^#\|^$" config.txt 
disable_overscan=1
hdmi_force_hotplug=1
dtoverlay=vc4-kms-v3d
mask_gpu_interrupt0=0x400
avoid_warnings=2
hdmi_drive=2
hdmi_group=1
hdmi_mode=16
device_tree_param=i2c_arm=on
device_tree_address=0x00000100
kernel=uboot.bin

In Uboot (tried 2016.01 and 2016.05-rc1) I can examine the device tree with the fdt command and the overlays also worked with older kernels.

I use the RPi sources with Ubuntu patches and modify the default rpi_2_defconfig with:

sed -i "s/CONFIG_BTRFS_FS=m/CONFIG_BTRFS_FS=y/" .config
sed -i "s/CONFIG_DRM_VC4=.*/CONFIG_DRM_VC4=y/" .config
sed -i "s/CONFIG_I2C_BCM2835=.*/CONFIG_I2C_BCM2835=y/" .config
sed -i "s/CONFIG_FB_BCM2708=.*/CONFIG_FB_BCM2708=n/" .config
sed -i "s/CONFIG_LOCALVERSION=.*/CONFIG_LOCALVERSION=/" .config

to use btrfs as root fs and the vc4 module.

The serial console shows me:

reading vmlinuz-4.4.7-rpi-vc4
4919556 bytes read in 750 ms (6.3 MiB/s)
Kernel image @ 0x1000000 [ 0x000000 - 0x4b1040 ]
## Flattened Device Tree blob at 00000100
   Booting using the fdt blob at 0x000100
   Using Device Tree in place at 00000100, end 000071a1

Starting kernel ...

Uncompressing Linux... done, booting the kernel.

and then the RPi2 turns off after a few seconds (what it never did before).

I admit that I was experimenting a lot with Uboot and the kernel config to learn but right now everything should be "vanilla". It's likely something stupid I did and I tried different things but without some sort of error message I'm a bit stuck right now.

[pelwell]

The problem is likely to be that the firmware thinks the device tree won't fit. The default value for device_tree_end is 0x4000 (because the Linux kernel or its decompressor uses the memory between 0x4000 and 0x8000, but the current dtbs are larger than that. Try setting device_tree_end=0x8000.

[popcornmix]

You can remove mask_gpu_interrupt0=0x400 - dtoverlay=vc4-kms-v3d will set that itself.

[anyc]

Unfortunately, both changes did not help. I removed the kernel=uboot.bin line to boot the kernel directly but it also stops after Uncompressing Linux... done, booting the kernel. and the HDMI display shows the colored box. I will try it again tomorrow after I got some sleep. Thank you!

[pelwell]

You may also want to update to today's firmware release, since that made a few changes to the DT handling that may be significant.

[anyc]

I started from scratch today and now it works if I set device_tree_address=0x17fdd200. 0x100 did not work with or without device_tree_end. I chose that address based on another number if found in a thread.

I guess this will work until uboot can dynamically calculate the address. Looks like swarren is already working on this: swarren/u-boot@94f80bc
Thank you!