Kernel panic on Pi 1 B(+) when using an initramfs with DeviceTree enabled

[diederikdehaas]

I'm the maintainer of raspbian-ua-netinst and we've had several reports about kernel panics on the Pi 1 B/B+ (see debian-pi/raspbian-ua-netinst#199 and debian-pi/raspbian-ua-netinst#201).
By default it's using the raspbian.org kernel and an initramfs and I have multiple times 'fixed' the issue by disabling the initramfs (which is generated/used by default with the raspbian.org kernel).

But that kernel is not (exactly) the same as the kernel build from this repo.
Plugwash told me on IRC that he was able to get a kernel panic by using an initramfs with a rpf kernel (this repo), so I tried whether I could accomplish the same ... and I have.

Procedure:

• install a basic system using the raspbian-ua-netinst and reboot
• this may or may not result in a kernel panic right away
• assuming it boots correctly, log in to the system with user root and password raspbian (the default)
• install the raspberrypi-bootloader package to get the rpf kernel (it will uninstall some other packages)
• reboot into the new kernel
• generate an initramfs by executing /etc/kernel/postinst.d/initramfs-tools $(uname -r)
• modify /boot/config.txt so it contains the line initramfs initrd.img-3.18.7+ followkernel (I have it under a [pi1] header)
• reboot and see it has a kernel panic with the rpf kernel/initramfs too

If it doesn't give a kernel panic, re-generating the initramfs with the earlier mentioned command often/sometimes gives a kernel panic on reboot. The 'problem' is that it doesn't consistently fail.

Log from start to kernel panic (which is also stored here, more dumps in debian-pi/raspbian-ua-netinst#199):

Uncompressing Linux... done, booting the kernel.
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.18.7+ (dc4@dc4-XPS13-9333) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03) ) #755 PREEMPT Thu Feb 12 17:14:31 GMT 2015
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] Machine model: Raspberry Pi Model B+
[    0.000000] cma: Reserved 8 MiB at 0x1b400000
[    0.000000] Memory policy: Data cache writeback
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 113792
[    0.000000] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708.boardrev=0x10 bcm2708.serial=0x9dee0f68 smsc95xx.macaddr=B8:27:EB:EE:0F:68 bcm2708_fb.fbswap=1 bcm2708.disk_led_gpio=47 bcm2708.disk_led_active_low=0 sdhci-bcm2708.emmc_clock_freq=250000000 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 console=tty1 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 elevator=deadline root=/dev/mmcblk0p2 rootfstype=ext4 rootwait
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 433896K/458752K available (5926K kernel code, 358K rwdata, 1876K rodata, 340K init, 734K bss, 24856K reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xffe00000   (2048 kB)
[    0.000000]     vmalloc : 0xdc800000 - 0xff000000   ( 552 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xdc000000   ( 448 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc07a6ad8   (7803 kB)
[    0.000000]       .init : 0xc07a7000 - 0xc07fc000   ( 340 kB)
[    0.000000]       .data : 0xc07fc000 - 0xc085588c   ( 359 kB)
[    0.000000]        .bss : 0xc085588c - 0xc090d128   ( 735 kB)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] NR_IRQS:522
[    0.000026] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483648000ns
[    0.000076] Switching to timer-based delay loop, resolution 1000ns
[    0.000356] Console: colour dummy device 80x30
[    0.001441] console [tty1] enabled
[    0.001487] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.001560] pid_max: default: 32768 minimum: 301
[    0.001941] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.002008] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.003018] Initializing cgroup subsys memory
[    0.003112] Initializing cgroup subsys devices
[    0.003173] Initializing cgroup subsys freezer
[    0.003227] Initializing cgroup subsys net_cls
[    0.003277] Initializing cgroup subsys blkio
[    0.003402] CPU: Testing write buffer coherency: ok
[    0.003516] ftrace: allocating 19479 entries in 58 pages
[    0.111593] Setting up static identity map for 0x55d058 - 0x55d0b4
[    0.114413] devtmpfs: initialized
[    0.131909] VFP support v0.3: implementor 41 architecture 1 part 20 variant b rev 5
[    0.135104] pinctrl core: initialized pinctrl subsystem
[    0.137859] NET: Registered protocol family 16
[    0.143394] DMA: preallocated 4096 KiB pool for atomic coherent allocations
[    0.171363] cpuidle: using governor ladder
[    0.201423] cpuidle: using governor menu
[    0.201970] bcm2708.uart_clock = 3000000
[    0.205068] No ATAGs?
[    0.205134] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[    0.205196] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.205263] mailbox: Broadcom VideoCore Mailbox driver
[    0.205426] bcm2708_vcio: mailbox at f200b880
[    0.205912] bcm_power: Broadcom power driver
[    0.205969] bcm_power_open() -> 0
[    0.205999] bcm_power_request(0, 8)
[    0.706737] bcm_mailbox_read -> 00000080, 0
[    0.706785] bcm_power_request -> 0
[    0.706983] Serial: AMBA PL011 UART driver
[    0.707218] dev:f1: ttyAMA0 at MMIO 0x20201000 (irq = 83, base_baud = 0) is a PL011 rev3
[    1.103493] console [ttyAMA0] enabled
[    1.176537] SCSI subsystem initialized
[    1.180693] usbcore: registered new interface driver usbfs
[    1.186483] usbcore: registered new interface driver hub
[    1.191983] usbcore: registered new device driver usb
[    1.199199] Switched to clocksource stc
[    1.233655] FS-Cache: Loaded
[    1.236988] CacheFiles: Loaded
[    1.257144] NET: Registered protocol family 2
[    1.263103] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    1.270518] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
[    1.277078] TCP: Hash tables configured (established 4096 bind 4096)
[    1.283588] TCP: reno registered
[    1.286855] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    1.292779] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    1.299518] NET: Registered protocol family 1
[    1.304526] RPC: Registered named UNIX socket transport module.
[    1.310613] RPC: Registered udp transport module.
[    1.315343] RPC: Registered tcp transport module.
[    1.320108] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.327010] Trying to unpack rootfs image as initramfs...
[    1.757137] Freeing initrd memory: 3316K (dbcb3000 - dbff0000)
[    1.775919] bcm2708_dma: DMA manager at f2007000
[    1.781020] vc-mem: phys_addr:0x00000000 mem_base=0x1ec00000 mem_size:0x20000000(512 MiB)
[    1.791016] futex hash table entries: 256 (order: -1, 3072 bytes)
[    1.797287] audit: initializing netlink subsys (disabled)
[    1.802972] audit: type=2000 audit(1.550:1): initialized
[    1.824051] VFS: Disk quotas dquot_6.5.2
[    1.828410] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    1.837977] FS-Cache: Netfs 'nfs' registered for caching
[    1.845172] NFS: Registering the id_resolver key type
[    1.850497] Key type id_resolver registered
[    1.854711] Key type id_legacy registered
[    1.860292] msgmni has been set to 869
[    1.866554] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    1.874562] io scheduler noop registered
[    1.878546] io scheduler deadline registered (default)
[    1.884282] io scheduler cfq registered
[    1.888387] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[    1.896724] pgd = c0004000
[    1.899576] [00000004] *pgd=00000000
[    1.903197] Internal error: Oops: 5 [#1] PREEMPT ARM
[    1.908179] Modules linked in:
[    1.911260] CPU: 0 PID: 1 Comm: swapper Not tainted 3.18.7+ #755
[    1.917284] task: dac88000 ti: dac4c000 task.ti: dac4c000
[    1.922711] PC is at strcmp+0x18/0x40
[    1.926400] LR is at of_bus_isa_match+0x20/0x30
[    1.930949] pc : [<c02f9954>]    lr : [<c045af54>]    psr: a0000013
[    1.930949] sp : dac4dc48  ip : dac4dc58  fp : dac4dc54
[    1.942450] r10: c08469b8  r9 : 00000000  r8 : dbff06f0
[    1.947687] r7 : 00000200  r6 : c00007a0  r5 : dbfef468  r4 : c084699c
[    1.954228] r3 : c045af34  r2 : dbfef468  r1 : c06f04a8  r0 : 00000004
[    1.960768] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[    1.968095] Control: 00c5387d  Table: 1ade8008  DAC: 00000015
[    1.973854] Process swapper (pid: 1, stack limit = 0xdac4c1b0)
[    1.979714] Stack: (0xdac4dc48 to 0xdac4e000)
[    1.984092] dc40:                   dac4dc64 dac4dc58 c045af54 c02f9948 dac4dc7c dac4dc68
[    1.992298] dc60: c045ad94 c045af40 dbff0328 dbff06f0 dac4dcfc dac4dc80 c045b310 c045ad74
[    2.000503] dc80: dac4dcac dac4dc90 c0456c44 c02f9948 dac4c000 dac4c030 a0000013 c06f050c
[    2.008708] dca0: dbfef468 dac4dcb0 c0456d18 00000001 00000001 00000000 c00007a0 dac4dd0c
[    2.016912] dcc0: 0000207e 00000000 dac4dcfc dac4dcd8 c045797c 00000000 c00007a0 dac4dd6c
[    2.025117] dce0: 00000200 dbff06f0 00000000 000000b4 dac4dd44 dac4dd00 c045ba14 c045b27c
[    2.033322] dd00: 00000000 c00c46bc 00000200 00000000 000000b4 00000000 dac4dd44 da225010
[    2.041527] dd20: dad1ae00 c0837650 c0837650 00000000 da21e420 dad1ae10 dac4ddb4 dac4dd48
[    2.049733] dd40: c0315d84 c045b9a4 da224050 dad1b870 dac4dd7c da224050 da2240f0 dad1b870
[    2.057939] dd60: dbff06f0 00000001 da21e420 00000000 dac4dda4 dac4dd80 c01ad9b4 c01a9fc4
[    2.066143] dd80: 00000001 dad1ae10 dad1ae18 dad1ae10 fffffdfb c0837650 c0837650 00000000
[    2.074348] dda0: da21e420 00000000 dac4ddd4 dac4ddb8 c0367698 c0315d38 c08e4d88 dad1ae10
[    2.082554] ddc0: 00000000 c0837650 dac4ddfc dac4ddd8 c03656e8 c0367650 00000007 dad1ae10
[    2.090760] dde0: c0837650 dad1ae44 00000000 c07df1f4 dac4de1c dac4de00 c0365988 c036566c
[    2.098965] de00: c03658e0 00000000 c0837650 c03658e0 dac4de44 dac4de20 c0363a84 c03658ec
[    2.107170] de20: dac2370c dad07c90 da21c5d4 c0837650 da21c5a0 c083c310 dac4de54 dac4de48
[    2.115376] de40: c0365274 c0363a20 dac4de7c dac4de58 c0364ef8 c0365258 c06cd688 dac4de68
[    2.123599] de60: c0837650 c08077f0 c07ce10c dac4c038 dac4de94 dac4de80 c0366328 c0364db0
[    2.131804] de80: c08077f0 c08077f0 dac4dea4 dac4de98 c03675b8 c03662ac dac4deb4 dac4dea8
[    2.140010] dea0: c07ce124 c036756c dac4df4c dac4deb8 c0008760 c07ce118 dad076c0 c05608a4
[    2.148214] dec0: 00000189 00000000 00000000 00000000 dac4defc c004f670 dac4df04 dac4dee8
[    2.156419] dee0: dac4df00 dac4def0 c07a75e8 dbfffbe5 c05a26b8 000000e8 dac4df4c dac4df08
[    2.164623] df00: c003e650 c07a75f4 00000000 00000006 00000006 dbfffbed c07312c4 00000000
[    2.172828] df20: 00000000 c07fb248 00000006 c08558a0 c08558a0 c07df1f4 000000e8 c07df200
[    2.181034] df40: dac4df94 dac4df50 c07a7e08 c00086dc 00000006 00000006 c07a75e8 c08295f0
[    2.189239] df60: dac4c038 dac88d80 dac4df94 00000000 c05513b8 00000000 00000000 00000000
[    2.197444] df80: 00000000 00000000 dac4dfac dac4df98 c05513d0 c07a7d1c dac4c000 00000000
[    2.205648] dfa0: 00000000 dac4dfb0 c000e988 c05513c4 00000000 00000000 00000000 00000000
[    2.213852] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.222055] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[    2.230284] [<c02f9954>] (strcmp) from [<c045af54>] (of_bus_isa_match+0x20/0x30)
[    2.237719] [<c045af54>] (of_bus_isa_match) from [<c045ad94>] (of_match_bus+0x2c/0x70)
[    2.245675] [<c045ad94>] (of_match_bus) from [<c045b310>] (__of_translate_address+0xa0/0x294)
[    2.254239] [<c045b310>] (__of_translate_address) from [<c045ba14>] (of_address_to_resource+0x7c/0x100)
[    2.263683] [<c045ba14>] (of_address_to_resource) from [<c0315d84>] (bcm2835_pinctrl_probe+0x58/0x400)
[    2.273049] [<c0315d84>] (bcm2835_pinctrl_probe) from [<c0367698>] (platform_drv_probe+0x54/0xb4)
[    2.281969] [<c0367698>] (platform_drv_probe) from [<c03656e8>] (really_probe+0x88/0x21c)
[    2.290187] [<c03656e8>] (really_probe) from [<c0365988>] (__driver_attach+0xa8/0xac)
[    2.298056] [<c0365988>] (__driver_attach) from [<c0363a84>] (bus_for_each_dev+0x70/0xa4)
[    2.306270] [<c0363a84>] (bus_for_each_dev) from [<c0365274>] (driver_attach+0x28/0x30)
[    2.314311] [<c0365274>] (driver_attach) from [<c0364ef8>] (bus_add_driver+0x154/0x200)
[    2.322354] [<c0364ef8>] (bus_add_driver) from [<c0366328>] (driver_register+0x88/0x108)
[    2.330484] [<c0366328>] (driver_register) from [<c03675b8>] (__platform_driver_register+0x58/0x6c)
[    2.339573] [<c03675b8>] (__platform_driver_register) from [<c07ce124>] (bcm2835_pinctrl_driver_init+0x18/0x20)
[    2.349703] [<c07ce124>] (bcm2835_pinctrl_driver_init) from [<c0008760>] (do_one_initcall+0x90/0x1e4)
[    2.358979] [<c0008760>] (do_one_initcall) from [<c07a7e08>] (kernel_init_freeable+0xf8/0x1b8)
[    2.367636] [<c07a7e08>] (kernel_init_freeable) from [<c05513d0>] (kernel_init+0x18/0xfc)
[    2.375860] [<c05513d0>] (kernel_init) from [<c000e988>] (ret_from_fork+0x14/0x20)
[    2.383462] Code: e24cb004 ea000001 e3530000 0a000007 (e4d03001)
[    2.389809] ---[ end trace 4c6abb27342ccfdb ]---
[    2.394525] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.394525]
[    2.403702] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.403702]

[popcornmix]

Can you test if it does it with the latest "rpi-update" firmware/kernel?

[diederikdehaas]

I've managed to also produce this kernel panic starting from the 2015-02-16-raspbian-wheezy.img image:

• Write image to sd card and boot up
• do sudo apt-get update and sudo apt-get dist-upgrade and reboot
• generate initramfs with /etc/kernel/postinst.d/initramfs-tools $(uname -r)
• modify /boot/config.txt so it has the line initramfs initrd.img-3.18.7+ followkernel. This time I did not put it under a [pi1] header
• reboot ... and see a kernel panic

Log:

Uncompressing Linux... done, booting the kernel.
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.18.7+ (dc4@dc4-XPS13-9333) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03) ) #755 PREEMPT Thu Feb 12 17:14:31 GMT 2015
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[    0.000000] Machine model: Raspberry Pi Model B+
[    0.000000] cma: Reserved 8 MiB at 0x1b400000
[    0.000000] Memory policy: Data cache writeback
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 113792
[    0.000000] Kernel command line: dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708.boardrev=0x10 bcm2708.serial=0x9dee0f68 smsc95xx.macaddr=B8:27:EB:EE:0F:68 bcm2708_fb.fbswap=1 bcm2708.disk_led_gpio=47 bcm2708.disk_led_active_low=0 sdhci-bcm2708.emmc_clock_freq=250000000 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 console=tty1 console=ttyAMA0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 433244K/458752K available (5926K kernel code, 358K rwdata, 1876K rodata, 340K init, 734K bss, 25508K reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xffe00000   (2048 kB)
[    0.000000]     vmalloc : 0xdc800000 - 0xff000000   ( 552 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xdc000000   ( 448 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc07a6ad8   (7803 kB)
[    0.000000]       .init : 0xc07a7000 - 0xc07fc000   ( 340 kB)
[    0.000000]       .data : 0xc07fc000 - 0xc085588c   ( 359 kB)
[    0.000000]        .bss : 0xc085588c - 0xc090d128   ( 735 kB)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] NR_IRQS:522
[    0.000024] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483648000ns
[    0.000075] Switching to timer-based delay loop, resolution 1000ns
[    0.000355] Console: colour dummy device 80x30
[    0.001430] console [tty1] enabled
[    0.001476] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.001552] pid_max: default: 32768 minimum: 301
[    0.001932] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.001998] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.002999] Initializing cgroup subsys memory
[    0.003093] Initializing cgroup subsys devices
[    0.003154] Initializing cgroup subsys freezer
[    0.003208] Initializing cgroup subsys net_cls
[    0.003259] Initializing cgroup subsys blkio
[    0.003382] CPU: Testing write buffer coherency: ok
[    0.003496] ftrace: allocating 19479 entries in 58 pages
[    0.111557] Setting up static identity map for 0x55d058 - 0x55d0b4
[    0.114386] devtmpfs: initialized
[    0.131854] VFP support v0.3: implementor 41 architecture 1 part 20 variant b rev 5
[    0.135042] pinctrl core: initialized pinctrl subsystem
[    0.137794] NET: Registered protocol family 16
[    0.143290] DMA: preallocated 4096 KiB pool for atomic coherent allocations
[    0.171337] cpuidle: using governor ladder
[    0.201399] cpuidle: using governor menu
[    0.201916] bcm2708.uart_clock = 3000000
[    0.205005] No ATAGs?
[    0.205071] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[    0.205133] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.205199] mailbox: Broadcom VideoCore Mailbox driver
[    0.205365] bcm2708_vcio: mailbox at f200b880
[    0.205857] bcm_power: Broadcom power driver
[    0.205915] bcm_power_open() -> 0
[    0.205945] bcm_power_request(0, 8)
[    0.706686] bcm_mailbox_read -> 00000080, 0
[    0.706732] bcm_power_request -> 0
[    0.706931] Serial: AMBA PL011 UART driver
[    0.707167] dev:f1: ttyAMA0 at MMIO 0x20201000 (irq = 83, base_baud = 0) is a PL011 rev3
[    1.101538] console [ttyAMA0] enabled
[    1.174598] SCSI subsystem initialized
[    1.178748] usbcore: registered new interface driver usbfs
[    1.184540] usbcore: registered new interface driver hub
[    1.190038] usbcore: registered new device driver usb
[    1.197228] Switched to clocksource stc
[    1.231863] FS-Cache: Loaded
[    1.235200] CacheFiles: Loaded
[    1.255304] NET: Registered protocol family 2
[    1.261264] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    1.268679] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
[    1.275239] TCP: Hash tables configured (established 4096 bind 4096)
[    1.281747] TCP: reno registered
[    1.285014] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    1.290939] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    1.297662] NET: Registered protocol family 1
[    1.302668] RPC: Registered named UNIX socket transport module.
[    1.308758] RPC: Registered udp transport module.
[    1.313488] RPC: Registered tcp transport module.
[    1.318255] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.325157] Trying to unpack rootfs image as initramfs...
[    1.828092] Freeing initrd memory: 3968K (dbc10000 - dbff0000)
[    1.847792] bcm2708_dma: DMA manager at f2007000
[    1.852682] vc-mem: phys_addr:0x00000000 mem_base=0x1ec00000 mem_size:0x20000000(512 MiB)
[    1.862591] futex hash table entries: 256 (order: -1, 3072 bytes)
[    1.869055] audit: initializing netlink subsys (disabled)
[    1.874580] audit: type=2000 audit(1.620:1): initialized
[    1.895689] VFS: Disk quotas dquot_6.5.2
[    1.900268] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    1.909848] FS-Cache: Netfs 'nfs' registered for caching
[    1.916949] NFS: Registering the id_resolver key type
[    1.922284] Key type id_resolver registered
[    1.926498] Key type id_legacy registered
[    1.932035] msgmni has been set to 869
[    1.938478] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    1.946324] io scheduler noop registered
[    1.950485] io scheduler deadline registered (default)
[    1.956067] io scheduler cfq registered
[    1.960354] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[    1.968667] pgd = c0004000
[    1.971400] [00000004] *pgd=00000000
[    1.975008] Internal error: Oops: 5 [#1] PREEMPT ARM
[    1.979987] Modules linked in:
[    1.983067] CPU: 0 PID: 1 Comm: swapper Not tainted 3.18.7+ #755
[    1.989091] task: dac88000 ti: dac4c000 task.ti: dac4c000
[    1.994516] PC is at strcmp+0x18/0x40
[    1.998204] LR is at of_bus_isa_match+0x20/0x30
[    2.002754] pc : [<c02f9954>]    lr : [<c045af54>]    psr: a0000013
[    2.002754] sp : dac4dc48  ip : dac4dc58  fp : dac4dc54
[    2.014254] r10: c08469b8  r9 : 00000000  r8 : dbff06f0
[    2.019491] r7 : 00000200  r6 : c0000788  r5 : dbfef468  r4 : c084699c
[    2.026031] r3 : c045af34  r2 : dbfef468  r1 : c06f04a8  r0 : 00000004
[    2.032572] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[    2.039899] Control: 00c5387d  Table: 1adec008  DAC: 00000015
[    2.045659] Process swapper (pid: 1, stack limit = 0xdac4c1b0)
[    2.051505] Stack: (0xdac4dc48 to 0xdac4e000)
[    2.055883] dc40:                   dac4dc64 dac4dc58 c045af54 c02f9948 dac4dc7c dac4dc68
[    2.064089] dc60: c045ad94 c045af40 dbff0328 dbff06f0 dac4dcfc dac4dc80 c045b310 c045ad74
[    2.072293] dc80: dac4dcac dac4dc90 c0456c44 c02f9948 dac4c000 dac4c030 a0000013 c06f050c
[    2.080498] dca0: dbfef468 dac4dcb0 c0456d18 00000001 00000001 00000000 c0000788 dac4dd0c
[    2.088703] dcc0: 0000207e 00000000 dac4dcfc dac4dcd8 c045797c 00000000 c0000788 dac4dd6c
[    2.096908] dce0: 00000200 dbff06f0 00000000 000000b4 dac4dd44 dac4dd00 c045ba14 c045b27c
[    2.105112] dd00: 00000000 c00c46bc 00000200 00000000 000000b4 00000000 dac4dd44 da37b010
[    2.113317] dd20: dad1ae00 c0837650 c0837650 00000000 da374420 dad1ae10 dac4ddb4 dac4dd48
[    2.121523] dd40: c0315d84 c045b9a4 da37a050 dad1b870 dac4dd7c da37a050 da37a0f0 dad1b870
[    2.129727] dd60: dbff06f0 00000001 da374420 00000000 dac4dda4 dac4dd80 c01ad9b4 c01a9fc4
[    2.137933] dd80: 00000001 dad1ae10 dad1ae18 dad1ae10 fffffdfb c0837650 c0837650 00000000
[    2.146139] dda0: da374420 00000000 dac4ddd4 dac4ddb8 c0367698 c0315d38 c08e4d88 dad1ae10
[    2.154343] ddc0: 00000000 c0837650 dac4ddfc dac4ddd8 c03656e8 c0367650 00000007 dad1ae10
[    2.162549] dde0: c0837650 dad1ae44 00000000 c07df1f4 dac4de1c dac4de00 c0365988 c036566c
[    2.170754] de00: c03658e0 00000000 c0837650 c03658e0 dac4de44 dac4de20 c0363a84 c03658ec
[    2.178960] de20: dac2370c dad07c90 da3725d4 c0837650 da3725a0 c083c310 dac4de54 dac4de48
[    2.187165] de40: c0365274 c0363a20 dac4de7c dac4de58 c0364ef8 c0365258 c06cd688 dac4de68
[    2.195370] de60: c0837650 c08077f0 c07ce10c dac4c038 dac4de94 dac4de80 c0366328 c0364db0
[    2.203576] de80: c08077f0 c08077f0 dac4dea4 dac4de98 c03675b8 c03662ac dac4deb4 dac4dea8
[    2.211782] dea0: c07ce124 c036756c dac4df4c dac4deb8 c0008760 c07ce118 000000e8 dbfffbbe
[    2.219987] dec0: c06e7240 00000000 c0732144 dac4df4c dac4df08 dac4dee8 dac4df04 dac4dee8
[    2.228192] dee0: dac4df00 dac4def0 c07a75e8 dbfffbef c05a26b8 000000e8 dac4df4c dac4df08
[    2.236397] df00: c003e650 c07a75f4 00000000 00000006 00000006 dbfffbf7 c07312c4 00000000
[    2.244602] df20: 00000000 c07fb248 00000006 c08558a0 c08558a0 c07df1f4 000000e8 c07df200
[    2.252807] df40: dac4df94 dac4df50 c07a7e08 c00086dc 00000006 00000006 c07a75e8 c08295f0
[    2.261020] df60: dac4c038 dac88d80 dac4df94 00000000 c05513b8 00000000 00000000 00000000
[    2.269224] df80: 00000000 00000000 dac4dfac dac4df98 c05513d0 c07a7d1c dac4c000 00000000
[    2.277428] dfa0: 00000000 dac4dfb0 c000e988 c05513c4 00000000 00000000 00000000 00000000
[    2.285631] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.293836] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 40904000 42002800
[    2.302064] [<c02f9954>] (strcmp) from [<c045af54>] (of_bus_isa_match+0x20/0x30)
[    2.309503] [<c045af54>] (of_bus_isa_match) from [<c045ad94>] (of_match_bus+0x2c/0x70)
[    2.317460] [<c045ad94>] (of_match_bus) from [<c045b310>] (__of_translate_address+0xa0/0x294)
[    2.326023] [<c045b310>] (__of_translate_address) from [<c045ba14>] (of_address_to_resource+0x7c/0x100)
[    2.335467] [<c045ba14>] (of_address_to_resource) from [<c0315d84>] (bcm2835_pinctrl_probe+0x58/0x400)
[    2.344834] [<c0315d84>] (bcm2835_pinctrl_probe) from [<c0367698>] (platform_drv_probe+0x54/0xb4)
[    2.353753] [<c0367698>] (platform_drv_probe) from [<c03656e8>] (really_probe+0x88/0x21c)
[    2.361971] [<c03656e8>] (really_probe) from [<c0365988>] (__driver_attach+0xa8/0xac)
[    2.369841] [<c0365988>] (__driver_attach) from [<c0363a84>] (bus_for_each_dev+0x70/0xa4)
[    2.378057] [<c0363a84>] (bus_for_each_dev) from [<c0365274>] (driver_attach+0x28/0x30)
[    2.386098] [<c0365274>] (driver_attach) from [<c0364ef8>] (bus_add_driver+0x154/0x200)
[    2.394141] [<c0364ef8>] (bus_add_driver) from [<c0366328>] (driver_register+0x88/0x108)
[    2.402271] [<c0366328>] (driver_register) from [<c03675b8>] (__platform_driver_register+0x58/0x6c)
[    2.411362] [<c03675b8>] (__platform_driver_register) from [<c07ce124>] (bcm2835_pinctrl_driver_init+0x18/0x20)
[    2.421492] [<c07ce124>] (bcm2835_pinctrl_driver_init) from [<c0008760>] (do_one_initcall+0x90/0x1e4)
[    2.430768] [<c0008760>] (do_one_initcall) from [<c07a7e08>] (kernel_init_freeable+0xf8/0x1b8)
[    2.439425] [<c07a7e08>] (kernel_init_freeable) from [<c05513d0>] (kernel_init+0x18/0xfc)
[    2.447654] [<c05513d0>] (kernel_init) from [<c000e988>] (ret_from_fork+0x14/0x20)
[    2.455256] Code: e24cb004 ea000001 e3530000 0a000007 (e4d03001)
[    2.461607] ---[ end trace 682122b6d7e0f900 ]---
[    2.466322] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.466322]
[    2.475500] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.475500]

[diederikdehaas]

Can you test if it does it with the latest "rpi-update" firmware/kernel?

Will do that now.

• It updated the kernel/firmware to commit id 40185a95ac04ffcec406d9e1ef934406d7221939
• rebooted
• generated the initramfs and enabled it in /boot/config.txt as described earlier
• rebooted and did NOT get a kernel panic

This is the 2nd time I got this result (no kernel panic). I'll re-generate the initramfs a couple of more time to see whether I can get a kernel panic again.

[diederikdehaas]

I have now regenerated the initramfs 5 times and I haven't got a kernel panic (yet?).
Often times it needs to fail in the first try, so I'll setup the system from scratch a few more times.
If you want me to test other things and/or commit id, let me know.
But there is a real possibility that 3.18.10+ fixes the kernel panic.

[diederikdehaas]

I'm quite sure the issue is fixed with the 3.18.10+ kernel.
Plugwash has made a kernel which is similar to 3.18.10+ and it's fixed there too.
So, feel free to close this issue.

[diederikdehaas]

I cheered too soon, I have just gotten a kernel crash with with the latest kernel from rpi-update on a Pi 1 B+ (with an initramfs).
I'm now also trying to get it to crash on a Pi 1 B, but the hard thing about this issue is that it doesn't happen consistently.

I have done the installation with the raspbian-ua-netinst with some custom configuration (release=jessie) and a script which is run near the end of the installation which installs and runs rpi-update, generates an initramfs for that kernel and specifies that in /boot/config.txt so that the first boot of the new system is with the rpi-update kernel.

I can provide the configuration files for the installer if you like with which I have gotten the kernel panic.

[diederikdehaas]

And it happened on a Pi 1 B too, this time with release=wheezy

[diederikdehaas]

Shiftplusone/@XECDesign asked for an image which would have the crashing kernel and I have just uploaded one: http://cknow.org/rpi/raspbian-wheezy-rpi1b-kernel-panic.img.tar.gz

I have shrunk the image so it should fit on any 8GB card. You can log into the image with user root and password raspbian. The output defaults to the serial console and if you don't like that just change the config in /boot/cmdline.txt.
The most interesting thing can be found in /boot/config.txt with the following contents:

[pi1]
#kernel=vmlinuz-3.18.0-trunk-rpi
#initramfs initrd.img-3.18.0-trunk-rpi followkernel
kernel=vmlinuz-3.18.10+
initramfs initrd.img-3.18.10+ followkernel
#initramfs initrd.img-3.18.10+-crash followkernel

The 3.18.10+ kernel is installed from rpi-update which ran around 2015-04-04T13:19:00.
As you can see there's also a commented out initramfs line for the 3.18.10+ kernel and as the name suggests, that is the initramfs which crashed. So if you want to see it crash, remove the '#' in front of that line and put a '#' in front of the other. That way you can toggle between a working and a crashing system. Both initramfs files are present in /boot/ so that can be inspected as well.

If you have any further questions, just let me know and I'll answer them as best as I can.

[diederikdehaas]

It took a while and many tries, but there is now also an image from a kernel panic on a RPi 1B+ with Raspbian Jessie as OS version: http://cknow.org/rpi/raspbian-jessie-rpi1b+-kernel-panic.img.tar.gz

Apart from now also having a documented kernel crash on a RPi 1B+ (next to a 1B) another (potential) relevant difference is a newer version (from Jessie) of initramfs-tools used to generate the initramfs.
Other then the above described differences, the images are the same.
Hope this further helps to find the cause and solution for the reported issue.

[pelwell]

Sadly that image is too large for the 8GB cards I have at my disposal:

NOOBS - 15415296 (eb3800) sectors = 7,892,631,552 bytes
Transcend - 15523840 (0xece000) sectors = 7,948,206,080 bytes
but
Image - 15644672 (eeb800) sectors = 8,010,072,064 bytes

So the image is at least 61MB too large.

I'll bring in a larger card tomorrow and see if that works.

[diederikdehaas]

I've made them smaller, so they now they should (really) fit on a 8GB card:
http://cknow.org/rpi/raspbian-wheezy-rpi1b-kernel-panic-6GB.img.tar.gz
http://cknow.org/rpi/raspbian-jessie-rpi1b+-kernel-panic-6GB.img.tar.gz

[pelwell]

I've downloaded both images, and both produce crashes on a B and a B+. Interestingly, they crash in different locations on the two models, but between runs and the two images, all crashes are remarkably consistent.

On the B+, the abridged callstack looks like this:

[    1.931867] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[    1.940006] pgd = c0004000
[    1.942882] [00000000] *pgd=00000000
...
[    1.966102] PC is at strcmp+0x18/0x40
[    1.969792] LR is at of_bus_isa_match+0x20/0x30
[    1.974340] pc : [<c02f8034>]    lr : [<c04512ec>]    psr: a0000013
[    1.974340] sp : da83bc50  ip : da83bc60  fp : da83bc5c
[    1.985842] r10: c083874c  r9 : 00000000  r8 : dbff0444
[    1.991079] r7 : 00000200  r6 : c0000808  r5 : dbfef120  r4 : c0838730
[    1.997619] r3 : c04512cc  r2 : dbfef120  r1 : c06e65a0  r0 : 00000000
...
[    2.273602] [<c02f8034>] (strcmp) from [<c04512ec>] (of_bus_isa_match+0x20/0x30)
[    2.281038] [<c04512ec>] (of_bus_isa_match) from [<c045112c>] (of_match_bus+0x2c/0x70)
[    2.288994] [<c045112c>] (of_match_bus) from [<c04516a8>] (__of_translate_address+0xa0/0x294)
[    2.297557] [<c04516a8>] (__of_translate_address) from [<c0451dac>] (of_address_to_resource+0x7c/0x100)
[    2.306998] [<c0451dac>] (of_address_to_resource) from [<c0314418>] (bcm2835_pinctrl_probe+0x54/0x48c)
[    2.316347] [<c0314418>] (bcm2835_pinctrl_probe) from [<c0369b78>] (platform_drv_probe+0x3c/0x6c)
[    2.325271] [<c0369b78>] (platform_drv_probe) from [<c0368110>] (really_probe+0x88/0x21c)
[    2.333490] [<c0368110>] (really_probe) from [<c03683b0>] (__driver_attach+0xa8/0xac)
[    2.341361] [<c03683b0>] (__driver_attach) from [<c03664bc>] (bus_for_each_dev+0x70/0xa4)

while on a B you get:

[    1.813861] Unable to handle kernel paging request at virtual address 0012a42c
[    1.821107] pgd = c0004000
[    1.823824] [0012a42c] *pgd=00000000
...
[    1.847020] PC is at pcpu_alloc+0x1cc/0x83c
[    1.851220] LR is at pcpu_alloc+0x188/0x83c
[    1.855418] pc : [<c01032cc>]    lr : [<c0103288>]    psr: 00000093
[    1.855418] sp : cac3bce0  ip : cac3bce0  fp : cac3bd34
[    1.866918] r10: c07f93c4  r9 : cac3a000  r8 : 00000000
[    1.872155] r7 : 0000000c  r6 : 00000004  r5 : 00000060  r4 : 0012a420
[    1.878694] r3 : cbfff080  r2 : 00000000  r1 : c07f93c4  r0 : 00000001
...
[    2.113744] [<c01032cc>] (pcpu_alloc) from [<c0103980>] (__alloc_percpu+0x20/0x24)
[    2.121365] [<c0103980>] (__alloc_percpu) from [<c00d91c8>] (perf_pmu_register+0x30/0x2c0)
[    2.129682] [<c00d91c8>] (perf_pmu_register) from [<c0018410>] (armpmu_register+0xa4/0xd4)
[    2.137992] [<c0018410>] (armpmu_register) from [<c0019310>] (cpu_pmu_device_probe+0xf8/0x39c)
[    2.146647] [<c0019310>] (cpu_pmu_device_probe) from [<c0369b78>] (platform_drv_probe+0x3c/0x6c)
[    2.155488] [<c0369b78>] (platform_drv_probe) from [<c0368110>] (really_probe+0x88/0x21c)
[    2.163709] [<c0368110>] (really_probe) from [<c03683b0>] (__driver_attach+0xa8/0xac)
[    2.171581] [<c03683b0>] (__driver_attach) from [<c03664bc>] (bus_for_each_dev+0x70/0xa4)
[    2.179796] [<c03664bc>] (bus_for_each_dev) from [<c0367c9c>] (driver_attach+0x28/0x30)
[    2.187838] [<c0367c9c>] (driver_attach) from [<c0367930>] (bus_add_driver+0x154/0x200)

Getting different failures like this suggests that both code paths stumble across memory corruption of some kind, but that's just a hunch.

By correlating multiple runs it appears that the B+ code is failing after accessing [r0], while the B code is reading [r4 + 0xc]. The next step would be to extract the kernel and see if the source of the two addresses can be determined by disassembly.

[pelwell]

[ The comments above refer to the larger images, after extracting them onto a 32GB card. ]