Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

[geissonator]

This popped up on a test run of our latest master this morning.

[    4.290000] input: gpio-keys-polled as /devices/platform/gpio-keys-polled/input/input1
[    4.320000] ubi0: default fastmap pool size: 25
[    4.320000] ubi0: default fastmap WL pool size: 12
[    4.320000] ubi0: attaching mtd3
[    4.330000] ubi0: scanning is finished
[    4.350000] ubi0: attached mtd3 (name "obmc-ubi", size 31 MiB)
[    4.350000] ubi0: PEB size: 65536 bytes (64 KiB), LEB size: 65408 bytes
[    4.350000] ubi0: min./max. I/O unit sizes: 1/256, sub-page size 1
[    4.380000] ubi0: VID header offset: 64 (aligned 64), data offset: 128
[    4.380000] ubi0: good PEBs: 504, bad PEBs: 0, corrupted PEBs: 0
[    4.380000] ubi0: user volume: 4, internal volumes: 1, max. volumes count: 128
[    4.380000] ubi0: max/mean erase counter: 150/120, WL threshold: 4096, image sequence number: 1358235831
[    4.380000] ubi0: available PEBs: 46, total reserved PEBs: 458, PEBs reserved for bad PEB handling: 0
[    4.420000] ubi0: background thread "ubi_bgt0d" started, PID 503
[    4.420000] ubi4: default fastmap pool size: 25
[    4.420000] ubi4: default fastmap WL pool size: 12
[    4.420000] ubi4: attaching mtd7
[    4.460000] ubi4: scanning is finished
[    4.510000] ubi4: attached mtd7 (name "alt-obmc-ubi", size 31 MiB)
[    4.510000] ubi4: PEB size: 65536 bytes (64 KiB), LEB size: 65408 bytes
[    4.510000] ubi4: min./max. I/O unit sizes: 1/256, sub-page size 1
[    4.510000] ubi4: VID header offset: 64 (aligned 64), data offset: 128
[    4.510000] ubi4: good PEBs: 504, bad PEBs: 0, corrupted PEBs: 0
[    4.510000] ubi4: user volume: 4, internal volumes: 1, max. volumes count: 128
[    4.570000] ubi4: max/mean erase counter: 119/95, WL threshold: 4096, image sequence number: 1358235831
[    4.570000] ubi4: available PEBs: 46, total reserved PEBs: 458, PEBs reserved for bad PEB handling: 0
[    4.570000] ubi4: background thread "ubi_bgt4d" started, PID 509
[    7.450000] block ubiblock0_0: created from ubi0:0(rofs-ea65c77f)
[    7.450000] console [netcon0] enabled
[    7.450000] netconsole: network logging started
[    7.450000] input: gpio-keys as /devices/platform/gpio-keys/input/input2
[    7.450000] rtc-rv8803 11-0032: setting system clock to 2017-10-14 13:46:09 UTC (1507988769)
[    7.450000] VFS: Cannot open root device "ubiblock4_0" or unknown-block(0,0): error -6
[    7.450000] Please append a correct "root=" boot option; here are the available partitions:
[    7.450000] 0100            4096 ram0 
[    7.450000]  (driver?)
[    7.450000] 0101            4096 ram1 
[    7.450000]  (driver?)
[    7.520000] 0102            4096 ram2 
[    7.520000]  (driver?)
[    7.520000] 0103            4096 ram3 
[    7.520000]  (driver?)
[    7.520000] 0104            4096 ram4 
[    7.520000]  (driver?)
[    7.520000] 0105            4096 ram5 
[    7.520000]  (driver?)
[    7.520000] 0106            4096 ram6 
[    7.520000]  (driver?)
[    7.520000] 0107            4096 ram7 
[    7.520000]  (driver?)
[    7.520000] 0108            4096 ram8 
[    7.520000]  (driver?)
[    7.570000] 0109            4096 ram9 
[    7.570000]  (driver?)
[    7.570000] 010a            4096 ram10 
[    7.570000]  (driver?)
[    7.580000] 010b            4096 ram11 
[    7.580000]  (driver?)
[    7.580000] 010c            4096 ram12 
[    7.580000]  (driver?)
[    7.580000] 010d            4096 ram13 
[    7.580000]  (driver?)
[    7.580000] 010e            4096 ram14 
[    7.580000]  (driver?)
[    7.600000] 010f            4096 ram15 
[    7.600000]  (driver?)
[    7.600000] 1f00           32768 mtdblock0 
[    7.600000]  (driver?)
[    7.600000] 1f01             384 mtdblock1 
[    7.600000]  (driver?)
[    7.600000] 1f02             128 mtdblock2 
[    7.600000]  (driver?)
[    7.630000] 1f03           32256 mtdblock3 
[    7.630000]  (driver?)
[    7.630000] 1f04           32768 mtdblock4 
[    7.630000]  (driver?)
[    7.630000] 1f05             384 mtdblock5 
[    7.630000]  (driver?)
[    7.630000] 1f06             128 mtdblock6 
[    7.630000]  (driver?)
[    7.630000] 1f07           32256 mtdblock7 
[    7.630000]  (driver?)
[    7.630000] 1f08          131072 mtdblock8 
[    7.630000]  (driver?)
[    7.670000] fe00           14660 ubiblock0_0 
[    7.670000]  (driver?)
[    7.670000] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    7.670000] CPU: 0 PID: 1 Comm: swapper Not tainted 4.10.17-97988a39b46d73304a5697b8d3a5cf08da287350 #1
[    7.670000] Hardware name: ASpeed SoC
[    7.670000] [<80010080>] (unwind_backtrace) from [<8000d934>] (show_stack+0x20/0x24)
[    7.670000] [<8000d934>] (show_stack) from [<801dbf2c>] (dump_stack+0x20/0x28)
[    7.670000] [<801dbf2c>] (dump_stack) from [<800a4a40>] (panic+0xc8/0x264)
[    7.670000] [<800a4a40>] (panic) from [<805884f8>] (mount_block_root+0x214/0x300)
[    7.670000] [<805884f8>] (mount_block_root) from [<80588740>] (mount_root+0x60/0x7c)
[    7.670000] [<80588740>] (mount_root) from [<805888c8>] (prepare_namespace+0x16c/0x1d0)
[    7.670000] [<805888c8>] (prepare_namespace) from [<805880b0>] (kernel_init_freeable+0x184/0x1cc)
[    7.670000] [<805880b0>] (kernel_init_freeable) from [<803fdb40>] (kernel_init+0x18/0x104)
[    7.670000] [<803fdb40>] (kernel_init) from [<8000a5e8>] (ret_from_fork+0x14/0x2c)

It's based on the a4f4a03 level of openbmc/openbmc

ast# env print
02009f36=2
050d0894=2
091e83f4=1
0fb99c13=2
10f7eafa=2
12dabbba=2
13916e85=2
14821299=2
182021a2=2
1835c7a7=2
18dfc7e8=2
21912c1a=2
22d52c4d=2
270096c5=2
2b498149=2
2b62d6a9=2
2bb17531=2
2d6c8c7b=2
2e77ee54=2
30a69178=2
34d35caa=2
35b0d283=2
36f783a0=2
384ba07f=2
39d0fe95=2
3a085f99=2
3a3217e5=2
3d0f9df4=2
3d26cc2e=2
403b617c=2
4202d34c=2
44e20ca6=2
45ab6f88=2
4898b71a=2
4b850b88=2
4c11050f=2
4ddcd098=2
500b5c26=2
50c74b8d=2
51371b39=2
537cb1b1=2
53c3935d=2
57ab565b=2
58493a9a=2
59ca2c90=2
5a239cd5=2
5aad71e0=2
5c05d9ed=2
5d365258=2
5d86d7c1=2
5f818f7d=2
5feb7448=2
61dee88f=2
62e939a5=2
671ce696=2
6af2ee04=2
6d7a1c9d=2
70527030=2
710b5425=2
71665137=2
7305cfb6=2
754aca46=2
75d0a53f=2
763dce9b=2
77ba5d5c=2
7d523cdc=2
7ddfa857=2
7eee2367=2
7fdbf1b8=2
83a98bc2=2
85e7c8d2=2
863d431b=2
86c81e5c=2
886a170c=2
8ad49792=2
8c1268a4=2
90ed9b65=2
94ca9e8e=2
959f396d=2
95f2cddf=2
9657814d=2
96a2bd2f=2
96b5b9fb=2
97a76838=2
98a396fc=2
99122781=2
9bd2404b=2
9ce8b737=2
9e0e2f74=2
9e35c989=2
9e6cd79c=2
9f695d68=2
a046528b=2
a1a1a015=2
a3a5c9f5=2
a3cb2c43=2
a4f9965c=2
a5039c89=2
a58ee214=2
a61de300=2
a73594b6=2
ab368874=2
abc871a5=2
ac1119bf=2
ade83346=2
af631876=2
b07ad00d=2
b4b33462=2
bacdfd1c=2
baudrate=115200
bbc0213e=2
bc946f32=2
bootargs=console=ttyS4,115200n8 ubi.mtd=obmc-ubi,0,0,0 ubi.mtd=alt-obmc-ubi,0,0,4 ro rootfstype=squashfs
bootcmd=run set_bootargs; run obmc_bootcmd
bootdelay=2
c18554be=2
ca5b1ad6=2
cb0fdb=2
cb7e0667=2
cc7d96cb=2
cd2d97bd=2
cd522902=2
d283a568=2
d32f00d8=2
d4569315=2
d5e429e6=2
d63ca77a=2
d7bcdb37=2
d8cddfb8=2
d8d8ca3e=2
d91be30f=2
d9364d88=2
dc44e837=2
dc7f534d=2
ddd9bf3a=2
de71580b=2
e2ea522a=2
e829a5df=2
e91b2019=2
ea65c77f=0
eb8fdf9e=2
ethact=aspeednic#0
ethaddr=70:e2:84:14:23:e1
f015cd64=2
f2f4b194=2
f4a313fe=2
f4c30935=2
f5758428=2
f699dfe5=2
f85f21f8=2
f872c741=2
f90d8d3a=2
fccb9e84=2
fe9bfb3a=2
kernelname=kernel-ea65c77f
loadaddr=80800000
mtddevname=u-boot
mtddevnum=0
mtdids=nor0=bmc
mtdparts=mtdparts=bmc:384k(u-boot),128k(u-boot-env),-(obmc-ubi)
obmc_bootcmd=ubi part obmc-ubi; ubi read ${loadaddr} ${kernelname}; bootm ${loadaddr} 
partition=nor0,0
root=/dev/ubiblock4_0
set_bootargs=setenv bootargs console=ttyS4,115200n8 ubi.mtd=obmc-ubi,0,0,0 ubi.mtd=alt-obmc-ubi,0,0,4 ro rootfstype=squashfs ubi.block=${ubiblock} root=${root} 
spi_dma=yes
stderr=serial
stdin=serial
stdout=serial
ubiblock=0,0
verify=yes

Environment size: 2707/65531 bytes

[geissonator]

From @saqibkh: "Not sure how, but wrong ubiblock was selected in uboot."

[geissonator]

I did notice that /var was full after Saqib fixed the system, if that's maybe related.

[geissonator]

We've had an assortment of fixes in this area in recent weeks. I have not seen this since, so closing out.