"UBI init error 22" on some 128MB SPI NOR chips

[anoo1]

Witherspoon-128 is a relatively new OpenBMC system that is the same as witherspoon but with 128MB NOR chips (instead of 32MB). We've been flashing a few systems with the latest master but lately with a new batch of chips which are the same part number as the first ones (Micron MT25QL01GBBB8ESF), u-boot is failing to read the UBI volume where the kernel and rootfs reside:

ast# ubi part obmc-ubi
ubi0: attaching mtd1
UBI init error 22
ast#

These systems can be netbooted, and the kernel and user space are able to mount and read the UBI volume (will attach dmesg logs below), so seems just u-boot is having an issue.

Collected data for comparison, "w81" is a working machine, "r217" is a failing machine so it's booted using netboot.

1. Check both systems are booted from the primary BMC chip:

root@witherspoon-128:~# cat /sys/class/watchdog/watchdog1/bootstatus
0

2. Check the device tree on both:

root@witherspoon-128:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 08000000 00010000 "bmc"
mtd1: 00060000 00010000 "u-boot"
mtd2: 00020000 00010000 "u-boot-env"
mtd3: 07f80000 00010000 "obmc-ubi"
mtd4: 08000000 00010000 "alt-bmc"
mtd5: 00060000 00010000 "alt-u-boot"
mtd6: 00020000 00010000 "alt-u-boot-env"
mtd7: 07f80000 00010000 "alt-obmc-ubi"
mtd8: 08000000 00010000 "pnor"

3. Erase the alt bmc ubi mtd (mtd7), both systems have the same checksum.

root@witherspoon-128:~# flash_eraseall /dev/mtd7

root@witherspoon-128:~# hexdump -C /dev/mtd7
00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
07f80000

root@witherspoon-128:~# md5sum /dev/mtd7
9ef627a57edcdfa4df2c5b413ee77ccb  /dev/mtd7

4. Ubiformat and save off the contents of the volume:

root@witherspoon-128:~# ubiformat /dev/mtd7
ubiformat: mtd7 (nor), size 133693440 bytes (127.5 MiB), 2040 eraseblocks of 65536 bytes (64.0 KiB), min. I/O size 1 bytes
libscan: scanning eraseblock 2039 -- 100 % complete
ubiformat: 2040 eraseblocks are supposedly empty
ubiformat: formatting eraseblock 2039 -- 100 % complete

root@witherspoon-128:~# hexdump -C /dev/mtd7 > /tmp/mtd7-hexdump.txt

5. Both systems can attach it:

root@witherspoon-128:~# ubiattach -m 7
[ 1096.197592] ubi1: default fastmap pool size: 100
[ 1096.202237] ubi1: default fastmap WL pool size: 50
[ 1096.207280] ubi1: attaching mtd7
[ 1096.241599] ubi1: scanning is finished
[ 1096.264117] ubi1: attached mtd7 (name "alt-obmc-ubi", size 127 MiB)
[ 1096.270412] ubi1: PEB size: 65536 bytes (64 KiB), LEB size: 65408 bytes
[ 1096.277205] ubi1: min./max. I/O unit sizes: 1/256, sub-page size 1
[ 1096.283399] ubi1: VID header offset: 64 (aligned 64), data offset: 128
[ 1096.290045] ubi1: good PEBs: 2040, bad PEBs: 0, corrupted PEBs: 0
[ 1096.296256] ubi1: user volume: 0, internal volumes: 1, max. volumes count: 128
[ 1096.303633] ubi1: max/mean erase counter: 0/0, WL threshold: 4096, image sequence number: 1888444368
[ 1096.312766] ubi1: available PEBs: 2034, total reserved PEBs: 6, PEBs reserved for bad PEB handling: 0
[ 1096.333675] ubi1: background thread "ubi_bgt1d" started, PID 1937
UBI device number 1, total 2040 LEBs (133432320 bytes, 127.2 MiB), available 2034 LEBs (133039872 bytes, 126.8 MiB), LEB size 65408 bytes (63.8 KiB)

6. Try to read it from u-boot by switching to the alternate chip:

root@witherspoon-128:~# reboot

ast# run bootalt

w81:

ast# ubi part obmc-ubi
ubi0: attaching mtd1
ubi0: scanning is finished
ubi0: attached mtd1 (name "mtd=2", size 127 MiB)
ubi0: PEB size: 65536 bytes (64 KiB), LEB size: 65408 bytes
ubi0: min./max. I/O unit sizes: 1/1, sub-page size 1
ubi0: VID header offset: 64 (aligned 64), data offset: 128
ubi0: good PEBs: 2040, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 0, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 0/0, WL threshold: 4096, image sequence number: 968759556
ubi0: available PEBs: 2036, total reserved PEBs: 4, PEBs reserved for bad PEB handling: 0
ast# 

r217:

ast# ubi part obmc-ubi
ubi0: attaching mtd1
UBI init error 22
ast#

I added some debug traces to u-boot and seems that in drivers/mtd/ubi/attach.c the good system reads UBI_IO_FF and moves on to the next sector but the failing system goes off on a different path into ubi_add_to_av() and ubi_compare_lebs().

Files:
The hexdump output shows the content of the ubi volume only differs in the image_seq value which is expected to be a different number every time.

r217-dmesg.txt
r217-mtd7-hexdump.txt
w81-dmesg.txt
w81-mtd7-hexdump.txt

[anoo1]

@legoater will probably need your help and expertise with this one. @shenki fyi.

[anoo1]

I flashed a 128MB image on the alt bmc chip on both systems and verified the checksum, and got the same result where r217 couldn't attach the ubi volume.

root@witherspoon-128:~# md5sum /tmp/obmc-phosphor-image-witherspoon-128.ubi.mtd
6261c6251572b5c0056e13589d9feee5  /tmp/obmc-phosphor-image-witherspoon-128.ubi.mtd

root@witherspoon-128:~# flashcp -v /tmp/obmc-phosphor-image-witherspoon-128.ubi.mtd /dev/mtd4
Erasing block: 2048/2048 (100%)
Writing kb: 131072/131072 (100%)
Verifying kb: 131072/131072 (100%)

root@witherspoon-128:~# md5sum /dev/mtd4
6261c6251572b5c0056e13589d9feee5  /dev/mtd4

[legoater]

We should make sure that U-Boot (aspeed flash driver) and Linux (spi-nor driver) have support for this new flash chip and build flash images with the required changes, if needed.

Regarding QEMU, we should check that the m25p80 model has the same support and I think we should add a new aspeed witherspoon-bmc machine with the definition of this new flash chip. That would be cleaner.

QEMU defines :

    { INFO_STACKED("n25q00",    0x20ba21, 0x1000, 64 << 10, 2048, ER_4K, 4) },
    { INFO_STACKED("n25q00a",   0x20bb21, 0x1000, 64 << 10, 2048, ER_4K, 4) },
    { INFO_STACKED("mt25ql01g", 0x20ba21, 0x1040, 64 << 10, 2048, ER_4K, 2) },
    { INFO_STACKED("mt25qu01g", 0x20bb21, 0x1040, 64 << 10, 2048, ER_4K, 2) },

Is that the correct chip ? but OpenBMC u-boot defines :

 #define MT25QL01GB             0x21ba20

May be we have typo ?

If this is the datasheet :

https://www.micron.com/-/media/client/global/documents/products/data-sheet/nor-flash/serial-nor/mt25q/die-rev-a/mt25q_qlks_l_01g_bba_0.pdf

it would nice to refer to it in the commit logs.

[legoater]

Is that the correct chip ? but OpenBMC u-boot defines :

 #define MT25QL01GB             0x21ba20

Unfortunately, this is not the problem.

[legoater]

The issue can be reproduced with QEMU. Include this patch to define a new machine :

--- qemu-aspeed.git.orig/hw/arm/aspeed.c
+++ qemu-aspeed.git/hw/arm/aspeed.c
@@ -441,6 +441,17 @@ static const AspeedBoardConfig aspeed_bo
         .i2c_init  = witherspoon_bmc_i2c_init,
         .ram       = 512 * MiB,
         .num_cpus  = 1,
+     }, {
+        .name      = MACHINE_TYPE_NAME("witherspoon-bmc-128"),
+        .desc      = "OpenPOWER Witherspoon BMC (ARM1176)",
+        .soc_name  = "ast2500-a1",
+        .hw_strap1 = WITHERSPOON_BMC_HW_STRAP1,
+        .fmc_model = "mt25ql01g",
+        .spi_model = "mx66l1g45g",
+        .num_cs    = 2,
+        .i2c_init  = witherspoon_bmc_i2c_init,
+        .ram       = 512 * MiB,
+        .num_cpus  = 1,
     }, {
         .name      = MACHINE_TYPE_NAME("ast2600-evb"),
         .desc      = "Aspeed AST2600 EVB (Cortex A7)",

[legoater]

I pushed a patch for this new witherspoon machine under :

https://github.com/legoater/qemu/commits/aspeed-4.1

[legoater]

@anoo1 what you can start by doing is activate debug under u-boot. I think UBI has its own debug macros. Check that the partitions and the UBIFS are correctly read.

To generate a custom flash image, you only have to dd the file u-boot.bin at the beginning of an existing flash image. Something like :

dd  if=u-boot.bin of=./flash-witherspoon-128-test.ubi conv=notrunc 

[legoater]

this is fixing the issue on QEMU :

From d95e7bf8f0a7c6352f8e8325e0b807118fbab6b8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= <clg@kaod.org>
Date: Thu, 16 May 2019 17:54:17 +0200
Subject: [PATCH u-boot] aspeed/flash: fix definition of the MT25QL01GB chip
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Signed-off-by: Cédric Le Goater <clg@kaod.org>
Fixes: 3c33d1e84ebb ("aspeed/flash: Add MT25QL01GB chip")
---

 It should fix https://github.com/openbmc/u-boot/issues/18

 arch/arm/mach-aspeed/flash.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/arm/mach-aspeed/flash.c b/arch/arm/mach-aspeed/flash.c
index 54a6059f4216..db69d514952f 100644
--- a/arch/arm/mach-aspeed/flash.c
+++ b/arch/arm/mach-aspeed/flash.c
@@ -1149,10 +1149,11 @@ static ulong flash_get_size (ulong base, flash_info_t *info)
 			info->readcmd = 0x0b;
 			info->dualport = 0;
 			info->dummybyte = 1;
-			info->buffersize = 1;
+			info->buffersize = 256;
 			WriteClk = 50;
 			EraseClk = 25;
 			ReadClk  = 50;
+			info->address32 = 1;
 			break;
 
 		default:	/* use JEDEC ID */
-- 
2.20.1

Can you give it a try on a real system ?

[anoo1]

Can you give it a try on a real system ?

This patch solves the issue. I've flashed it on 2 systems that were previously unable to boot. Thanks @legoater !

[anoo1]

Closing issue. Patch: https://patchwork.ozlabs.org/patch/1100654/

[shenki]

We generally close the issue when the patch lands.