ramips/mt7621: OpenWrt 22.03.3 release fails to boot with LZMA ERROR 1

[BetTD]

Describe the bug

My Afoundry EW1200 was running fine on an old stable release. Today, I went to update it with a sysupgrade image and now it's stuck in a boot loop. After hooking it up via a serial interface, I can see the following:

===================================================================
                MT7621   stage1 code 13:14:00 (ASIC)
                CPU=50000000 HZ BUS=16666666 HZ
==================================================================
Change MPLL source from XTAL to CR...
do MEMPLL setting..
MEMPLL Config : 0x11000000
3PLL mode + External loopback
=== XTAL-40Mhz === DDR-1200Mhz ===
PLL2 FB_DL: 0x12, 1/0 = 543/481 49000000
PLL3 FB_DL: 0x14, 1/0 = 623/401 51000000
MEMPLL 3PLL mode calibration fail
do DDR setting..[00320000]
Apply DDR3 Setting...(use default AC)
          0    8   16   24   32   40   48   56   64   72   80   88   96  104  112  120
      --------------------------------------------------------------------------------
0000:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0001:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0002:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0003:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0004:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0005:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0006:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0007:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0008:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0009:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000A:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000B:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000C:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000D:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000E:|    0    0    0    0    0    0    0    0    0    0    1    1    1    1    1    1
000F:|    0    0    0    0    0    1    1    1    1    1    1    1    1    1    1    1
0010:|    1    1    1    1    1    1    1    1    1    1    0    0    0    0    0    0
0011:|    1    1    1    1    0    0    0    0    0    0    0    0    0    0    0    0
0012:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0013:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0014:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0015:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0016:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0017:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0018:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0019:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001A:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001B:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001C:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001D:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001E:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001F:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
rank 0 coarse = 15
rank 0 fine = 80
B:|    0    0    0    0    0    0    0    0    1    1    1    0    0    0    0    0
opt_dle value:9
DRAMC_R0DELDLY[018]=00001E1E
==================================================================
                RX      DQS perbit delay software calibration
==================================================================
1.0-15 bit dq delay value
==================================================================
bit|     0  1  2  3  4  5  6  7  8  9
--------------------------------------
0 |    13 13 12 15 12 12 13 9 11 11
10 |    14 15 12 13 12 13
--------------------------------------

==================================================================
2.dqs window
x=pass dqs delay value (min~max)center
y=0-7bit DQ of every group
input delay:DQS0 =30 DQS1 = 30
==================================================================
bit     DQS0     bit      DQS1
0  (1~58)29  8  (1~57)29
1  (1~58)29  9  (1~57)29
2  (1~55)28  10  (1~59)30
3  (1~60)30  11  (1~58)29
4  (1~58)29  12  (1~59)30
5  (1~60)30  13  (1~55)28
6  (2~58)30  14  (1~59)30
7  (1~56)28  15  (1~58)29
==================================================================
3.dq delay value last
==================================================================
bit|    0  1  2  3  4  5  6  7  8   9
--------------------------------------
0 |    14 14 14 15 13 12 13 11 12 12
10 |    14 15 12 15 12 14
==================================================================
==================================================================
     TX  perbyte calibration
==================================================================
DQS loop = 15, cmp_err_1 = ffff0000
dqs_perbyte_dly.last_dqsdly_pass[0]=15,  finish count=1
dqs_perbyte_dly.last_dqsdly_pass[1]=15,  finish count=2
DQ loop=15, cmp_err_1 = ffff0080
dqs_perbyte_dly.last_dqdly_pass[1]=15,  finish count=1
DQ loop=14, cmp_err_1 = ffff0000
dqs_perbyte_dly.last_dqdly_pass[0]=14,  finish count=2
byte:0, (DQS,DQ)=(8,8)
byte:1, (DQS,DQ)=(8,8)
20,data:88
[EMI] DRAMC calibration passed

===================================================================
                MT7621   stage1 code done
                CPU=50000000 HZ BUS=16666666 HZ
===================================================================


Uboot identify string is mt7621:s1200en:1.5.7(U-Boot 1.1.3 (Nov  3 2016 - 13:37:42))

Board: Ralink APSoC DRAM:  128 MB
relocate_code Pointer at: 87fb8000

Config XHCI 40M PLL
******************************
Software System Reset Occurred
******************************
flash manufacture id: c2, device id 20 18
find flash: MX25L12805D
*** Warning - bad CRC, using default environment

============================================
Ralink UBoot Version: 4.2.1.0
--------------------------------------------
ASIC 7621_MP (MAC to MT7530 Mode)
DRAM_CONF_FROM: Auto-Detection
DRAM_TYPE: DDR3
DRAM bus: 16 bit
Xtal Mode=3 OCP Ratio=1/3
Flash component: SPI Flash
Date:Nov  3 2016  Time:13:37:42
============================================
icache: sets:256, ways:4, linesz:32 ,total:32768
dcache: sets:256, ways:4, linesz:32 ,total:32768

 ##### The CPU freq = 880 MHZ ####
 estimate memory size =128 Mbytes
#Reset_MT7530
set LAN/WAN LLLLW
Setting UART to GPIO
Setting I2C to GPIO

Please choose the operation:
   1: Load system code to SDRAM via TFTP.
   2: Load system code then write to Flash via TFTP.
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.
   7: Load Boot Loader code then write to Flash via Serial.
   9: Load Boot Loader code then write to Flash via TFTP.                                                                                                  0
Upgrade mode read(1:2:0)

3: System Boot system code via Flash.
## Booting image at bfc50000 ...
   Image Name:   MIPS OpenWrt Linux-5.15.98
   Image Type:   MIPS Linux Kernel Image (lzma compressed)
   Data Size:    5518401 Bytes =  5.3 MB
   Load Address: 80001000
   Entry Point:  80001000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... LZMA ERROR 1 - must RESET board to recover

I also tried flashing the firmware (both release and snapshot) directly over TFTP, but that also results in this LZMA ERROR 1 after it's done flashing the image:

2: System Load Linux Kernel then write to Flash via TFTP.
 Warning!! Erase Linux in Flash then burn new one. Are you sure?(Y/N)
 Please Input new ones /or Ctrl-C to discard
        Input device IP (10.10.10.123) ==:192.168.0.204
        Input server IP (10.10.10.3) ==:192.168.0.10
        Input Linux Kernel filename () ==:initramfs2.bin

 netboot_common, argc= 3

 NetTxPacket = 0x87FE4E40

 KSEG1ADDR(NetTxPacket) = 0xA7FE4E40

 NetLoop,call eth_halt !

 NetLoop,call eth_init !
Trying Eth0 (10/100-M)

 Waitting for RX_DMA_BUSY status Start... done


 ETH_STATE_ACTIVE!!
TFTP from server 192.168.0.10; our IP address is 192.168.0.204
Filename 'initramfs2.bin'.

 TIMEOUT_COUNT=10,Load address: 0x80100000
Loading: Got ARP REQUEST, return our IP
Got ARP REQUEST, return our IP
Got ARP REQUEST, return our IP
Got ARP REQUEST, return our IP
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
Got ARP REQUEST, return our IP
checksum bad
checksum bad
Got ARP REPLY, set server/gtwy eth addr (2c:4d:54:e9:65:59)
Got it
#################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ######################################
done
Bytes transferred = 5518465 (543481 hex)
NetBootFileXferSize= 00543481
....................................................................................
....................................................................................
.
.
Done!
## Booting image at bfc50000 ...
   Image Name:   MIPS OpenWrt Linux-5.15.98
   Image Type:   MIPS Linux Kernel Image (lzma compressed)
   Data Size:    5518401 Bytes =  5.3 MB
   Load Address: 80001000
   Entry Point:  80001000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... LZMA ERROR 1 - must RESET board to recover

I have also tried flashing OpenWrt 21.02.5, and this was successful.

OpenWrt version

22.03.3 (I cannot run the command on this version as it fails to boot)

OpenWrt target/subtarget

ramips/mt7621

Device

Afoundry EW1200

Image kind

Official downloaded image

Steps to reproduce

1. Flash OpenWrt 22.03.3 (or latest snapshot).
2. System won't boot

Actual behaviour

The system never boots, it fails with LZMA ERROR 1 when attempting to decompress the kernel image.

Expected behaviour

The system should decompress the image fine and complete the boot process.

Additional info

No response

Diffconfig

No response

Terms

• I am reporting an issue for OpenWrt, not an unsupported fork.

[Spudz76]

I think this usually occurs when the kernel has gotten larger than the original kernel partition, thus trying to read 5+MB from a let's say 4MB partition will truncate the read and fail decompression.

However,

find flash: MX25L12805D
*** Warning - bad CRC, using default environment

This bothers me maybe you have actually bad flash, or the clock rate is too high, etc. It's only 16MB also which alludes it won't have a kernel partition larger than 4MB. Would bet the 21.02.5 kernel ends up below 4MB or whatever the kernel partition size is. Even if it were 5MB it would still be too big.

I would hack around at the DTS for your device since that's where both of the above are defined.

[Spudz76]

Then again the ram-load through TFTP should work regardless of flash situation.

[BetTD]

I think this usually occurs when the kernel has gotten larger than the original kernel partition, thus trying to read 5+MB from a let's say 4MB partition will truncate the read and fail decompression.

Wouldn't this result in an error during flashing and not after? You may be right in that the kernel is now larger than 4 MB. What options do I have now if I want to stay up to date?

However,

find flash: MX25L12805D
*** Warning - bad CRC, using default environment

This bothers me maybe you have actually bad flash, or the clock rate is too high, etc. It's only 16MB also which alludes it won't have a kernel partition larger than 4MB. Would bet the 21.02.5 kernel ends up below 4MB or whatever the kernel partition size is. Even if it were 5MB it would still be too big.

I would hack around at the DTS for your device since that's where both of the above are defined.

This may be a dumb question, but what do you mean by DTS?

It may just have bad flash as well, I bought this second-hand for quite cheap, and the 2.4 GHz band doesn't even work, but as a cheap 5 GHz-only access point it gets the job done (after replacing the stock firmware with OpenWrt, that is). I'd much rather keep the system up to date though.

[Spudz76]

DTS is the device tree source file, and each "board" (platform/model) has one

Yours is target/linux/ramips/dts/mt7621_afoundry_ew1200.dts

Reading through there it seems the partition is unified (uimage) and has 16449536 bytes (uboot+env+factory take up 0x50000 ir 327680 bytes of the total 16MB flash) therefore kernel size should not be a factor. Must be some LZMA format incompatibility, which matches the TFTP result.

Flash clock speed is the spi-max-frequency on line 54 of the DTS, although again since TFTP fails too and that happens all in RAM, it's probably also not the clock speed (although I'd prefer if the env-checksum message went away).

To some extent this is all a guessing game since nobody knows exactly what uboot+mods are on the device or what bugs/quirks it might support other than to try stuff like a black-box situation. There have been several "flavors" of LZMA, maybe it wants the oldest version. Lots of uboots didn't even support LZMA, on those platforms the lzma-loader is used and the uImage has the kernel marked as "uncompressed" and the lzma-loader does its own decompression (using current LZMA code) rather than hoping the builtin LZMA decompressor isn't broken, or using gzip.

[Spudz76]

Confirmed your device is not using uimage-lzma-loader like most of the other MT7621's do. Thus you are stuck at guessing what the builtin LZMA decompressor likes. Some other models use special lzma compression flags, maybe copying some of those over would help.

Also possibly uboot is not allocating enough destination space in RAM. Error "1" is generic failure and tells nobody anything really other than something went wrong.

Trying to find a reason the older version works, but maybe it uses older lzma compressor (different versions of host-tools? different tactics in the Makefiles?).

Also just noticed after re-reading your serial outputs that you didn't RAM-load the kernel from TFTP but rather flashed it, so flash is still involved (not just RAM-only). So I'd next turn down the flash speed in the DTS and then see if it gets rid of corruption.

You could try option 1 (load to SDRAM) and see if that launches, if so then definitely flash speed or bad flash.

[Mijzelf]

This looks like a bug I had on my ZyXEL P2812HNU-F1. Read here. It turned out u-boot only reserved 8MiB target space for decompressing the kernel.

[xabolcs]

Hi @BetTD,

did you try earlier 22.03 releases or even -rcs?
Please try to flash and boot 22.03.0 too! I think it will throw LZMA ERROR 1 too.

May I ask for a serial log from the latest 21.02 release (e.g. 21.02.5)? Please paste only the relevant part, starting from the bootmenu, and please include a few lines about OpenWrt booting too!

I'm sure that the fix will be that oneliner what @Spudz76 wrote.
For examples see commit history, e.g.:

• 25ede51
• b63d6d4

As you can see it would be nice to check if the initramfs boot (boot menu 1) too!

[Spudz76]

I guess if you go back to the last tested working 21.02 version, while you've got it up you could dump mtd0 and attach it to a reply, so it can be disassembled and checked for misbehavior like only allocating 8MB of RAM for the expanded destination or such.

But also enabling the lzma-loader might work around both issues, since uboot will copy the ~5.3MB into the 8MB region and then launch the loader which should then reallocate proper amounts of RAM for the expansion (beyond whatever uboot did). As long as the compressed image never gets above 8MB it should "just work" without replacing uboot with a patched version.

So an mtd0 dump would mostly be for curiosity. Working on a patch (based on snapshot/master) to enable lzma-loader. If that is cool then it can possibly be backported to the releases that don't work in case anyone that also has one of these prefers releases over snapshots. Or at least drop the non-working ones so they don't exist (users without serial capabilities would have sort of a hassle un-bricking, probably).

[Spudz76]

Patch was actually a one-liner, now available in the above PR

@BetTD or anyone with one of these devices that could test it.

You could apply by checking out master and then from the base directory:

wget -qO- https://github.com/openwrt/openwrt/pull/12211.patch | patch -p1

And then configure/build/test, please.

[BetTD]

Hey, apologies for the delay.

@Spudz76 - Good news! The patch you made seems to have worked, thanks a lot!

Checked out master and applied the patch, configured, built, and flashed.

Please choose the operation:
   1: Load system code to SDRAM via TFTP.
   2: Load system code then write to Flash via TFTP.
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.
   7: Load Boot Loader code then write to Flash via Serial.
   9: Load Boot Loader code then write to Flash via TFTP.                                                                                                  4
You choosed 2
                                                                                                                                                           0
Upgrade mode read(1:2:0)


2: System Load Linux Kernel then write to Flash via TFTP.
 Warning!! Erase Linux in Flash then burn new one. Are you sure?(Y/N)
 Please Input new ones /or Ctrl-C to discard
        Input device IP (10.10.10.123) ==:192.168.0.204
        Input server IP (10.10.10.3) ==:192.168.0.10
        Input Linux Kernel filename () ==:initramfs3.bin

 netboot_common, argc= 3

 NetTxPacket = 0x87FE4E40

 KSEG1ADDR(NetTxPacket) = 0xA7FE4E40

 NetLoop,call eth_halt !

 NetLoop,call eth_init !
Trying Eth0 (10/100-M)

 Waitting for RX_DMA_BUSY status Start... done


 ETH_STATE_ACTIVE!!
TFTP from server 192.168.0.10; our IP address is 192.168.0.204
Filename 'initramfs3.bin'.

 TIMEOUT_COUNT=10,Load address: 0x80100000
Loading: checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
checksum bad
T checksum bad
checksum bad
T Got ARP REPLY, set server/gtwy eth addr (2c:4d:54:e9:65:59)
Got it
#################################################################
         ####checksum bad
#checksum bad
############################################################
         #################################################################
         #################################################################
         ########checksum bad
################checksum bad
#########################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ############################################################checksum bad
#####
         ####checksum bad
#############################################################
         #################################################################
         ###########################################################checksum bad
checksum bad
######
         ####################################################
done
Bytes transferred = 6251609 (5f6459 hex)
NetBootFileXferSize= 005f6459
...............................................................................................
...............................................................................................
.
.
Done!
## Booting image at bfc50000 ...
   Image Name:   MIPS OpenWrt Linux-5.15.102
   Image Type:   MIPS Linux Kernel Image (uncompressed)
   Data Size:    6251545 Bytes =  6 MB
   Load Address: 80001000
   Entry Point:  80001000
   Verifying Checksum ... OK
OK
No initrd
## Transferring control to Linux (at address 80001000) ...
## Giving linux memsize in MB, 128

Starting kernel ...



OpenWrt kernel loader for MIPS based SoC
Copyright (C) 2011 Gabor Juhos <juhosg@openwrt.org>
Decompressing kernel... done!
Starting kernel at 80001000...

[    0.000000] Linux version 5.15.102 (compiler@bcn-01) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 12.2.0 r22314-c98e09f01b) 12.2.0, GNU ld (GNU Binutils) 2.40.0) #0 SMP Sun Mar 19 17:20:18 2023
[    0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
[    0.000000] printk: bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
[    0.000000] MIPS: machine is EW1200
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] VPE topology {2,2} total 4
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.000000] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] percpu: Embedded 11 pages/cpu s15632 r8192 d21232 u45056
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32480
[    0.000000] Kernel command line: console=ttyS0,57600 rootfstype=squashfs,jffs2
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 115628K/131072K available (7106K kernel code, 628K rwdata, 1444K rodata, 4504K init, 242K bss, 15444K reserved, 0K cma-reserved, 0K highmem)
[...]

Now my question is, am I good to continue running OpenWrt SNAPSHOT with this patch until a new release is out that includes it? Or should I stay in 21.02.5 until then?

[Spudz76]

Yes, also snapshot has some non-backport changes to ramips/mt7621 that might help something, like the broken 2.4G wifi.