[22.03] ath79: mtdblock5 too small #11417

odon-noda · 2022-12-04T15:14:41Z

Symptom

Lost 640 kB on / after upgrade from 21.02.3 to 22.03.2.
/dev/mtd5 size shrunk from 2048 kB to 1408 kB.

Impact

No space left on device during following
# opkg upgrade `list-upgradable` ... (libwolfssl5.5.3.ee39414e required by wpad-basic-wolfssl)
wpad-basic-wolfssl upgrade requires 1M.

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock5            1.4M    252.0K      1.1M  18% /overlay
overlayfs:/overlay        1.4M    252.0K      1.1M  18% /

# opkg upgrade wpad-basic-wolfssl --no-check-certificate
Upgrading wpad-basic-wolfssl on root from 2022-01-16-cff80b4f-13.1 to 2022-01-16-cff80b4f-14.1...
Downloading https://downloads.openwrt.org/releases/22.03.2/packages/mips_24kc/base/wpad-basic-wolfssl_2022-01-16-cff80b4f-14.1_mips_24kc.ipk
Upgrading hostapd-common on root from 2022-01-16-cff80b4f-13.1 to 2022-01-16-cff80b4f-14.1...
Downloading https://downloads.openwrt.org/releases/22.03.2/packages/mips_24kc/base/hostapd-common_2022-01-16-cff80b4f-14.1_mips_24kc.ipk
Installing libwolfssl5.5.3.ee39414e (5.5.3-stable-1) to root...
Downloading https://downloads.openwrt.org/releases/22.03.2/packages/mips_24kc/base/libwolfssl5.5.3.ee39414e_5.5.3-stable-1_mips_24kc.ipk
Configuring hostapd-common.
Configuring libwolfssl5.5.3.ee39414e.
Configuring wpad-basic-wolfssl.

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock5            1.4M      1.2M    136.0K  90% /overlay
overlayfs:/overlay        1.4M      1.2M    136.0K  90% /

# grep 'info' from /usr/lib/opkg/info/libwolfssl5.5.3.ee39414e.control
Package: libwolfssl5.5.3.ee39414e
Version: 5.5.3-stable-1
Maintainer: Eneas U de Queiroz <cotequeiroz@gmail.com>
Installed-Size: 507376
Description:  wolfSSL (formerly CyaSSL) is an SSL library optimized for small footprint, both on disk and for memory use.

# grep 'info' from /usr/lib/opkg/info/wpad-basic-wolfssl.control
Package: wpad-basic-wolfssl
Version: 2022-01-16-cff80b4f-14.1
Maintainer: Felix Fietkau <nbd@nbd.name>
Installed-Size: 395568
#

Other terms

/dev/mtd5, mtdblock5, Flash layout

Device

The same router was used for Analysis.
TP-Link Archer C6 v2 EU
Qualcomm Atheros QCA956X ver 1 rev 0
ath79/generic

Steps to reproduce

21.02.3
# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock5            2.0M    556.0K      1.5M  27% /overlay
overlayfs:/overlay        2.0M    556.0K      1.5M  27% /

22.03.2
# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock5            1.4M    260.0K      1.1M  18% /overlay
overlayfs:/overlay        1.4M    260.0K      1.1M  18% /

21.02.5
# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mtdblock5            1.9M    264.0K      1.7M  13% /overlay
overlayfs:/overlay        1.9M    264.0K      1.7M  13% /
#

Analysis

Table	mtdblock5 size
Version	mtdblock5	hex
21.02.3	2048	0x800
21.02.5	1984	0x7C0
22.03.0	1408	0x580
22.03.2	1408	0x580
22.03 SNAPSHOT	1536	0x600

-----------------------------------------------------
OpenWrt 21.02.3, r16554-1d4dea6d4f
-----------------------------------------------------
# cat /proc/partitions
major minor  #blocks  name

  31        0        128 mtdblock0
  31        1         64 mtdblock1
  31        2       7808 mtdblock2
  31        3       2001 mtdblock3
  31        4       5806 mtdblock4
  31        5       2048 mtdblock5
  31        6        128 mtdblock6
  31        7         64 mtdblock7
# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00020000 00010000 "u-boot"
mtd1: 00010000 00010000 "info"
mtd2: 007a0000 00010000 "firmware"
mtd3: 001f4751 00010000 "kernel"
mtd4: 005ab8af 00010000 "rootfs"
mtd5: 00200000 00010000 "rootfs_data"
mtd6: 00020000 00010000 "tplink"
mtd7: 00010000 00010000 "art"
#

-----------------------------------------------------
OpenWrt 21.02.5, r16688-fa9a932fdb
-----------------------------------------------------
# cat /proc/partitions
major minor  #blocks  name

  31        0        128 mtdblock0
  31        1         64 mtdblock1
  31        2       7808 mtdblock2
  31        3       2015 mtdblock3
  31        4       5792 mtdblock4
  31        5       1984 mtdblock5
  31        6        128 mtdblock6
  31        7         64 mtdblock7
# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00020000 00010000 "u-boot"
mtd1: 00010000 00010000 "info"
mtd2: 007a0000 00010000 "firmware"
mtd3: 001f7c25 00010000 "kernel"
mtd4: 005a83db 00010000 "rootfs"
mtd5: 001f0000 00010000 "rootfs_data"
mtd6: 00020000 00010000 "tplink"
mtd7: 00010000 00010000 "art"

----------------------------------------------------
OpenWrt 22.03.0, r19685-512e76967f
-----------------------------------------------------
# cat  /proc/partitions
major minor  #blocks  name

  31        0        128 mtdblock0
  31        1         64 mtdblock1
  31        2       7808 mtdblock2
  31        3       2191 mtdblock3
  31        4       5616 mtdblock4
  31        5       1408 mtdblock5
  31        6        128 mtdblock6
  31        7         64 mtdblock7
# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00020000 00010000 "u-boot"
mtd1: 00010000 00010000 "info"
mtd2: 007a0000 00010000 "firmware"
mtd3: 00223c28 00010000 "kernel"
mtd4: 0057c3d8 00010000 "rootfs"
mtd5: 00160000 00010000 "rootfs_data"
mtd6: 00020000 00010000 "tplink"
mtd7: 00010000 00010000 "art"
#

-----------------------------------------------------
OpenWrt 22.03.2, r19803-9a599fee93
-----------------------------------------------------
# cat  /proc/partitions
major minor  #blocks  name

  31        0        128 mtdblock0
  31        1         64 mtdblock1
  31        2       7808 mtdblock2
  31        3       2191 mtdblock3
  31        4       5616 mtdblock4
  31        5       1408 mtdblock5
  31        6        128 mtdblock6
  31        7         64 mtdblock7
# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00020000 00010000 "u-boot"
mtd1: 00010000 00010000 "info"
mtd2: 007a0000 00010000 "firmware"
mtd3: 00223d8b 00010000 "kernel"
mtd4: 0057c275 00010000 "rootfs"
mtd5: 00160000 00010000 "rootfs_data"
mtd6: 00020000 00010000 "tplink"
mtd7: 00010000 00010000 "art"
#

-----------------------------------------------------
OpenWrt SNAPSHOT, r21374-366bcffa0e
-----------------------------------------------------
# cat  /proc/partitions
major minor  #blocks  name

  31        0        128 mtdblock0
  31        1         64 mtdblock1
  31        2       7808 mtdblock2
  31        3       2440 mtdblock3
  31        4       5367 mtdblock4
  31        5       1536 mtdblock5
  31        6        128 mtdblock6
  31        7         64 mtdblock7
# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00020000 00010000 "u-boot"
mtd1: 00010000 00010000 "info"
mtd2: 007a0000 00010000 "firmware"
mtd3: 002621ef 00010000 "kernel"
mtd4: 0053de11 00010000 "rootfs"
mtd5: 00180000 00010000 "rootfs_data"
mtd6: 00020000 00010000 "tplink"
mtd7: 00010000 00010000 "art"
#

Most likely reason: Flash layout mtdblock5 length has been shortened

The text was updated successfully, but these errors were encountered:

brada4 · 2022-12-05T13:48:08Z

https://openwrt.org/supported_devices/864_warning
With 22.03 you can reduce rootfs by removing some packages in web from before resorting to compiling compact kernel, like no luci web but still up-to-date access point, but it is approaching end of the road, no less.

brada4 · 2022-12-07T16:26:43Z

You can use auc to upgrade without duplication of same files.
First flash the firmware to clear overlay, then use install auc and do it.

odon-noda · 2022-12-13T00:35:56Z

Hi @brada4
Thank you for you fast reaction, RCA and valuable advice.
I updated Archer C6 V2 Flash Layout and summarized this conclusion

Root cause

The OpenWrt Flash Layout and 'opkg upgrade'.

Old version of the updated file exists under /root (lower),
new/upgraded version exists under /overley (upper).
Example:
21.02.3 upgraded, libwolfssl exists twice, ~1MB each

# ls -lh /rom/usr/lib/libwolfssl*
-rwxr-xr-x    1 root     root        1.0M Apr 16  2022 /rom/usr/lib/libwolfssl.so.5.2.0.99a5b54a
# ls -lh /overlay/upper/usr/lib/libwolfssl*
c---------    1 root     root        0,   0 Dec 18 19:55 /overlay/upper/usr/lib/libwolfssl.so.5.2.0.99a5b54a
-rwxr-xr-x    1 root     root        1.1M Nov 27 16:36 /overlay/upper/usr/lib/libwolfssl.so.5.5.3.99a5b54a

Firmware size increases naturally with every new release.
https://openwrt.org/supported_devices/864_warning#analysis_of_firmware_size_growth
Hence #11417 is not an issue, this is the feature.

Solution

Avoid using 'opkg upgrade'. Create a new image with the updated packages instead.

Saving firmware space and RAM

Option 1 - standard image updated with auc

Procedure:

Have "-1" version installed (e.g. 21.02.3)
Upgrade it with current master distribution (e.g. 22.03.2)
Update it with auc. This is the crucial step.

Pros
small footprint after upgrade+auc
all packages are up-to-date
Cons
none

Option 2 - custom image using Image Builder

Pros
Simple
easily include custom packages, files, scripts into rootfs
suitable for mass distribution (couple of / dozens of the same devices)
Cons
A higher effort than Option 1

Another Image Builder frontends

Option 3 - custom build

compiling compact image

Pros
only required essential functions
long-term
Cons
demanding

Option 4 - HW replacement

buy a new device

Pros
...
Cons
lost investment (flash ROM lifetime ~11 years)
higher power consumption (Archer C6 V2 has 3.06 W (!))

brada4 · 2022-12-13T00:59:49Z

1.5 You can upgrade standard image components with auc.

odon-noda · 2022-12-13T11:47:11Z

Oh @brada4
Finally! I got it.
Option 1 = standard upgrade + auc is the right choice

BEFORE = 21.02.3
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.8M      3.8M         0 100% /rom
/dev/mtdblock5            2.0M      1.9M    144.0K  93% /overlay
# opkg update ; opkg list-upgradable ; echo $?
0

AFTER = sysupgrade to 22.03.2 + auc
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.3M      4.3M         0 100% /rom
/dev/mtdblock5            1.4M    268.0K      1.1M  19% /overlay
# opkg update ; opkg list-upgradable ; echo $?
0

brada4 · 2022-12-13T13:45:24Z

it is even less scary - you dont need opkg upgrade anymore. auc server builds image from latest versions of packages for that release.

odon-noda closed this as completed Dec 13, 2022

odon-noda changed the title ~~ath79 mtdblock5 too small in 22.03~~ [22.03] ath79: mtdblock5 too small in 22.03 Jan 8, 2023

odon-noda changed the title ~~[22.03] ath79: mtdblock5 too small in 22.03~~ [22.03] ath79: mtdblock5 too small Jan 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[22.03] ath79: mtdblock5 too small #11417

[22.03] ath79: mtdblock5 too small #11417

odon-noda commented Dec 4, 2022 •

edited

Loading

brada4 commented Dec 5, 2022 •

edited

Loading

brada4 commented Dec 7, 2022 •

edited

Loading

odon-noda commented Dec 13, 2022 •

edited

Loading

brada4 commented Dec 13, 2022

odon-noda commented Dec 13, 2022 •

edited

Loading

brada4 commented Dec 13, 2022

[22.03] ath79: mtdblock5 too small #11417

[22.03] ath79: mtdblock5 too small #11417

Comments

odon-noda commented Dec 4, 2022 • edited Loading

Symptom

Impact

Other terms

Device

Steps to reproduce

Analysis

Most likely reason: Flash layout mtdblock5 length has been shortened

brada4 commented Dec 5, 2022 • edited Loading

brada4 commented Dec 7, 2022 • edited Loading

odon-noda commented Dec 13, 2022 • edited Loading

Root cause

Solution

Saving firmware space and RAM

Option 1 - standard image updated with auc

Option 2 - custom image using Image Builder

Another Image Builder frontends

Option 3 - custom build

Option 4 - HW replacement

brada4 commented Dec 13, 2022

odon-noda commented Dec 13, 2022 • edited Loading

brada4 commented Dec 13, 2022

odon-noda commented Dec 4, 2022 •

edited

Loading

brada4 commented Dec 5, 2022 •

edited

Loading

brada4 commented Dec 7, 2022 •

edited

Loading

odon-noda commented Dec 13, 2022 •

edited

Loading

odon-noda commented Dec 13, 2022 •

edited

Loading