ath79: fixes for Senao devices #3872

mpratt14 · 2021-02-15T06:51:50Z

ath79: lzma-loader: allow setting custom magic

and allow setting FLASH_MAX

The mtdsplit parser was recently refactored
to allow the kernel to have a custom image header magic

Let's also do this for the lzma-loader

For example:
When implemented together,
this allows the kernel to "appear" to be a rootfs
in order to trick OEM software to write an image
that is actually kernel + rootfs

At the same time, it would boot to openwrt normally
by setting the same magic in DTS

ath79: remove 'fakeroot' for Senao devices

by using a custom kernel header magic
in both OKLI, DTS, and makefile
this hack is not necessary anymore

however, "rootfs" size and checksum
must now be supplied by the factory.bin
because OEM script assumes a squashfs header
and takes size value from the header that would be there,
but this header no longer exists

This allows freeing up of 1 eraseblock
previously used by the "fakeroot" partition
for bypassing the OEM image verification.

Also:
these Senao devices with a 'failsafe' partition
and the tar-gz factory.bin platform would otherwise require
flashing the new upcoming tar-gz sysupgrade.bin afterward.

so this also prevents having to flash twice from OEM

ath79: rename 'engenius' makefile definitions to 'senao'

These recipes and definitions can apply
to devices from other vendors
with PCB boards or SDK produced by Senao
not only the brand Engenius

for example:
Extreme Networks, WatchGuard, OpenMesh,
Fortinet, ALLNET, OCEDO, etc.

so rename all of these items
and move DEVICE_VENDOR from common to generic/tiny.mk

ath79: add factory.bin for ALLNET ALL-WAP02860AC

This device is a Senao-based product
with the tar-gz platform for factory.bin
and checksum verification at boot time
using checksums stored in uboot environment.

Extremely similar hardware/software to Engenius EAP1200H

ath79: cleanup DTS for ALLNET ALL-WAP02860AC

edit nodes to match similar boards

eth0
pcie0

bump SPI frequency to 40 MHz

remove &pll node: the property is defined already
in qca955x.dtsi

spelling of qca,no-eeprom;
and move to the correct node pcie0

ath79: add Senao 'failsafe' upgrade procedure

Use a similar method for sysupgrade
for Senao boards with a "failsafe" partition
from the OEM factory.bin upgrade method

Make loader partition writable so that sysupgrade works for it

Benefits are:

This allows freeing up of 1 eraseblock
previously used by the "fakeroot" partition
for bypassing the OEM image verification.

These boards are now unbrickable.
Previously, if the real kernel was damaged or missing
the OKLI kernel loader is programmed to halt
and the only way to recover was with UART serial console
because the the bootloader did not check the real kernel.

Consequences are:

The user must perform a sysupgrade after factory.bin flash

tested with
ENS202EXT v1

ath79: adjust tiny Engenius APs to 4k blocksize

ath79/tiny kernel config has
CONFIG_MTD_SPI_NOR_USE_4K_SECTORS=y
from commit
05d3540

Because of this, these changes are required for 2 reasons:

the boards in ath79/tiny using the new sysupgrade procedure
for Senao devices with a 'failsafe' partition
are failing to reboot to openwrt after a sysupgrade.

the stored checksum is calculated using the length
of the image to be flashed,
and the actual checksum changes due to JFFS2 space
being formatted within the calculated length of the rootfs

for boards with flash chips that support 4k sectors:
saving configuration over sysupgrade is not possible
because sysupgrade.tgz is appended at a 64k boundary
and the mtd parser starts JFFS2 at a 4k boundary.

for boards with flash chips that do not support 4k sectors:
partitioning with 4k boundaries causes a boot loop
from the mtd parser not finding kernel and rootfs.

example:
0x440000 length of kernel + rootfs (from sysupgrade.bin)
0x439000 offset of rootfs_data (from kernel log)

therefore partitioning must be different for these devices
depending on the flash chip model...
so the partitions node is split between dtsi and dts

DTSI files for Senao APs

mpratt14 · 2021-02-15T07:19:48Z

the DTSI patches were build tested, I'm in the process of flashing on those boards
for qca955x senao DTSI, &partitions node is needed because some models to be added in the future have 32 MB flash with ART at the end of flash

mpratt14 · 2021-02-15T07:20:41Z

@tmn505 are you available to test changes for ALLNET ALL-WAP02860AC ?

target/linux/ath79/dts/ar9341_engenius_eap300-v2.dts

adschm · 2021-02-15T16:48:06Z

These recipes and definitions can apply
to devices from other vendors
with PCB boards or SDK produced by Senao
not only the brand Engenius

But is this really about the boards or the SDK and not just about the implementation Engenius uses for it's firmware?

adschm · 2021-02-15T16:54:13Z

The proper way to do the DTSI consolidation would probably be like this:
https://git.openwrt.org/?p=openwrt/staging/adrian.git;a=shortlog;h=refs/heads/senaodtsi

Note that I'm not really convinced yet whether this should be "Senao" or "Engenius". It's even more messy with qca955x where we have Engenius devices with other setup, so the generic name won't be accurate.

This should see some additional eyes and discussion.

target/linux/ath79/dts/qca9557_engenius_eap1200h.dts

target/linux/ath79/dts/qca9558_allnet_all-wap02860ac.dts

target/linux/ath79/dts/qca955x_senao_ap.dtsi

mpratt14 · 2021-02-15T17:46:35Z

But is this really about the boards or the SDK and not just about the implementation Engenius uses for it's firmware?

We have 3 examples already of other vendors that have almost the same situation (board / SDK from Senao)

ALLNET board in this PR
OpenMesh products (now that I think of it...they can use this new DTSI) (except for partitions...)
Plasma Cloud products
(devolo boards maybe, but not similar software)

for OpenMesh dualboot, images are verified almost the same way, and uboot environment has the same variables, plus the extra one for active / inactive boot partition.

qca955x where we have Engenius devices with other setup

this is true, however for the Engenius devices that do not apply to this, there is senao-header command instead which is consistent with senao-tar-gz

mpratt14 · 2021-02-15T18:31:01Z

@adschm how about a more specific name for DTSI?

ar724x_senao_failsafe_ap.dtsi
etc.

mpratt14 · 2021-02-17T15:30:08Z

@ecsv Hi Sven, I used your script as a base for making my own for the "senao upgrade procedure" commit. There are some improvements that you may be interested in passing on to the other two scripts, and maybe they can be combined somehow as well

for example:
dd with bs=1 is extremely slow and makes sysupgrade take as long as 7 minutes (ar934x device)
dd with bs=4k is as quick as a sysupgrade for any other board

ecsv · 2021-02-17T15:32:07Z

@mpratt14 See ecsv@f7c9b43

mpratt14 · 2021-02-17T15:59:08Z

@ecsv ok nice

however I wonder...do you actually need dd for writing? for my case, it is only needed for the checksum

Also, I had significant trouble with doing mtd erase first and write later so I took the model from default_do_upgrade instead
the other way caused some blocks at the beginning of the partition to be lost when appending the backup file to JFFS2

mtd has an option for offset I believe would work as well

-p <number> write beginning at partition offset

ecsv · 2021-02-17T16:03:52Z

Since the data has to be extracted from the CE01 image - yes, dd is needed (or some other tool to extract the data from the correct position). And if you're data is correctly aligned with the erase block size then you should not have problems with "lost blocks"

And the script is already using -p for mtd.

mpratt14 · 2021-02-17T16:15:46Z

yeah I cannot explain why I was having that issue...the only difference is using mtd as erasing and writing at the same time, instead of separately...and only with the backup file

mpratt14 · 2021-02-21T10:16:21Z

@adschm I discovered a small regression in the tiny subtarget that has been giving me problems...

4k mtd erasesize was enabled by this commit
05d3540

every mtd device is showing up as 4k erasesize
however in the makefiles, all of ath79 is set to 64k
BLOCKSIZE := 64k

this is causing rootfs to be padded to 64k before the 0xdeadcode marker for JFFS2
then the mtd parser sees empty space before the marker and starts JFFS2 there instead

for example in my case, kernel + rootfs length is 0x440000 + 4 bytes for 0xdeadcode
on the next boot, rootfs_data partition shows as starting at 0x439000
after another reboot, the checksum fails since 0x439000 to 0x440000 was changed

this breaks the checksum that I'm trying to set in the uboot environment
and also, saving the configuration during sysupgrade is not possible...

I am guessing the best solution is to set all of tiny subtarget to BLOCKSIZE := 4k
but I am not sure the best way to do this

mpratt14 · 2021-02-22T04:01:35Z

nevermind...it has to be handled per device...

mpratt14 · 2021-02-22T18:53:13Z

@adschm ready for review again

refactored the sysupgrade script to save memory

added blocksize fix for the boards in tiny

moved the changes to ALLNET to the bottom so they are easier to drop
(in case they never get tested)

tested on all boards except EnstationAC and WAP02860AC

mpratt14 · 2021-02-22T19:42:42Z

updated commit message

adschm · 2021-02-23T12:44:33Z

ar724x_senao_failsafe_ap.dtsi

Why "failsafe"?

mpratt14 · 2021-02-28T00:04:58Z

right now we have

  IMAGE/factory.bin := append-kernel | pad-to $$$$(BLOCKSIZE) | append-rootfs | pad-rootfs | \
	check-size | senao-tar-gz $$$$(SENAO_IMGNAME)
  IMAGE/sysupgrade.bin := append-kernel | pad-to $$$$(BLOCKSIZE) | append-rootfs | \
	pad-rootfs | check-size | senao-tar-gz $$$$(SENAO_IMGNAME) | append-metadata

is this acceptable?

  IMAGE/sysupgrade.bin := $(IMAGE/factory.bin) | append-metadata

mpratt14 · 2021-03-03T05:48:26Z

@adschm Hi Adrian, I do not mean to "rush" the process, but I am hoping to get these changes into the new release branch after this. I believe this is the last set of changes that are needed for complete support that fully handles all cases with these boards.

Also as I said, you can drop the last commit if you want to leave that DTSI file alone.

As always, I appreciate your reviews, and if there is anything that needs to be changed, let me know and I will handle it immediately.

thanks :D

mpratt14 · 2021-03-06T23:02:48Z

edited commit title and rebased

mpratt14 · 2021-03-08T19:51:28Z

@blocktrron any comment on the first commit? do you like the idea?

mpratt14 · 2021-03-16T03:10:33Z

@adschm ping

mpratt14 · 2021-03-24T03:34:45Z

@adschm rebased and edited commit messages to be much more clear and concise

mpratt14 · 2021-03-31T00:41:51Z

@adschm may I please have a (final?) review

...and max flash offset The mtdsplit parser was recently refactored to allow the kernel to have custom image header magic. Let's also do this for the lzma-loader For example: When implemented together, this allows the kernel to "appear" to be a rootfs by OEM software in order to write an image that is actually kernel + rootfs. At the same time, it would boot to openwrt normally by setting the same magic in DTS. Both of the variables have a default value that is unchanged when not defined in the makefiles This has no effect on the size of the loader when lzma compressed. Signed-off-by: Michael Pratt <mcpratt@pm.me>

By using the same custom kernel header magic in both OKLI lzma-loader, DTS, and makefile this hack is not necessary anymore However, "rootfs" size and checksum must now be supplied by the factory.bin image through a script that is accepted by the OEM upgrade script. This is because Senao OEM scripts assume a squashfs header exists at the offset for the original "rootfs" partition which is actually the kernel + rootfs in this implementation, and takes size value from the header that would be there with hexdump, but this offset is now the uImage header instead. This frees up 1 eraseblock previously used by the "fakeroot" partition for bypassing the OEM image verification. Also, these Senao devices with a 'failsafe' partition and the tar-gz factory.bin platform would otherwise require flashing the new tar-gz sysupgrade.bin afterward. So this also prevents having to flash both images when starting from OEM or 'failsafe' the OEM upgrade script verifies the header magic numbers, but only the first two bytes. Example: [ "${magic_word_kernel}" = "2705" ] && [ "${magic_word_rootfs}" = "7371" -o "${magic_word_rootfs}" = "6873" ] && errcode="0" therefore picked the magic number 0x73714f4b which is 'sqOK' Signed-off-by: Michael Pratt <mcpratt@pm.me>

ath79/tiny kernel config has CONFIG_MTD_SPI_NOR_USE_4K_SECTORS=y from commit 05d3540 Because of this, these changes are required for 2 reasons: 1. Senao devices in ath79/tiny with a 'failsafe' partition and the tar.gz sysupgrade platform and a flash chip that supports 4k sectors will fail to reboot to openwrt after a sysupgrade. the stored checksum is made with the 64k blocksize length of the image to be flashed, and the actual checksum changes after flashing due to JFFS2 space being formatted within the length of the rootfs from the image example: 0x440000 length of kernel + rootfs (from sysupgrade.bin) 0x439000 offset of rootfs_data (from kernel log) 2. for boards with flash chips that support 4k sectors: saving configuration over sysupgrade is not possible because sysupgrade.tgz is appended at a 64k boundary and the mtd parser starts JFFS2 at a 4k boundary. for boards with flash chips that do not support 4k sectors: partitioning with 4k boundaries causes a boot loop from the mtd parser not finding kernel and rootfs. Also: Some of the Senao boards that belong in ath79/tiny, for example ENH202, have a flash chip that does not support 4k sectors (no SECT_4K symbol in upstream source). Because of this, partitioning must be different for these devices depending on the flash chip model detected by the kernel. Therefore: this creates 2 DTSI files to replace the single one with 64k partitioning for 4k and 64k partitioning respectively. Signed-off-by: Michael Pratt <mcpratt@pm.me>

Use a similar upgrade method for sysupgrade.bin, like factory.bin, for Senao boards with the tar.gz OEM upgrade platform, and 'failsafe' image which is loaded on checksum failure. This is inspired by the OEM upgrade script /etc/fwupgrade.sh and the existing platforms for dual-boot Senao boards. Previously, if the real kernel was damaged or missing the only way to recover was with UART serial console, because the OKLI lzma-loader is programmed to halt. uboot did not detect cases where kernel or rootfs is damaged and boots OKLI instead of the failsafe image, because the checksums stored in uboot environment did not include the real kernel and rootfs space. Now, the stored checksums include the space for both the lzma-loader, kernel, and rootfs. Therefore, these boards are now practically unbrickable. Also, the factory.bin and sysupgrade.bin are now the same, except for image metadata. This allows for flashing OEM image directly from openwrt as well as flashing openwrt image directly from OEM. Make 'loader' partition writable so that it can be updated during a sysupgrade. tested with ENS202EXT v1 EAP1200H EAP350 v1 EAP600 ECB350 v1 ECB600 ENH202 v1 Signed-off-by: Michael Pratt <mcpratt@pm.me>

This device is a Senao-based product using hardware and software from Senao with the tar-gz platform for factory.bin and checksum verification at boot time using variables stored in uboot environment and a 'failsafe' image when it fails. Extremely similar hardware/software to Engenius EAP1200H and other Engenius APs with qca955x Tested-by: Tomasz Maciej Nowak <tmn505@gmail.com> Signed-off-by: Michael Pratt <mcpratt@pm.me>

use qca955x_senao_loader.dtsi because it is the same hardware / partitioning and some cleanup Effects: nodes to match similar boards - keys - eth0 - pcie0 bumps SPI frequency to 40 MHz removes &pll node: the property is defined in qca955x.dtsi removes qca,no-eeprom: has no effect with mtd-cal-data property (also spelling) Tested-by: Tomasz Maciej Nowak <tmn505@gmail.com> Signed-off-by: Michael Pratt <mcpratt@pm.me>

pcie0 is the same for this generation of Senao APs while eth0, eth1, and wmac can differ the qca,no-eeprom property has no effect for the ath10k drivers Signed-off-by: Michael Pratt <mcpratt@pm.me>

mpratt14 · 2021-04-08T00:04:04Z

@ynezz this is ready for merge IMO. I have tested these commits on 10+ boards (some of which are not added yet).

Adrian gave me excellent and tough reviews and thanks to him, this PR is now in great condition. However, he has been silent for a month now (I assume he is busy with other things).

This should be in the stable release because of the added sysupgrade platform, and otherwise, there would be migration for the affected boards between this coming stable release and the next one. Also, this PR adds a factory.bin for a device that would otherwise require UART to flash.

In order for it to safely land in stable release without any issue, let's get it into master now so I can flash snapshots and verify that everything is ok....

thanks for your time

ynezz · 2021-04-08T07:11:17Z

Thanks! Pulled into my staging tree at https://git.openwrt.org/openwrt/staging/ynezz.git

adschm · 2021-04-10T17:13:47Z

Thanks for taking care, this was too big for me at the moment.

mpratt14 · 2021-04-11T12:10:36Z

@danielfdickinson looking at the old ar71xx sysupgrade platforms (allnet.sh), this Senao platform for factory.bin and sysupgrade should apply to your devices as well (CAP324, CR3000, CR5000)

it should be easy to add those to this....if you can verify that they have the same situation (OEM has a failsafe image thats booted when checksums fail, and do some testing...

these 2 commits are good examples, probably we can do it all in 1 commit though
37ea5d9
e800da9

danielfdickinson · 2021-04-12T20:27:36Z

@mpratt14 The CAP324, CR3000, and CR5000 had different bootloader and don't have a failsafe image, at least to my knowledge. I will try to make some time to look closer, but I don't think these commits will works with these devices.

mpratt14 · 2021-04-12T21:45:44Z

@danielfdickinson if that turns out to be true, I'm curious why it used the same sysupgrade script as these boards would use in the old target ar71xx. Is it possible that theres no failsafe image, but checksums are still verified at boot? and it does something else instead when it fails?

danielfdickinson · 2021-04-27T04:31:18Z

Since I added the ar71xx support, and I don't totally remember, my guess is that it was based on the CAP4200, even though the bootloader is different. I will try and make some time this month, but I'm pretty sure the CAP324 an CR3000/CR5000 do not check even check a checksum, I do know they don't have enough flash for a fallback (8M flash, fully utilized by the stock firmware for the CR3000/CR5000). The CAP324 had 16M flash but didn't split into a fallback.

danielfdickinson · 2021-06-30T09:20:24Z

Ok, so the PowerCloud stuff (CR3000, CR5000, and CAP324) don't have a failsafe. (Sorry for taking so long to get back to you). They used a u-boot that was modified from Senao's default u-boot. From what I can recall the reason for using the the same script many moons ago was that the boards were simply basing the sysupgrade on what worked for similar hardware. It worked, but was probably not ideal. IIRC they now use a generic sysupgrade (for ath79).

In any event I'm currently not using any of these devices, although I see someone is getting APXXX from Senao working based on the CAP324. Personally I would not take it badly if PowerCloud stuff were removed after the upcoming release, or even for this release, although at this late date there is always the chance there is someone else using it and who would be upset.

mpratt14 · 2021-06-30T18:13:50Z

@danielfdickinson ok no problem. my only question would be how the OEM sysupgrade worked in order to maybe have a factory.bin for that device

do you have an original upgrade image maybe? or a flash backup?

danielfdickinson · 2021-07-01T02:05:23Z

Ah, that I can answer. For the CR3000 and CR5000 the sysupgrade was entirely cloud-based -- there was no local firmware upgrade option. In the background it was a standard OpenWrt sysupgrade (the OEM firmware was OpenWrt-based not the usual Senao thing). For the CAP324 the upgrade was usually cloud-based but there was the option of using a local LuCI-based interface to do a sysupgrade as well (again OpenWrt-based firmware).

The details are in the Table of Hardware on the Wiki:

For the CAP324: https://openwrt.org/toh/powercloud_systems/cap324
For the CR5000: https://openwrt.org/toh/powercloud_systems/cr5000
For the CR3000: https://openwrt.org/toh/powercloud_systems/cr3000

HTH.

mpratt14 force-pushed the senao-fix-final branch from f08531d to e3c42b3 Compare February 15, 2021 07:36

adschm added the target/ath79 pull request/issue for ath79 target label Feb 15, 2021