Experts: Firmware flashing from internal file-system failed

[xuancong84]

Hi @danicampora @dpgeorge @jimmo @peterhinch
I have been trying to flashing firmware locally from image file on internal storage, but so far no success.

I am aware that 0x40200000-0x40300000 is mapped to the 1st 1MB of flash, so running any code from this region to flash firmware is going to fail because while flashing half-way, your code gets changed and will cause crash. Not only this, while flashing, any functions that you call cannot come from this memory region. Moreover, certain interrupt handlers may re-direct to this region upon handling special events, so you must disable watchdog, disable interrupt IRQ, etc. while flashing.

Taking care of all these, I have put my code in iram1 (40100000h-40108000h) using MP_FASTCODE and view disassembly to confirm that no function calls will ever reach 0x40200000-0x40300000. I will need to call spi_flash_erase_sector, spi_flash_read, spi_flash_write and uart_tx_one_char (for verbose), all these functions come from iram1. Even for reading the firmware image file, I digged out all the sectors and sector offsets for each fragment of the file, and did not use any filesystem function calls (which are all in 0x40200000-0x40300000).

Unfortunately, I can only flash the 1st portion (0x40200000-0x40209000) which is the boot-loader, the program hangs forever once writing into the flash region that is mapped to .irom0.text section, i.e., from 0x40209000 onwards.

Can someone explain why, is there some sort of memory protection or what? My firmware code is at https://github.com/xuancong84/micropython/ , the user-level Python code is at https://github.com/xuancong84/OpenSmartLight/blob/main/micropython/update.py

[andrewleech]

Could you please give more details about your build? What hardware, port, flash size, micropython version?

I know on STM that all the ISR functions are hard-coded to be in flash, including systick etc. I don't know which port you're using to know if this is likely a similar issue. Similarly on stm, the mboot bootloader has support for installing firmware from the flash filesystem already.

[xuancong84]

Thanks @andrewleech , I am running micropython master build, esp8266 port, 4M flash size, ESP8266_GENERIC.

You need to be a top-notch hardware&software expert of ESP8266 in order to answer this post. Because currently, nobody in the world can solve this OTA update issue without ping-ponging between two flash regions (which leads to reduced firmware space) within the 1st 1MB of flash storage, while my solution does not require so.

[andrewleech]

I see you're disabling WDT in your C code before flashing, but I don't see anything to disable all interrupts? I'm not sure if interrupts are going to be the problem but it might help (assuming they're not needed for the actual spiflash handling)?
https://github.com/xuancong84/micropython/blob/5c2332a605377fb55a5d170d2eb4b1736728e189/ports/esp8266/modesp.c#L430

There's also some discussion here suggesting that WDT can't really be disabled: #8597 that being said, perhaps that function will kill the HW watchdog, and if irq's are stopped then the software one will possibly also be stopped.
Or maybe you could just pat the watchdog in between each flash operation to ensure it's not getting in the way?

[andrewleech]

Also, flashing should not require IRQ because I can successfully flash the first few sectors (the bootloader region) with IRQ disabled, and verification is also successful.

I didn't think IRQ's haven't been disabled though? So bootloader flashing doesn't really say one way or the other?

[xuancong84]

The 2nd 1MB of flash is the LittleFS file-system, or you mean the 2nd half of the first 1MB?

[andrewleech]

I wasn't sure if the filesystem would start just there or not. I did mean flashing to there, it'd break the filesystem sure but maybe you could still see if a sector or two worked?

Though starting half way through the first 1mb would also be interesting, find out if it locks up from anything in that section being written, or just the first block.

[xuancong84]

I have tried to erase sectors (with verification that every byte in the sector must be 0xff after erase) backwards from the end of the 1st 1MB to the beginning, i.e., 0xfffff -> 0x0, (sector number 0xff -> 0x0, sector size is 4KB=0x1000). Interestingly, it crashes (with reset_cause=2, illegal opcode exception) after erasing 121 sectors, that is 495616 bytes from the end of the 1st 1MB, or 552960 bytes from the beginning of the flash. Moreover, the firmware image size is 651424. So, it turns out that erasing the 1st 32KB and the last ~100KB of the firmware image on the flash will not crash, but touching in between will cause crash.

The weird thing is that from the disassembly (using xtensa-lx106-elf-objdump) of my DFU function that is put inside iram1 (40100000h-40108000h), it never touches 0x40200000-0x40300000, which is mapped to the 1st 1MB of the flash that I am flashing. Moreover, with disabled WDT and IRQ, there should not be any chance to reach this forbidden zone during flashing. Yet, it still crashes.

[andrewleech]

That is interesting... my guess is that the illegal opcode is the 0xFF that's just been erased.... I too would have hoped that the disassembly looking like no flash addresses would be enough to know it's not being accessed, but perhaps there's a flash address being calculated and jumped to somewhere? Do you know about cache settings, I see mention of an IROM cache that could be involved?

There may be some interesting information hiding in https://www.github.com/RIOT-OS/RIOT/issues/16281 that could point at things to check too.

[xuancong84]

Thanks a lot @andrewleech !

I have eventually found out why:

It is because spi_flash_erase_sector calls spi_flash_erase_sector_check which is in the forbidden zone 0x40200000-0x40300000.

[andrewleech]

Good Find! I wonder if you can make your own replacement of this funtion that doesn't include the flash check?

[xuancong84]

Thanks for your reply!

By using the following replacement sector erasing function, I am able to erase (with verification) all the 256 sectors of the 1st 1MB of flash without crashing.

void MP_FASTCODE(ram_erase_sector)(size_t sector){
    uart_tx_one_char(0, '1');
    spi_flash_check_wr_protect();
    uart_tx_one_char(0, '2');
    Cache_Read_Disable_2();
    uart_tx_one_char(0, '3');
    SPIEraseSector(sector);
    uart_tx_one_char(0, '4');
    Cache_Read_Enable_2();
}

Moreover, I also found that all the three non-0x40200000 functions spi_flash_erase_sector, spi_flash_write, and spi_flash_read(), contain instructions that will call 0x40200000 functions in various manners, for checking status, protection, for feeding software WDT, etc. Even the mp_hal_delay_ms and mp_hal_delay_us functions calls 0x40200000 function if the delay duration is large enough.

Furthermore, some lowest-level functions such as Cache_Read_Disable_2(), Cache_Read_Enable_2(), and SPIEraseSector() are not documented in any SDK API documentations. I have to figure them out in a hard way. 😅

[xuancong84]

Hi @andrewleech ,
Finally gotten it working!!! 😁 (See 350b973)

To update firmware in place:

• copy firmware.bin into the internal filesystem, currently only support LFS2 (LittleFS2)
• import inisetup
• run inisetup.fwupdate('firmware.bin', erase_all=False, safe_check=True, verbose=True)

By default, file-system will not be touched unless you set erase_all=True.

Now, the problem is that I run out of iram0, need to drop espnow or link will fail. The strange thing is that in modespnow.c, none of the functions are defined as MP_FASTCODE, but they all go into iram0, why?

[andrewleech]

Congrats! That's really neat.

I'm not too sure about the ram use, is it definitely code going into ram there or communications buffers etc maybe?

You may be able to do other things like exciting any of your application code functions / tasks all the way back to main, then even sys.modules["my_application"] etc followed by gc.collect() to free up any application functions, code, globals, everything before starting the fw.update perhaps?

[xuancong84]

Solved. I migrated 3 non-time-critical functions of espnow from iram to irom to save the 100 bytes for my iram DFU function. Preparing for a pull request now-:)

[xuancong84]

The issue is solved in pull request #12429

[devbmv]

[xuancong84]

Hi @maik775 , did you post in the wrong discussion?