Make flash fails if ota_0 used as app partition

[nick-atmosphereiot]

Environment

• Development Kit: [AWS Lanyard]
• Core (if using chip or module): [ESP-WROOM-32]
• IDF version (git rev-parse --short HEAD to get the commit id.):
  4b91c82 (also in release 3.1)
• Development Env: [Make]
• Operating System: [Ubuntu]
• Power Supply: [USB]

Problem Description

make flash fails, thinking that the app is at 0x0. My partition table uses ota_0 instead of factory for the app location. If I change ota_0 to factory, make flash works just fine and detects the app at the correct space.

Here's the make flash output:

Flashing binaries to serial port /dev/ttyUSB0 (app at offset 0x0 )...
usage: esptool write_flash [-h] [--flash_freq {keep,40m,26m,20m,80m}]
                           [--flash_mode {keep,qio,qout,dio,dout}]
                           [--flash_size FLASH_SIZE]
                           [--spi-connection SPI_CONNECTION] [--no-progress]
                           [--verify] [--compress | --no-compress]
                           <address> <filename> [<address> <filename> ...]
esptool write_flash: error: argument <address> <filename>: Detected overlap at address: 0x1000 for file: /home/nick/Downloads/esptest4/build/bootloader/bootloader.bin
/home/nick/esp-idf/components/esptool_py/Makefile.projbuild:54: recipe for target 'flash' failed

Expected Behavior

It should detect ota_0 as an app space.

Actual Behavior

Make flash incorrectly reads the partition table, thinking the app is at 0x0

Steps to reproduce

Use this partition table:

# Name,   Type, SubType, Offset,   Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
#reserved,  data,   data,       0x0000,     4K
#bootldr,   app,    boot,       0x1000,     28K
#parttbl,   data,   part,       0x8000,     4K
nvs,        data,   nvs,        0x9000,     256K
otadata,    data,   ota,        ,     0x2000
ota_0,      app,    ota_0,      ,    1500K

Code to reproduce this issue

N/A

Debug Logs

See problem description for make flash log.

Uploaded partition and sdkconfig
esptest4.zip

[chegewara]

Like i said on forum, did you do make erase_flash before make flash?
If you work with ota its suggested to do make erase_flash each time to avoid weird behavior.

[nick-atmosphereiot]

Yes. I have done a make erase_flash. This error happens before it even tries to flash anything. I have not actually used OTA yet.

The error is that it reads the partition table and doesn't see that ota_0 is an app partition, and therefore it assumes that app is at 0x0 (or so it seems). According to the documentation, it is valid to use ota_0 as the main app partition to save space.

[chegewara]

Its true, i am using ota_0 all the time without factory partition. Could you use only make and then paste command python esptool.py .....

[nick-atmosphereiot]

This is from the latest commit on master, also the same for 3.1.

To flash all build output, run 'make flash' or:
python /home/nick/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 115200 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /home/nick/Downloads/esptest4/build/bootloader/bootloader.bin 0x0 /home/nick/Downloads/esptest4/build/atmosphere-project.bin 0x8000 /home/nick/Downloads/esptest4/build/partitions.bin

I believe the latest esptool is supposed to read the partition table to figure out where the app partition is.

[chegewara]

Here is the error:
0x0 /home/nick/Downloads/esptest4/build/atmosphere-project.bin
Even if esptool can read partition table (i dont know that) then you just overriding it with this command.
Change 0x0 to 0x50000 (i assume its still your ota partition start address).

[nick-atmosphereiot]

It's still a bug, since I shouldn't have to change that and run the command manually. If I use the factory type, then yeah that becomes 0x50000.

The point is that in previous versions, they had a configuration value to set this app offset. They took it out in the most recent release because they added smarts to the tool to figure that offset out.

[nick-atmosphereiot]

Looking at the partition_table component, I see this:

$(eval APP_OFFSET:=$(shell $(GET_PART_INFO) --type app --subtype factory --offset $(PARTITION_TABLE_BIN)))

This makes sense, because it's specifically looking for the factory subtype. I assume in my case, it's not finding that so it's defaulting to 0x0. I think the logic should, if it doesn't find the factory subtype, look for ota_0 instead.

[chegewara]

Ok, for testing purpose i compiled modified hello-world example and i can confirm that its somewhere bug because my flash command looks the same if there is no factory partition:

python /d/msys/home/admin/esp/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port com3 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size 2MB 0x1000 /home/admin/hello_world/build/bootloader/bootloader.bin 0x0 /home/admin/hello_world/build/hello-world.bin 0x8000 /home/admin/hello_world/build/partitions.bin

App bin is flashed from address 0x0. Here is my partition table:

# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x4000
otadata,  data, ota,     0xd000,  0x2000
phy_init, data, phy,     0xf000,  0x1000
ota_factory,  app,  ota_0,		, 0xB0000
ota,	  app,  ota_1,        	,  0x140000

[nick-atmosphereiot]

Here's a patch for esp-idf/components/partition_table/Makefile.projbuild that fixes the issue:

partition_table_get_info: $(PARTITION_TABLE_BIN)
	$(eval PHY_DATA_OFFSET:=$(shell $(GET_PART_INFO) --type data --subtype phy --offset $(PARTITION_TABLE_BIN)))
	$(eval APP_OFFSET:=$(shell $(GET_PART_INFO) --type app --subtype factory --offset $(PARTITION_TABLE_BIN)))
	$(eval OTA_0_OFFSET:=$(shell $(GET_PART_INFO) --type app --subtype ota_0 --offset $(PARTITION_TABLE_BIN)))
	$(eval APP_OFFSET=$(if $(filter $(APP_OFFSET), 0x0), $(OTA_0_OFFSET), $(APP_OFFSET)))
export APP_OFFSET
export PHY_DATA_OFFSET

It's a little tricky to do this within Makefiles, but that will essentially check to see if APP_OFFSET is 0x0. If so, it tries to use the ota_0 partition instead. Needed to use some filter magic for this one...

Full version of file attached.
Makefile.projbuild.zip

This might not be the best way to solve the problem, since my knowledge of the partition component is limited.

[fermuch]

I just ran into the same issue and can confirm @nick-atmosphereiot 's patch fixes it.

[projectgus]

Thanks for reporting this, folks.

I think the best fix will be to add some kind of --fallback option to parttool.py, which uses the same fallback logic as the bootloader (ie if factory doesn't exist, try ota_0, then ota_1, etc, etc). Will do this ASAP.