PlatformIO not recognising CONFIG_PARTITION_TABLE_OFFSET for debugger and upload

[cjstott94]

Configuration

Operating system:
Win 10
PlatformIO Version (platformio --version):
PlatformIO, version 5.0.1
framework-espidf 3.40100.200827 (4.1.0)

Description of problem

I'm checking out some extra features and logging of the bootloader and so the bootloader doesn't fit within the normal space allocated (0x1000 to 0x8000) and overflows to the partition table, and as app partitions need to be aligned by 0x10000 (64K) anyway, my main app is now at 0x20000 and so I put my partition table at 0x16000

When using this custom partitioning platformio doesn't seem to account for CONFIG_PARTITION_TABLE_OFFSET when uploading or using the debugger, resulting in an error related to the new offset.
Or I haven't been able to find the variable that platformio uses when generating the upload/debug commands.

I have managed to make some custom scripts that fix uploading for now, though I'm not sure how to do this for debugging

Steps to Reproduce

First tried only changing board_build.partitions = ...csv in platformio.ini

Then tried changing CONFIG_PARTITION_TABLE_OFFSET=0x16000:

• As build flag in platformio.ini
• via menuconfig (pio run -t menuconfig)
• via sdkconfig.defaults

Also tried replacing FLASH_EXTRA_IMAGES with a python script
env.Replace(FLASH_EXTRA_IMAGES=[ ('0x1000', '$BUILD_DIR\\bootloader.bin'), ('0x16000', '$BUILD_DIR\\partitions.bin')])

Actual Results

Uploading with esp-tool gives "Detected overlap at address: 0x8000 for file ...\partitions.bin"
Uploading with esp-prog works but results in a boot loop
Using debugger also results in a boot loop I think

print(env.subst('$UPLOADCMD') indeed shows wrong partition address of 0x8000 being used

Expected Results

partitions.bin is flashed to address as specified by CONFIG_PARTITION_TABLE_OFFSET
firmware.bin is flashed to the first app partition specified by CSV file (not sure if it's trying to do this or not)

The content of platformio.ini:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = espidf
debug_tool = esp-prog
upload_protocol = esp-prog ;for JTAG upload
board_build.embed_txtfiles =
  src/webpage.html
  src/server_root_cert.pem
board_build.partitions = partition_table_littleFs_factory_2ota.csv

; build_flags = 
;    -D CONFIG_PARTITION_TABLE_CUSTOM=y
;    -D CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partition_table_littleFs_factory_2ota.csv"
;    -D CONFIG_PARTITION_TABLE_FILENAME="partition_table_littleFs_factory_2ota.csv"
;    -D CONFIG_PARTITION_TABLE_OFFSET=0x16000
;    -D CONFIG_PARTITION_TABLE_MD5=y
;   -D CONFIG_BOOTLOADER_LOG_LEVEL_VERBOSE=y

The content of partition_table_littleFs_factory_2ota.csv:

#Name	Type	SubType	Offset	Size	Flags
nvs	data	nvs	0x17000	0x6000
otadata	data	ota	0x1D000	0x2000
phy_init	data	phy	0x1F000	0x1000
ota_0	app	ota_0	0x20000	0x1D0000
ota_1	app	ota_1	0x1F0000	0x1D0000
littlefs	data	spiffs	0x3C0000	256K

Script to replace esp-tool upload:

partition_offset = "0x16000"
firmware_offset = "0x20000"

env.Replace(
        UPLOADCMD=env.subst('$UPLOADER')
        + " --chip esp32 --port \"COM13\" --baud 1843200 "
        + "--before default_reset --after hard_reset write_flash "
        + "-z --flash_mode dio --flash_freq 40m --flash_size detect "
        + "0x1000 "
        + env.subst('$PROJECT_BUILD_DIR') + "\\"
        + env.subst('$PIOENV')+"\\bootloader.bin "
        + partition_offset+" "
        + env.subst('$PROJECT_BUILD_DIR') + "\\"
        + env.subst('$PIOENV')+"\\partitions.bin "
        + firmware_offset
        + env.subst('$PROJECT_BUILD_DIR') + "\\"
        + env.subst('$PIOENV')+"\\firmware.bin "
    )

Script to replace esp-prog upload:

env.Replace(
        UPLOADCMD="openocd -d1 -s \"" +
        env.subst('$PROJECT_PACKAGES_DIR') +
        "\\tool-openocd-esp32\\share\\openocd\\scripts\" "
        + "-f \"interface\\ftdi\\esp32_devkitj_v1.cfg\" -f \"board\\esp-wroom-32.cfg\" "

        + "-c \"program_esp32 "
        + env.subst('$PROJECT_BUILD_DIR').replace("\\", "/")+"/" +
        env.subst('$PIOENV').replace("\\", "/")+"/bootloader.bin "
        + "0x1000 verify\" "

        + "-c \"program_esp32 "
        + env.subst('$PROJECT_BUILD_DIR').replace("\\", "/") + "/" +
        env.subst('$PIOENV').replace("\\", "/")+"/partitions.bin "
        + partition_offset + " verify\" "

        + "-c \"program_esp32 "
        + env.subst('$PROJECT_BUILD_DIR').replace("\\", "/")+"/" +
        env.subst('$PIOENV').replace("\\", "/")+"/firmware.bin "
        + firmware_offset + " verify\" "

        + "-c \"reset run; shutdown\""
    )

[valeros]

There is new functionality in the dev branch that hasn't been released yet, including support for a custom partition table offset. See my comment #11 (comment). But keep in mind you still need to configure the file with partitions in the platformio.ini file.

[cjstott94]

Great, works fine now.
Maybe as a future feature, would be great if it could infer the partition table being 0x1000 below the first offset in CSV?
Also maybe grab the firmware app offset from the partition CSV?

I found and set the option:
board_upload.offset_address = 0x20000
As I'm using 0x20000 for my firmware offset (I'm just sort of testing out how much space I actually need ATM)
This works fine for esp-prog/debug upload , though it seems like it's hardcoded for esp-tool

[cjstott94]

Actually with the board_upload.offset_address it seems to work with esp-prog for the 'upload' target, though it's not working for the 'debug' target ie. when you want to step through the code

Debug target

** Programming Finished **
** Verify Started **
Info : Target halted. PRO_CPU: PC=0x400916EE (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x400916EE (active)    APP_CPU: PC=0x40000400 
read 3072 bytes from file .../Firmware/.pio/build/esp32dev/partitions.bin and flash bank 0 at offset 0x00016000 in 0.309680s (9.687 KiB/s)
contents match
read 3072 bytes from file .../Firmware/.pio/build/esp32dev/partitions.bin and flash bank 0 at offset 0x00016000 in 0.309680s (9.687 KiB/s)
contents match
** Verified OK **

...

** Programming Finished **
** Verify Started **
Info : Target halted. PRO_CPU: PC=0x400916EE (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x400916EE (active)    APP_CPU: PC=0x40000400 
read 631792 bytes from file .../Firmware/.pio/build/esp32dev/firmware.bin and flash bank 0 at offset 0x00010000 in 2.710952s (227.590 KiB/s)
contents match
read 631792 bytes from file .../Firmware/.pio/build/esp32dev/firmware.bin and flash bank 0 at offset 0x00010000 in 2.710952s (227.590 KiB/s)
contents match

which results in

Info : Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
Target halted. PRO_CPU: PC=0x40000400 (active)    APP_CPU: PC=0x40000400 
Hardware assisted breakpoint 1 at 0x400d31c0: file src\main.c, line 234.
PlatformIO: Initialization completed
PlatformIO: Resume the execution to `debug_init_break = thb app_main`
PlatformIO: More configuration options -> http://bit.ly/pio-debug
Error: esp32: Failed to read insn (-4)!
Error: esp32: Failed to add SW BP!
Error: can't add breakpoint: resource not available
Warning:
Cannot insert breakpoint 2.
Cannot access memory at address 0x400d3749

Command aborted.

when using upload target

** Programming Finished **
** Verify Started **
read 631792 bytes from file .pio\build\esp32dev\firmware.bin and flash bank 0 at offset 0x00020000 in 2.700883s (228.438 KiB/s)
contents match
** Verified OK **
** Programming Started **
auto erase enabled
wrote 28672 bytes from file ...ware\.pio\build\esp32dev/bootloader.bin in 1.002557s (27.929 KiB/s)
** Programming Finished **
** Verify Started **
read 27312 bytes from file ...\Firmware\.pio\build\esp32dev/bootloader.bin and flash bank 0 at offset 0x00001000 in 0.395537s (67.432 KiB/s)
contents match
** Verified OK **
** Programming Started **
auto erase enabled
wrote 4096 bytes from file ...\Firmware\.pio\build\esp32dev/partitions.bin in 0.629671s (6.353 KiB/s)
** Programming Finished **
** Verify Started **
read 3072 bytes from file ...\Firmware\.pio\build\esp32dev/partitions.bin 
and flash bank 0 at offset 0x00016000 in 0.323478s (9.274 KiB/s)
contents match
** Verified OK **