Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cant build Encryption flash, always partition orverlap or error (IDFGH-2344) #4474

Closed
filzek opened this issue Dec 11, 2019 · 8 comments
Closed

Comments

@filzek
Copy link

filzek commented Dec 11, 2019

  • Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Project version: v4.1-dev-1086-g93a8603c5-dirty
    -- Building ESP-IDF components for target esp32

-- Could NOT find Perl (missing: PERL_EXECUTABLE)
Traceback (most recent call last):
File "C:/Users/dev/Desktop/esp-idf/components/partition_table/parttool.py", line 98, in init
partition_table = gen.PartitionTable.from_binary(f.read())
File "C:\Users\dev\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 231, in from_binary
result.append(PartitionDefinition.from_binary(data))
File "C:\Users\dev\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 383, in from_binary
raise InputError("Invalid magic bytes (%r) for partition definition" % magic)
gen_esp32part.InputError: Invalid magic bytes (b'nv') for partition definition

seens that partitions overlap and problems, trying to do everything but always return erro.

@github-actions github-actions bot changed the title Cant build Encryption flash, always partition orverlap or error Cant build Encryption flash, always partition orverlap or error (IDFGH-2344) Dec 11, 2019
@projectgus
Copy link
Contributor

Hi @filzek ,

Can you please give a little more detail: which configuration do you have set for flash encryption? what partition configuration do you have set, if it's a custom partition table then you share the contents of the CSV please? Also, what is the full output from the build both before and after the error output?

Thanks,

Angus

@filzek
Copy link
Author

filzek commented Dec 12, 2019

send in private e-mail the files you requested

@chegewara
Copy link
Contributor

chegewara commented Dec 12, 2019

Maybe this is the case:
https://docs.espressif.com/projects/esp-idf/en/latest/security/secure-boot.html#bootloader-size

Please check bootloader size.

@projectgus
Copy link
Contributor

projectgus commented Dec 16, 2019

Hi @filzek,

Thanks for sending those files and logs. Will reply here as this may help other people who find similar problems.

gen_esp32part.InputError: Invalid magic bytes (b'\r\n') for partition definition

You can ignore this error, it's a shortcoming in parttool.py that this is logged. Will fix.

gen_esp32part.InputError: CSV Error: First partition offset 0x9000 overlaps end of partition table 0xa000

If partition table offset is set to 0x9000, then the custom partition file (partitions.csv in your case) can't specify any partition at an offset less than 0xa000 - otherwise the partition collides with the partition table (which is 0x1000 bytes long).

Please check the contents of your partitions.csv against the partition offset value set in sdkconfig.

@filzek
Copy link
Author

filzek commented Dec 16, 2019

this is not the true problem, as even if try to build the example in esp-idf\examples\security\flash_encryption it return the same error.

we did try to full remove esp-idf, uninstall esp-idf tool, and we reinstalled everything.

still same error.

now also, the idf.py menuconfig doesnt work.

C:\Users\filze\Desktop\esp-idf\examples\security\flash_encryption>idf.py menuconfig
Executing action: menuconfig
Traceback (most recent call last):
File "C:\Users\filze\Desktop\esp-idf\tools\idf.py", line 764, in
main()
File "C:\Users\filze\Desktop\esp-idf\tools\idf.py", line 682, in main
cli(sys.argv[1:], prog_name=PROG)
File "C:\Users\filze.espressif\python_env\idf4.1_py3.7_env\lib\site-packages\click\core.py", line 764, in call
return self.main(*args, **kwargs)
File "C:\Users\filze.espressif\python_env\idf4.1_py3.7_env\lib\site-packages\click\core.py", line 717, in main
rv = self.invoke(ctx)
File "C:\Users\filze.espressif\python_env\idf4.1_py3.7_env\lib\site-packages\click\core.py", line 1164, in invoke
return _process_result(rv)
File "C:\Users\filze.espressif\python_env\idf4.1_py3.7_env\lib\site-packages\click\core.py", line 1102, in _process_result
**ctx.params)
File "C:\Users\filze.espressif\python_env\idf4.1_py3.7_env\lib\site-packages\click\core.py", line 555, in invoke
return callback(*args, **kwargs)
File "C:\Users\filze\Desktop\esp-idf\tools\idf.py", line 614, in execute_tasks
task.run(ctx, global_args, task.action_args)
File "C:\Users\filze\Desktop\esp-idf\tools\idf.py", line 207, in run
self.callback(self.name, context, global_args, **action_args)
File "C:\Users\filze\Desktop\esp-idf\tools\idf_py_actions\core_ext.py", line 40, in menuconfig
os.environ['MENUCONFIG_STYLE'] = style.encode(sys.getfilesystemencoding() or 'utf-8')
File "C:\Users\filze.espressif\python_env\idf4.1_py3.7_env\lib\os.py", line 683, in setitem
value = self.encodevalue(value)
File "C:\Users\filze.espressif\python_env\idf4.1_py3.7_env\lib\os.py", line 739, in check_str
raise TypeError("str expected, not %s" % type(value).name)
TypeError: str expected, not bytes

C:\Users\filze\Desktop\esp-idf\examples\security\flash_encryption>

@filzek filzek closed this as completed Dec 16, 2019
@filzek
Copy link
Author

filzek commented Dec 16, 2019

Problem still, when following the example in flash_encryption the problem stills happen.

C:\Users\filze\Desktop\esp-idf\examples\security\flash_encryption>idf.py build flash monitor
Executing action: all (aliases: build)
Running ninja in directory c:\users\filze\desktop\esp-idf\examples\security\flash_encryption\build
Executing "ninja all"...
[0/1] Re-running CMake...
-- ccache will be used for faster recompilation
-- Project version: v4.1-dev-1533-g9dbb7526c-dirty
-- Building ESP-IDF components for target esp32
Loading defaults file C:/Users/filze/Desktop/esp-idf/examples/security/flash_encryption/sdkconfig.defaults...
Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 98, in init
partition_table = gen.PartitionTable.from_binary(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 231, in from_binary
result.append(PartitionDefinition.from_binary(data))
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 383, in from_binary
raise InputError("Invalid magic bytes (%r) for partition definition" % magic)
gen_esp32part.InputError: Invalid magic bytes (b'# ') for partition definition

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 338, in
main()
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 311, in main
target = ParttoolTarget(**target_args)
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 102, in init
partition_table = gen.PartitionTable.from_csv(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 122, in from_csv
% (e.offset, last_end))
gen_esp32part.InputError: CSV Error: First partition offset 0x9000 overlaps end of partition table 0xa000
Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 98, in init
partition_table = gen.PartitionTable.from_binary(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 231, in from_binary
result.append(PartitionDefinition.from_binary(data))
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 383, in from_binary
raise InputError("Invalid magic bytes (%r) for partition definition" % magic)
gen_esp32part.InputError: Invalid magic bytes (b'# ') for partition definition

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 338, in
main()
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 311, in main
target = ParttoolTarget(**target_args)
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 102, in init
partition_table = gen.PartitionTable.from_csv(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 122, in from_csv
% (e.offset, last_end))
gen_esp32part.InputError: CSV Error: First partition offset 0x9000 overlaps end of partition table 0xa000
-- Adding linker script C:/Users/filze/Desktop/esp-idf/examples/security/flash_encryption/build/esp-idf/esp32/esp32_out.ld
-- Adding linker script C:/Users/filze/Desktop/esp-idf/components/esp32/ld/esp32.project.ld.in
-- Adding linker script C:/Users/filze/Desktop/esp-idf/components/esp32/ld/esp32.peripherals.ld
-- Adding linker script C:/Users/filze/Desktop/esp-idf/components/esp_rom/esp32/ld/esp32.rom.ld
-- Adding linker script C:/Users/filze/Desktop/esp-idf/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld
-- Adding linker script C:/Users/filze/Desktop/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld
-- Adding linker script C:/Users/filze/Desktop/esp-idf/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld
-- Adding linker script C:/Users/filze/Desktop/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld
Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 98, in init
partition_table = gen.PartitionTable.from_binary(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 231, in from_binary
result.append(PartitionDefinition.from_binary(data))
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 383, in from_binary
raise InputError("Invalid magic bytes (%r) for partition definition" % magic)
gen_esp32part.InputError: Invalid magic bytes (b'# ') for partition definition

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 338, in
main()
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 311, in main
target = ParttoolTarget(**target_args)
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 102, in init
partition_table = gen.PartitionTable.from_csv(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 122, in from_csv
% (e.offset, last_end))
gen_esp32part.InputError: CSV Error: First partition offset 0x9000 overlaps end of partition table 0xa000
CMake Error at ../../../components/esptool_py/project_include.cmake:203 (file):
file RELATIVE_PATH must be passed a full path to the file: FLASH_IN_PROJECT
Call Stack (most recent call first):
../../../components/esptool_py/CMakeLists.txt:62 (esptool_py_flash_project_args)

CMake Error at ../../../components/esptool_py/project_include.cmake:203 (file):
file RELATIVE_PATH must be passed a full path to the file:
FLASH_FILE_TEMPLATE
Call Stack (most recent call first):
../../../components/esptool_py/CMakeLists.txt:66 (esptool_py_flash_project_args)

Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 98, in init
partition_table = gen.PartitionTable.from_binary(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 231, in from_binary
result.append(PartitionDefinition.from_binary(data))
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 383, in from_binary
raise InputError("Invalid magic bytes (%r) for partition definition" % magic)
gen_esp32part.InputError: Invalid magic bytes (b'# ') for partition definition

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 338, in
main()
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 311, in main
target = ParttoolTarget(**target_args)
File "C:/Users/filze/Desktop/esp-idf/components/partition_table/parttool.py", line 102, in init
partition_table = gen.PartitionTable.from_csv(f.read())
File "C:\Users\filze\Desktop\esp-idf\components\partition_table\gen_esp32part.py", line 122, in from_csv
% (e.offset, last_end))
gen_esp32part.InputError: CSV Error: First partition offset 0x9000 overlaps end of partition table 0xa000
-- Components: app_trace app_update asio bootloader bootloader_support bt cbor coap console cxx driver efuse esp-tls esp32 esp_adc_cal esp_common esp_eth esp_event esp_gdbstub esp_http_client esp_http_server esp_https_ota esp_https_server esp_local_ctrl esp_netif esp_ringbuf esp_rom esp_serial_slave_link esp_websocket_client esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos heap idf_test jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl partition_table perfmon protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport tcpip_adapter ulp unity vfs wear_levelling wifi_provisioning wpa_supplicant xtensa
-- Component paths: C:/Users/filze/Desktop/esp-idf/components/app_trace C:/Users/filze/Desktop/esp-idf/components/app_update C:/Users/filze/Desktop/esp-idf/components/asio C:/Users/filze/Desktop/esp-idf/components/bootloader C:/Users/filze/Desktop/esp-idf/components/bootloader_support C:/Users/filze/Desktop/esp-idf/components/bt C:/Users/filze/Desktop/esp-idf/components/cbor C:/Users/filze/Desktop/esp-idf/components/coap C:/Users/filze/Desktop/esp-idf/components/console C:/Users/filze/Desktop/esp-idf/components/cxx C:/Users/filze/Desktop/esp-idf/components/driver C:/Users/filze/Desktop/esp-idf/components/efuse C:/Users/filze/Desktop/esp-idf/components/esp-tls C:/Users/filze/Desktop/esp-idf/components/esp32 C:/Users/filze/Desktop/esp-idf/components/esp_adc_cal C:/Users/filze/Desktop/esp-idf/components/esp_common C:/Users/filze/Desktop/esp-idf/components/esp_eth C:/Users/filze/Desktop/esp-idf/components/esp_event C:/Users/filze/Desktop/esp-idf/components/esp_gdbstub C:/Users/filze/Desktop/esp-idf/components/esp_http_client C:/Users/filze/Desktop/esp-idf/components/esp_http_server C:/Users/filze/Desktop/esp-idf/components/esp_https_ota C:/Users/filze/Desktop/esp-idf/components/esp_https_server C:/Users/filze/Desktop/esp-idf/components/esp_local_ctrl C:/Users/filze/Desktop/esp-idf/components/esp_netif C:/Users/filze/Desktop/esp-idf/components/esp_ringbuf C:/Users/filze/Desktop/esp-idf/components/esp_rom C:/Users/filze/Desktop/esp-idf/components/esp_serial_slave_link C:/Users/filze/Desktop/esp-idf/components/esp_websocket_client C:/Users/filze/Desktop/esp-idf/components/esp_wifi C:/Users/filze/Desktop/esp-idf/components/espcoredump C:/Users/filze/Desktop/esp-idf/components/esptool_py C:/Users/filze/Desktop/esp-idf/components/expat C:/Users/filze/Desktop/esp-idf/components/fatfs C:/Users/filze/Desktop/esp-idf/components/freemodbus C:/Users/filze/Desktop/esp-idf/components/freertos C:/Users/filze/Desktop/esp-idf/components/heap C:/Users/filze/Desktop/esp-idf/components/idf_test C:/Users/filze/Desktop/esp-idf/components/jsmn C:/Users/filze/Desktop/esp-idf/components/json C:/Users/filze/Desktop/esp-idf/components/libsodium C:/Users/filze/Desktop/esp-idf/components/log C:/Users/filze/Desktop/esp-idf/components/lwip C:/Users/filze/Desktop/esp-idf/examples/security/flash_encryption/main C:/Users/filze/Desktop/esp-idf/components/mbedtls C:/Users/filze/Desktop/esp-idf/components/mdns C:/Users/filze/Desktop/esp-idf/components/mqtt C:/Users/filze/Desktop/esp-idf/components/newlib C:/Users/filze/Desktop/esp-idf/components/nghttp C:/Users/filze/Desktop/esp-idf/components/nvs_flash C:/Users/filze/Desktop/esp-idf/components/openssl C:/Users/filze/Desktop/esp-idf/components/partition_table C:/Users/filze/Desktop/esp-idf/components/perfmon C:/Users/filze/Desktop/esp-idf/components/protobuf-c C:/Users/filze/Desktop/esp-idf/components/protocomm C:/Users/filze/Desktop/esp-idf/components/pthread C:/Users/filze/Desktop/esp-idf/components/sdmmc C:/Users/filze/Desktop/esp-idf/components/soc C:/Users/filze/Desktop/esp-idf/components/spi_flash C:/Users/filze/Desktop/esp-idf/components/spiffs C:/Users/filze/Desktop/esp-idf/components/tcp_transport C:/Users/filze/Desktop/esp-idf/components/tcpip_adapter C:/Users/filze/Desktop/esp-idf/components/ulp C:/Users/filze/Desktop/esp-idf/components/unity C:/Users/filze/Desktop/esp-idf/components/vfs C:/Users/filze/Desktop/esp-idf/components/wear_levelling C:/Users/filze/Desktop/esp-idf/components/wifi_provisioning C:/Users/filze/Desktop/esp-idf/components/wpa_supplicant C:/Users/filze/Desktop/esp-idf/components/xtensa
-- Configuring incomplete, errors occurred!
See also "C:/Users/filze/Desktop/esp-idf/examples/security/flash_encryption/build/CMakeFiles/CMakeOutput.log".
FAILED: build.ninja
C:\Users\filze.espressif\tools\cmake\3.13.4\bin\cmake.exe -SC:\Users\filze\Desktop\esp-idf\examples\security\flash_encryption -BC:\Users\filze\Desktop\esp-idf\examples\security\flash_encryption\build
ninja: error: rebuilding 'build.ninja': subcommand failed
ninja failed with exit code 1

C:\Users\filze\Desktop\esp-idf\examples\security\flash_encryption>

cant solve the problem

@filzek filzek reopened this Dec 16, 2019
@filzek
Copy link
Author

filzek commented Dec 16, 2019

We figure out all the problem, its beacuse its missing a basic information about the true offset for the initial partition table in menuconfig and the correct initial in the partition file, so, if fix both it will work great.

To solve the problem, the example partitions_example.csv need to be fixed

Name, Type, SubType, Offset, Size, Flags

nvs, data, nvs, 0xaa00, 0x6000, #the 0xaa00 example was incorrect related to 0x9000

Extra partition to demonstrate reading/writing of encrypted flash

storage, data, 0xff, 0x10A00, 0x1000, encrypted
factory, app, factory, 0x20000, 1M, encrypted

So, a mistake to increase both numbers in CSV and in menuconfig. Menuconfig 0x9000 is perfect, so, but in CSX it need to be at least 0xaa00 to do not colide.

Example and messages are showing incorrect. doing it will build and boot.

@filzek filzek closed this as completed Dec 16, 2019
@projectgus
Copy link
Contributor

@filzek Glad you figured out. Suggest aligning your partition starts to multiples of 0x1000, ie use 0xa000 instead of 0xaa00 and 0x10000 instead of 0x10a00.

Having first partition at 0xa000 with partition table at 0x9000 will not collide.

espressif-bot pushed a commit that referenced this issue Jan 6, 2020
Previous version used exception handling for program control flow,
which makes for long and confusing backtrace messages if an error
occurs while parsing the CSV (as it also prints the phony error that
occurred while trying to parse as a binary, then prints
"During handling of the above exception, another exception occurred:",
then prints the real error).

Use the same heuristic that is applied in gen_esp32_part, instead.

Also, avoid printing the entire backtrace if the error is a gen_esp32_part
InputError, same as gen_esp32part does.

Found while looking into #4474
espressif-bot pushed a commit that referenced this issue Feb 12, 2020
Previous version used exception handling for program control flow,
which makes for long and confusing backtrace messages if an error
occurs while parsing the CSV (as it also prints the phony error that
occurred while trying to parse as a binary, then prints
"During handling of the above exception, another exception occurred:",
then prints the real error).

Use the same heuristic that is applied in gen_esp32_part, instead.

Also, avoid printing the entire backtrace if the error is a gen_esp32_part
InputError, same as gen_esp32part does.

Found while looking into #4474
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants