mbed framework is incompatible with pyelftools-0.30 which is installed by pio

[rishyak]

Configuration

Operating system: Arch Linux

PlatformIO Version: 6.1.13

Description of problem

PIO fails to initialise with certain boards because of a pyelftools dependency.

Steps to Reproduce

1. Create a new mbed project with any mbed-supported board
2. It will fail

Actual Results

Failure

CONFIGURATION: https://docs.platformio.org/page/boards/nordicnrf52/nrf52840_dk.html
PLATFORM: Nordic nRF52 (10.1.0) > Nordic nRF52840-DK
HARDWARE: NRF52840 64MHz, 256KB RAM, 1MB Flash
DEBUG: Current (jlink) On-board (cmsis-dap, jlink) External (blackmagic, stlink)
PACKAGES: 
 - framework-mbed @ 6.61700.230818 (6.17.0) 
 - tool-sreccat @ 1.164.0 (1.64) 
 - toolchain-gccarmnoneeabi @ 1.90201.191206 (9.2.1)
ModuleNotFoundError: No module named 'elftools.common.py3compat':
  File "/home/rishyak/.platformio/penv/lib/python3.11/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 609:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/rishyak/.platformio/platforms/nordicnrf52/builder/main.py", line 195:
    target_elf = env.BuildProgram()
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/rishyak/.platformio/penv/lib/python3.11/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/rishyak/.platformio/penv/lib/python3.11/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/rishyak/.platformio/penv/lib/python3.11/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 673:
    return method(*args, **kw)
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 609:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/rishyak/.platformio/platforms/nordicnrf52/builder/frameworks/mbed.py", line 35:
    SConscript(
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 673:
    return method(*args, **kw)
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 609:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/rishyak/.platformio/packages/tool-scons/scons-local-4.6.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/rishyak/.platformio/packages/framework-mbed/platformio/platformio-build.py", line 61:
    from pio_mbed_adapter import PlatformioMbedAdapter
  File "/home/rishyak/.platformio/packages/framework-mbed/platformio/pio_mbed_adapter.py", line 23:
    from tools.build_api import prepare_toolchain, UPDATE_WHITELIST
  File "/home/rishyak/.platformio/packages/framework-mbed/tools/build_api.py", line 36:
    from .arm_pack_manager import Cache
  File "/home/rishyak/.platformio/packages/framework-mbed/tools/arm_pack_manager/__init__.py", line 29:
    from tools.flash_algo import PackFlashAlgo
  File "/home/rishyak/.platformio/packages/framework-mbed/tools/flash_algo/__init__.py", line 33:
    from elftools.common.py3compat import bytes2str
========================== [FAILED] Took 0.15 seconds ==========================

Expected Results

It doesn't fail.

Additional info

Downgrading pyelftools to 0.29 fixes the issue. Seems like you're suffering from the same issue pwntools did in September 2023.

Seems like PIO installs a version 0.30 which moves bytes2str to elftools.common.utils. Frameworks use the same virtual environment and causes failure in the case of mbed.

Began tracking this issue when I got this.

[ivankravets]

Thanks for the report. Please re-test with pio upgrade --dev.

[lakshanwijerathne]

this issue was regenerated for pyelftools 0.31. should check this also.

[ivankravets]

Open PlatformIO Core CLI and type pip install pyelftools==0.29

[lakshanwijerathne]

hi Ivan Kravets, Did it already and sorted it out thanks to this (issue #4834 <#4834>). worth mentioning, that is why I made the comment. however, thank you for giving me this support. thank you.

…

On Tue, 26 Mar 2024 at 15:31, Ivan Kravets ***@***.***> wrote: Open PlatformIO Core CLI and type pip install pyelftools==0.29 — Reply to this email directly, view it on GitHub <#4834 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV5RGDESWCBLB7US5V5LKE3Y2E2OFAVCNFSM6AAAAABB3WQA3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRQGAYDAOBXGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[Limhes]

When I try to fix this issue by downgrading to pyElfTools v0.29, I get the following error:

Processing nucleo_f767zi (platform: ststm32; board: nucleo_f767zi; framework: mbed)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/ststm32/nucleo_f767zi.html
PLATFORM: ST STM32 (12.1.1) > ST Nucleo F767ZI
HARDWARE: STM32F767ZIT6 216MHz, 512KB RAM, 2MB Flash
DEBUG: Current (stlink) On-board (stlink) External (blackmagic, cmsis-dap, jlink)
PACKAGES: 
 - framework-mbed @ 6.60900.220911 (6.9.0) 
 - toolchain-gccarmnoneeabi @ 1.90201.191206 (9.2.1)
ModuleNotFoundError: No module named 'imp':
  File "/home/reneb/.platformio/penv/lib/python3.12/site-packages/platformio/builder/main.py", line 173:
    env.SConscript("$BUILD_SCRIPT")
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 612:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/reneb/.platformio/platforms/ststm32/builder/main.py", line 120:
    target_elf = env.BuildProgram()
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/reneb/.platformio/penv/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 61:
    env.ProcessProgramDeps()
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/reneb/.platformio/penv/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 121:
    env.BuildFrameworks(env.get("PIOFRAMEWORK"))
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Util/envs.py", line 251:
    return self.method(*nargs, **kwargs)
  File "/home/reneb/.platformio/penv/lib/python3.12/site-packages/platformio/builder/tools/piobuild.py", line 347:
    SConscript(env.GetFrameworkScript(name), exports="env")
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 676:
    return method(*args, **kw)
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 612:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/reneb/.platformio/platforms/ststm32/builder/frameworks/mbed.py", line 35:
    SConscript(
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 676:
    return method(*args, **kw)
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 612:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/home/reneb/.platformio/packages/tool-scons/scons-local-4.7.0/SCons/Script/SConscript.py", line 279:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/reneb/.platformio/packages/framework-mbed/platformio/platformio-build.py", line 61:
    from pio_mbed_adapter import PlatformioMbedAdapter
  File "/home/reneb/.platformio/packages/framework-mbed/platformio/pio_mbed_adapter.py", line 23:
    from tools.build_api import prepare_toolchain, UPDATE_WHITELIST
  File "/home/reneb/.platformio/packages/framework-mbed/tools/build_api.py", line 48:
    from .libraries import Library
  File "/home/reneb/.platformio/packages/framework-mbed/tools/libraries.py", line 25:
    from tools.tests import TEST_MBED_LIB, DEFAULT_SUPPORT
  File "/home/reneb/.platformio/packages/framework-mbed/tools/tests.py", line 18:
    from past.builtins import cmp
  File "/home/reneb/.platformio/packages/framework-mbed/platformio/package_deps/py3/past/builtins/__init__.py", line 54:
    from past.builtins.misc import (apply, chr, cmp, execfile, intern, oct,
  File "/home/reneb/.platformio/packages/framework-mbed/platformio/package_deps/py3/past/builtins/misc.py", line 45:
    from imp import reload
============================================================================= [FAILED] Took 0.33 seconds =============================================================================

 *  The terminal process "platformio 'run', '--environment', 'nucleo_f767zi'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it.