ESPLoader may fail to release serial port if caller did not use 'with' (ESPTOOL-1062)

[424778940z]

Operating System

Windows 10 LTSC 2021

Esptool Version

v4.8.1

Python Version

Python 3.11.2

Full Esptool Command Line that Was Run

N/A, used as library import

Esptool Output

Not from esptool but caller script

PS C:\Users\localadmin\Downloads\projectName\firmware\components\esp-cryptoauthlib\esp_cryptoauth_utility> python .\secure_cert_mfg.py --port com6 --target_chip esp32 --type --i2c-sda-pin 26 --i2c-scl-pin 25
C:\Users\localadmin\Downloads\projectName\firmware\components\esp-cryptoauthlib\esp_cryptoauth_utility\sample_bins\secure_cert_mfg_esp32.bin
Connecting...
.
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting...
Failed to get PID of a device on com6, using standard reset sequence.
..
Detecting chip type... ESP32
Chip detected
RAM boot...
Downloading 66312 bytes at 3ffb0000... done!
Downloading 520 bytes at 3ffc1950... done!
Downloading 169740 bytes at 40080000... done!
Downloading 20 bytes at 400a970c... done!
All segments done, executing at 40082668
Time required to load the app into the RAM = 10.692219972610474s
Traceback (most recent call last):
  File "C:\Users\localadmin\Downloads\projectName\firmware\components\esp-cryptoauthlib\esp_cryptoauth_utility\secure_cert_mfg.py", line 245, in <module>
    main()
  File "C:\Users\localadmin\Downloads\projectName\firmware\components\esp-cryptoauthlib\esp_cryptoauth_utility\secure_cert_mfg.py", line 125, in main
    init_mfg = hs.serial.cmd_interpreter(port=args.port)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\localadmin\Downloads\projectName\firmware\components\esp-cryptoauthlib\esp_cryptoauth_utility\helper_scripts\serial.py", line 43, in __init__    
    self.port.open()
  File "C:\_home\_dev\.espressif\python_env\idf5.4_py3.11_env\Lib\site-packages\serial\serialwin32.py", line 64, in open
    raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError()))
serial.serialutil.SerialException: could not open port 'com6': PermissionError(13, '拒绝访问。', None, 5)

What is the Expected Behaviour?

Still not from esptool but caller script

PS C:\Users\localadmin\Downloads\projectName\firmware\components\esp-cryptoauthlib\esp_cryptoauth_utility> python .\secure_cert_mfg.py --port COM6 --target_chip esp32 --type --i2c-sda-pin 26 --i2c-scl-pin 25
C:\Users\localadmin\Downloads\projectName\firmware\components\esp-cryptoauthlib\esp_cryptoauth_utility\sample_bins\secure_cert_mfg_esp32.bin
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32
Chip detected
RAM boot...
Downloading 66312 bytes at 3ffb0000... done!
Downloading 520 bytes at 3ffc1950... done!
Downloading 169740 bytes at 40080000... done!
Downloading 20 bytes at 400a970c... done!
All segments done, executing at 40082668
Time required to load the app into the RAM = 10.68584942817688s
Wait for init
2nd stage bootloader

I (10910) boot: compile time Jul 10 2023 19:50:59

I (10911) boot: Multicore bootloader

I (10915) boot.esp32: SPI Speed : 40MHz

I (10920) boot.esp32: SPI Mode : SLOW READ

More Information

The issue is on this line

Serial port is only released if __exit__() is called, but this function is for contex manager api in python, which seems won't be called if caller use yyy = xxx.open() instead of with xxx.open() as yyy: , thus the serial port being held and causing anything after it to fail

Caller is here, which is also ESP provided code

Append esp.__exit__(None,None,None) to load_app_stub() to force it call ESPLoader.__exit__() has been tested works as a workaround, but proper fix should be implemented

Other Steps to Reproduce

follow this document setup a basic venv with requirements.txt, then run any command from secure_cert_mfg.py will cause the error

[github-actions]

Hi @424778940z! Please be aware that (1) the following suggestions are generated by a bot and haven't been fact-checked by Espressif Systems, (2) burning eFuses and enabling security features are irreversible operations and can damage your ESP32 device. We hope that this message will help you until an Espressif Engineer looks at your issue.

---

Based on the information provided, this appears to be a legitimate issue with esptool's handling of serial port resources when used as a library. Let me analyze the problem:

Issue Analysis

The problem occurs when esptool is used as a Python module rather than a command-line tool. Specifically:

1. When using esptool's Python API to load an application into ESP32 RAM, the serial port isn't properly released if the caller doesn't use Python's context manager pattern (with statement).

2. The error message shows that after esptool loads the application into RAM, a subsequent attempt to open the same serial port fails with "PermissionError(13, '拒绝访问。', None, 5)" (Access Denied).

3. The user identified the issue in esptool/loader.py where the serial port is only released when __exit__() is called, which only happens when using the context manager pattern.

Relevant API Information

Looking at the esptool Python API documentation:

esptool.cmds.load_ram(esp: ESPLoader, input: Union[str, bytes, IO[bytes]]) -> None

This function loads firmware into RAM and executes it on the ESP device, but doesn't explicitly close the serial connection afterward Esptool Scripting.

Recommended Solution

The user's suggested workaround of adding esp.__exit__(None,None,None) to load_app_stub() is a valid temporary fix, but a proper solution would be one of the following:

1. Modify the esptool library to explicitly close the serial port after operations like load_ram() complete, regardless of whether a context manager was used.

2. Update the caller code in esp-cryptoauthlib to use the context manager pattern when working with esptool.

3. Add explicit cleanup code in the caller to release the serial port after esptool operations.

For the ESP-Cryptoauthlib utility, the simplest fix would be to modify the code to properly release the serial port after using esptool's functions, either by using the context manager pattern or by explicitly calling the cleanup methods.

This is a valid bug that should be addressed in the esptool library to make it more robust when used as a Python module.

[424778940z]

Here is an update, after looking deeper in to it, it's a gc problem
At the script main logic

    hs.serial.load_app_stub(args.port, baudrate, binary_stub_path)
    import gc
    gc.collect()
    init_mfg = hs.serial.cmd_interpreter(port=args.port)

If I trigger gc manually like this, it will work, certenly the port is not released on time causing the issue.
However, this is still not a proper fix, AND a fix should be done at esptool level instead of "user" script

[radimkarnis]

Hi @424778940z,

a fix should be done at esptool level instead of "user" script

I don’t think that’s the case. Esptool supports both proper and manual cleanup of the serial port:

• If esp = detect_chip(PORT), is used, the user is responsible for closing the port and cleaning up when he's done.
• If you use with detect_chip(PORT) as esp:, the context manager will automatically handle cleanup and close the port once the block finishes — just like how Python handles files or sockets.

The script in the esp-cryptoauthlib repo uses the first approach, but forgets to close the port. That’s a bug in the script, not in esptool.

So the fix belongs in the esp-cryptoauthlib project, not here. I already let my coworkers working on that project know about the issue.

[424778940z]

I don’t think that’s the case. Esptool supports both proper and manual cleanup of the serial port

I don't think 100% trust caller and gc is a good idea when it comes to hardware related resources

And why a loader/commander class would like to grab the port on construct instead of only occupy it when there is actually work to do?

I don't think there is that much overheads for grab and release the port, at least consider how slow the serial port is

[nileshkale123]

Hello @424778940z ,

We have raised the MR to fix this. It will be merged soon.
Thanks