esptool.py misses the shebang line (IDFGH-9577) #10926

xinwenfu · 2023-03-06T15:55:41Z

Answers checklist.

I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

esp-idf/components/esptool_py/esptool/esptool.py misses the following shebang line and will not run normally as described at https://docs.espressif.com/projects/esptool/en/latest/esp32s3/esptool/basic-commands.html

#!/usr/bin/env python

dobairoland · 2023-03-06T16:11:06Z

@xinwenfu Are you running the command from an ESP-IDF environment, i.e. from a terminal where . export.sh has been called? If you do so then esptool.py will be available.

The suggested shebang is not necessary. components/esptool_py/esptool/esptool.py is kept for compatibility but it is not executed directly.

xinwenfu · 2023-03-06T22:15:06Z

The context is I tried to do everything within VS Code with the ESP-IDF Explorer. ESP-IDF was installed through VS Code. With this setup, I did run esptool.py from an ESP-IDF environment after running . export.sh within either a Linux terminal or ESP-IDF terminal of VS-Code. Here are the errors without the shebang line

$ esptool.py
/home/iot/esp/esp-idf/components/esptool_py/esptool/esptool.py: line 7: import: command not found
/home/iot/esp/esp-idf/components/esptool_py/esptool/esptool.py: line 8: import: command not found
/home/iot/esp/esp-idf/components/esptool_py/esptool/esptool.py: line 11: syntax error near unexpected token subprocess.run' /home/iot/esp/esp-idf/components/esptool_py/esptool/esptool.py: line 11: sys.exit(subprocess.run([sys.executable, '-m', 'esptool'] + sys.argv[1:]).returncode)'

dobairoland · 2023-03-07T09:01:22Z

I don't think you have esptool installed or esptool.py would be available. As a matter of fact, components/esptool_py/esptool/esptool.py is not even used in that case and it is kept just as a compatibility wrapper.

@brianignacio5 Could you please check this issue please? I don't exactly know how this should work in the case of Vscode. If the esptool package is installed then esptool.py is available and the mentioned wrapper doesn't have to be used.

xinwenfu · 2023-03-07T15:09:52Z

Below is what I got after running export.sh within an ESP-IDF terminal within VS code. Apprently, /home/iot/esp/esp-idf/components/esptool_py/esptool (where esptool.py is located) is added to PATH. esptool.py is only available in /home/iot/esp/esp-idf/components/esptool_py/esptool in the entire esp/esp-idf folder.

$ . $HOME/esp/esp-idf/export.sh
Detecting the Python interpreter
Checking "python3" ...
Python 3.8.10
"python3" has been detected
Checking Python compatibility
Checking other ESP-IDF version.
Adding ESP-IDF tools to PATH...
Checking if Python packages are up to date...
Constraint file: /home/iot/.espressif/espidf.constraints.v5.0.txt
Requirement files:
/home/iot/esp/esp-idf/tools/requirements/requirements.core.txt
Python being checked: /home/iot/.espressif/python_env/idf5.0_py3.8_env/bin/python
Python requirements are satisfied.
Added the following directories to PATH:
/home/iot/esp/esp-idf/components/esptool_py/esptool
/home/iot/esp/esp-idf/components/espcoredump
/home/iot/esp/esp-idf/components/partition_table
/home/iot/esp/esp-idf/components/app_update
/home/iot/.espressif/tools/xtensa-esp-elf-gdb/11.2_20220823/xtensa-esp-elf-gdb/bin
/home/iot/.espressif/tools/xtensa-esp32-elf/esp-2022r1-11.2.0/xtensa-esp32-elf/bin
/home/iot/.espressif/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin
/home/iot/.espressif/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/bin
/home/iot/.espressif/python_env/idf5.0_py3.8_env/bin
/home/iot/.espressif/tools/riscv32-esp-elf-gdb/11.2_20220823/riscv32-esp-elf-gdb/bin
/home/iot/.espressif/tools/xtensa-esp32s2-elf/esp-2022r1-11.2.0/xtensa-esp32s2-elf/bin
/home/iot/.espressif/tools/xtensa-esp32s3-elf/esp-2022r1-11.2.0/xtensa-esp32s3-elf/bin
/home/iot/.espressif/tools/riscv32-esp-elf/esp-2022r1-11.2.0/riscv32-esp-elf/bin
/home/iot/.espressif/tools/cmake/3.24.0/bin
/home/iot/.espressif/tools/ninja/1.10.2
/home/iot/esp/esp-idf/tools
/usr/bin
/home/iot/esp/esp-idf/tools
/home/iot/.local/bin
/home/iot/esp/esp-idf/tools
/home/iot/.local/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games
/snap/bin

Detected installed tools that are not currently used by active ESP-IDF version.
For removing cmake, esp32s2ulp-elf, xtensa-esp32s3-elf, riscv32-esp-elf-gdb, ninja, xtensa-esp32s2-elf, riscv32-esp-elf use command 'python /home/iot/esp/esp-idf/tools/idf_tools.py uninstall'
To free up even more space, remove installation packages of those tools. Use option 'python3 /home/iot/esp/esp-idf/tools/idf_tools.py uninstall --remove-archives'.

Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

idf.py build

dobairoland · 2023-03-15T15:18:15Z

I assume that we cannot fix your issue on the ESP-IDF side.

I can advise to uninstall Vscode, un-do all the settings in environment variables and try to set it up based on instructions at https://github.com/espressif/vscode-esp-idf-extension. If you still experience the issue then please report it at https://github.com/espressif/vscode-esp-idf-extension/issues.

brianignacio5 · 2023-03-16T08:54:06Z

@dobairoland From what the user @xinwenfu is telling I don't see any vscode extension configuration being used. Maybe is related to the current terminal being used in vscode as shown in Terminal shells vscode documentation.

If you are using the esp-idf vscode extension please open an issue with required troubleshooting information as @dobairoland mentioned.

xinwenfu · 2023-03-17T04:02:00Z

I reinstalled everything, including VS Code, the esp-idf extension and esp-idf based on instructions at https://github.com/espressif/vscode-esp-idf-extension. When I opened ESP-IDF Terminal within VS Code, the same error happened. I had to add the shebang line so that esptool.py works.

dobairoland · 2023-03-17T05:52:44Z

@brianignacio5 If esptool is installed with pip then esptool.py is already available and directly callable. In this user case components/esptool_py/esptool is forced to the PATH and is taking precedence before the installed module with pip. This is not happening in ESP-IDF terminal, so I don't understand what is causing it.

brianignacio5 · 2023-03-17T06:31:51Z

There is a very easy way to check which Python packages are in the virtual env with the ESP-IDF: Doctor command which shows python packages in there.

Isn't components/esptool_py/esptool still being added to PATH in export.sh ?

dobairoland · 2023-03-17T06:57:01Z

Isn't components/esptool_py/esptool still being added to PATH in export.sh ?

That is a leftover. We can remove it if it helps. But I don't understand why is this happening with Vscode only. I have not seen reports from those who use ESP-IDF terminal without Vscode.

dobairoland · 2023-03-17T08:18:50Z

@xinwenfu Could you please check the files in components/esptool_py/esptool? Do they have executable bit set? You can check this e.g. by running ls -al in the directory.

xinwenfu · 2023-03-18T14:11:04Z

Within the Terminal in VS Code with the ESP-IDF Explorer, if I run export.sh, it still adds /home/iot/esp/esp-idf/components/esptool_py/esptool to PATH as shown below. export.sh is executable. But it is treated as a general Linux script without that shbang line.

. $HOME/esp/esp-idf/export.sh
Detecting the Python interpreter
Checking "python3" ...
Python 3.8.10
"python3" has been detected
Checking Python compatibility
Checking other ESP-IDF version.
WARNING: No IDF variables to remove from environment found. Deactivation of previous esp-idf version was not successful.
Adding ESP-IDF tools to PATH...
Using a supported version of tool cmake found in PATH: 3.16.3.
However the recommended version is 3.24.0.
Checking if Python packages are up to date...
Constraint file: /home/iot/.espressif/espidf.constraints.v5.0.txt
Requirement files:
/home/iot/esp/esp-idf/tools/requirements/requirements.core.txt
Python being checked: /home/iot/.espressif/python_env/idf5.0_py3.8_env/bin/python
Python requirements are satisfied.
Added the following directories to PATH:
/home/iot/esp/esp-idf/components/esptool_py/esptool
/home/iot/esp/esp-idf/components/espcoredump
/home/iot/esp/esp-idf/components/partition_table
/home/iot/esp/esp-idf/components/app_update
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

idf.py build

xinwenfu · 2023-03-21T13:46:16Z

esptool.py is executable. But it is treated as a general Linux script without that shebang line.

dobairoland · 2023-03-21T14:55:59Z

esptool.py is executable. But it is treated as a general Linux script without that shebang line.

Maybe your local git configuration makes it executable. It should not be executable if git is not configured to change that on local checkout:

git ls-tree HEAD components/esptool_py/esptool/esptool.py 
100644 blob 828129e6d6d6d22df81f4308b5a3a4dff5aee897    components/esptool_py/esptool/esptool.py

In the above case, the attribute would be 755 if the executable bit would be set.

dobairoland · 2023-03-21T15:14:58Z

Please try to remove components/esptool_py/esptool from your PATH. That is the fix I'm going to submit.

If the environment is correctly installed then esptool.py will be available from the esptool package and the above wrapper won't be invoked at all.

Esptool.py is available from the esptool package which is installed in every ESP-IDF v5.0+ shells. Therefore, the path to the legacy wrapper scripts is not needed to be in the PATH. In some cases like #10926 it even cases problem because the wrapper doesn't have a shebang line and is not executable. Note that the wrapper is kept only for invoking esptool as "python $IDF_PATH/..../esptool.py" which was previously used mostly in CMake files. There should be no other use-case where the esptool wrappers are still needed.

xinwenfu · 2023-04-12T04:55:01Z

I got time to revisit the issue. export.sh has issues and has the following line adding the components/esptool_py/esptool to PATH.

IDF_ADD_PATHS_EXTRAS="${IDF_PATH}/components/esptool_py/esptool"

After running . $HOME/esp/esp-idf/export.sh, the wrong esptool.py is then used even if sudo apt install esptool is used to install the esptool package.

dobairoland · 2023-04-12T06:32:20Z

I got time to revisit the issue. export.sh has issues and has the following line adding the components/esptool_py/esptool to PATH.

Please check the linked commits above. This is not true anymore.

xinwenfu · 2023-04-12T11:58:29Z

Yesterday, I followed https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/linux-macos-setup.html and installed ESP-IDF again. export.sh still had the issue. Thought the problem was still there. Good to know it is resolved. Thanks

espressif-bot added the Status: Opened Issue is new label Mar 6, 2023

github-actions bot changed the title ~~esptool.py misses the shebang line~~ esptool.py misses the shebang line (IDFGH-9577) Mar 6, 2023

dobairoland closed this as completed Mar 6, 2023

espressif-bot assigned dobairoland Mar 6, 2023

espressif-bot added Status: Selected for Development Issue is selected for development and removed Status: Opened Issue is new labels Mar 20, 2023

achou11 mentioned this issue Jan 27, 2024

Is sourcing $IDF_PATH/export.sh in xs-dev-export.sh file still necessary? (ESP32 setup) HipsterBrown/xs-dev#161

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

esptool.py misses the shebang line (IDFGH-9577) #10926

esptool.py misses the shebang line (IDFGH-9577) #10926

xinwenfu commented Mar 6, 2023

dobairoland commented Mar 6, 2023

xinwenfu commented Mar 6, 2023

dobairoland commented Mar 7, 2023

xinwenfu commented Mar 7, 2023 •

edited

dobairoland commented Mar 15, 2023

brianignacio5 commented Mar 16, 2023

xinwenfu commented Mar 17, 2023

dobairoland commented Mar 17, 2023

brianignacio5 commented Mar 17, 2023

dobairoland commented Mar 17, 2023

dobairoland commented Mar 17, 2023

xinwenfu commented Mar 18, 2023

xinwenfu commented Mar 21, 2023

dobairoland commented Mar 21, 2023

dobairoland commented Mar 21, 2023

xinwenfu commented Apr 12, 2023

dobairoland commented Apr 12, 2023

xinwenfu commented Apr 12, 2023

esptool.py misses the shebang line (IDFGH-9577) #10926

esptool.py misses the shebang line (IDFGH-9577) #10926

Comments

xinwenfu commented Mar 6, 2023

Answers checklist.

General issue report

dobairoland commented Mar 6, 2023

xinwenfu commented Mar 6, 2023

dobairoland commented Mar 7, 2023

xinwenfu commented Mar 7, 2023 • edited

dobairoland commented Mar 15, 2023

brianignacio5 commented Mar 16, 2023

xinwenfu commented Mar 17, 2023

dobairoland commented Mar 17, 2023

brianignacio5 commented Mar 17, 2023

dobairoland commented Mar 17, 2023

dobairoland commented Mar 17, 2023

xinwenfu commented Mar 18, 2023

xinwenfu commented Mar 21, 2023

dobairoland commented Mar 21, 2023

dobairoland commented Mar 21, 2023

xinwenfu commented Apr 12, 2023

dobairoland commented Apr 12, 2023

xinwenfu commented Apr 12, 2023

xinwenfu commented Mar 7, 2023 •

edited