subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', 'install', '--user', 'virtualenv']' returned non-zero exit status 1. (IDFGH-11154)

[franz-ms-muc]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

esp-idf-v4.2.5

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

I expected it to run the installation if i call

:~/esp-idf-v4.2.5$ ./install.sh all

What is the actual behavior?

Instead it give a Error Message

Steps to reproduce.

1. Step
2. Step
3. Step
   ...

Build or installation Logs.

franz@franz-Nitro-AN517-55:~/esp-idf-v4.2.5$ ./install.sh all
Detecting the Python interpreter
Checking "python" ...
Python 3.11.4
"python" has been detected
Installing ESP-IDF tools
Selected targets are: esp32s2, esp32s3, esp32
Installing tools: xtensa-esp32-elf, xtensa-esp32s2-elf, esp32ulp-elf, esp32s2ulp-elf, openocd-esp32
Installing xtensa-esp32-elf@esp-2020r3-8.4.0
Downloading xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64.tar.gz to /home/franz/.espressif/dist/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64.tar.gz.tmp
Done
Extracting /home/franz/.espressif/dist/xtensa-esp32-elf-gcc8_4_0-esp-2020r3-linux-amd64.tar.gz to /home/franz/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0
Installing xtensa-esp32s2-elf@esp-2020r3-8.4.0
Downloading xtensa-esp32s2-elf-gcc8_4_0-esp-2020r3-linux-amd64.tar.gz to /home/franz/.espressif/dist/xtensa-esp32s2-elf-gcc8_4_0-esp-2020r3-linux-amd64.tar.gz.tmp
Done
Extracting /home/franz/.espressif/dist/xtensa-esp32s2-elf-gcc8_4_0-esp-2020r3-linux-amd64.tar.gz to /home/franz/.espressif/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0
Installing esp32ulp-elf@2.28.51-esp-20191205
Downloading binutils-esp32ulp-linux-amd64-2.28.51-esp-20191205.tar.gz to /home/franz/.espressif/dist/binutils-esp32ulp-linux-amd64-2.28.51-esp-20191205.tar.gz.tmp
Done
Extracting /home/franz/.espressif/dist/binutils-esp32ulp-linux-amd64-2.28.51-esp-20191205.tar.gz to /home/franz/.espressif/tools/esp32ulp-elf/2.28.51-esp-20191205
Installing esp32s2ulp-elf@2.28.51-esp-20191205
Downloading binutils-esp32s2ulp-linux-amd64-2.28.51-esp-20191205.tar.gz to /home/franz/.espressif/dist/binutils-esp32s2ulp-linux-amd64-2.28.51-esp-20191205.tar.gz.tmp
Done
Extracting /home/franz/.espressif/dist/binutils-esp32s2ulp-linux-amd64-2.28.51-esp-20191205.tar.gz to /home/franz/.espressif/tools/esp32s2ulp-elf/2.28.51-esp-20191205
Installing openocd-esp32@v0.11.0-esp32-20220706
Downloading openocd-esp32-linux-amd64-0.11.0-esp32-20220706.tar.gz to /home/franz/.espressif/dist/openocd-esp32-linux-amd64-0.11.0-esp32-20220706.tar.gz.tmp
Done
Extracting /home/franz/.espressif/dist/openocd-esp32-linux-amd64-0.11.0-esp32-20220706.tar.gz to /home/franz/.espressif/tools/openocd-esp32/v0.11.0-esp32-20220706
Installing Python environment and packages
Creating a new Python environment in /home/franz/.espressif/python_env/idf4.2_py3.11_env
Installing virtualenv
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Traceback (most recent call last):
  File "/home/franz/esp-idf-v4.2.5/tools/idf_tools.py", line 1390, in action_install_python_env
    import virtualenv   # noqa: F401
    ^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'virtualenv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/franz/esp-idf-v4.2.5/tools/idf_tools.py", line 1696, in <module>
    main(sys.argv[1:])
  File "/home/franz/esp-idf-v4.2.5/tools/idf_tools.py", line 1692, in main
    action_func(args)
  File "/home/franz/esp-idf-v4.2.5/tools/idf_tools.py", line 1393, in action_install_python_env
    subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--user', 'virtualenv'],
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'pip', 'install', '--user', 'virtualenv']' returned non-zero exit status 1.

More Information.

No response

[franz-ms-muc]

sudo apt install python3-virtualenv

did fix this, but i am unsure if this is the right mehthod to handle this.

Thanks,
Franz

[AxelLin]

esp-idf v4.2 branch is already EOL.
https://www.espressif.com/sites/default/files/advisory_downloads/AR2023-004%20End-of-Life%20Advisory%20for%20ESP-IDF%20v4.2%20Release%20Branch%20EN.pdf

[zinke-ct-video]

I experienced the same issue today with esp-idf v4.4.

[franz-ms-muc]

esp-idf v4.2 branch is already EOL. https://www.espressif.com/sites/default/files/advisory_downloads/AR2023-004%20End-of-Life%20Advisory%20for%20ESP-IDF%20v4.2%20Release%20Branch%20EN.pdf

Agree.

i have here a Project which uses MCPWM, and this Peripherial only runs stable in 4.2.5 or earlier.
i will make a extra issue once i understood the Details from my colleagues.

Therfore i needed to install 4.2.5

i absolutely share your Opinion that EOL ESP-IDF should not be used anymore.

[dobairoland]

did fix this, but i am unsure if this is the right mehthod to handle this.

Yes, that is the correct way. You haven't mentioned the OS you are using but since you mentioned apt and using Python 3.11, I assume it is some new version of Debian/Ubuntu or derivatives.

Earlier versions used to install more packages by default and packages had more strict dependencies. python3-virualenv used to be pulled in automatically when one installed the prerequisite packages from the getting started guide. This package needs to be installed now manually with newer OS versions.

[dobairoland]

One nice tip for using old ESP-IDF releases is with docker. Older versions, such as v4.2.5, can be found as well. With docker one doesn't have to bother to install and set up everything.

(Thanks for the tip @DNedic!)