No module named pip (Ubuntu 20.04 LTS) (IDFGH-4859)

[JarrettR]

Environment

• IDF version: 4.2, 4.3, current dev branch
• Operating System: Ubuntu 20.04 LTS
• Using an IDE?: Was trying with the VSCode extension, but I don't believe that's the issue.

Problem Description

Installing the IDF on Ubuntu 20.04 LTS fails. Running .install.sh fails with an error message complaining that pip is not found.

Worth noting that my system does not have Python 2 installed. Pip3 is installed, and it seems that pip is symlinked to pip3.

Steps to reproduce

• Download release
• Type .install.sh

Debug Logs

Downloading Toolchain for Xtensa (ESP32) based on GCC
Found xtensa-esp32-elf in /home/jarrett/.espressif/dist
Downloading Toolchain for Xtensa (ESP32-S2) based on GCC
Found xtensa-esp32s2-elf in /home/jarrett/.espressif/dist
Downloading Toolchain for ESP32 ULP coprocessor
Found esp32ulp-elf in /home/jarrett/.espressif/dist
Downloading Toolchain for ESP32-S2 ULP coprocessor
Found esp32s2ulp-elf in /home/jarrett/.espressif/dist
Downloading OpenOCD for ESP32
Found openocd-esp32 in /home/jarrett/.espressif/dist

Installing package Toolchain for Xtensa (ESP32) based on GCC
Installing tar.gz package Toolchain for Xtensa (ESP32) based on GCC
Installing package Toolchain for Xtensa (ESP32-S2) based on GCC
Installing tar.gz package Toolchain for Xtensa (ESP32-S2) based on GCC
Installing package Toolchain for ESP32 ULP coprocessor
Installing tar.gz package Toolchain for ESP32 ULP coprocessor
Installing package Toolchain for ESP32-S2 ULP coprocessor
Installing tar.gz package Toolchain for ESP32-S2 ULP coprocessor
Installing package OpenOCD for ESP32
Installing tar.gz package OpenOCD for ESP32

Installing ESP-IDF extension python packages in /home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python ...


/home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python: No module named pip
Command failed: "/home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python" -m pip install --no-warn-script-location  -r "/home/jarrett/.vscode/extensions/espressif.esp-idf-extension-1.0.0/requirements.txt"
/home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python: No module named pip
Command failed: "/home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python" -m pip install --no-warn-script-location  -r "/home/jarrett/.vscode/extensions/espressif.esp-idf-extension-1.0.0/requirements.txt"
/home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python: No module named pip

This isn't a VSCode issue as far as I can tell, manually running install.sh as recommended by this repo's docs are essentially the same:

/install.sh 
Detecting the Python interpreter
Checking "python" ...
Python 3.8.5
"python" has been detected
Installing ESP-IDF tools
Installing tools: xtensa-esp32-elf, xtensa-esp32s2-elf, esp32ulp-elf, esp32s2ulp-elf, openocd-esp32
Skipping xtensa-esp32-elf@esp-2020r3-8.4.0 (already installed)
Skipping xtensa-esp32s2-elf@esp-2020r3-8.4.0 (already installed)
Skipping esp32ulp-elf@2.28.51-esp-20191205 (already installed)
Skipping esp32s2ulp-elf@2.28.51-esp-20191205 (already installed)
Skipping openocd-esp32@v0.10.0-esp32-20200709 (already installed)
Installing Python environment and packages
Installing Python packages from /home/jarrett/esp/esp-idf/requirements.txt
/home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python: No module named pip
Traceback (most recent call last):
  File "/home/jarrett/esp/esp-idf/tools/idf_tools.py", line 1492, in <module>
    main(sys.argv[1:])
  File "/home/jarrett/esp/esp-idf/tools/idf_tools.py", line 1488, in main
    action_func(args)
  File "/home/jarrett/esp/esp-idf/tools/idf_tools.py", line 1215, in action_install_python_env
    subprocess.check_call(run_args, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/jarrett/.espressif/python_env/idf4.2_py3.8_env/bin/python', '-m', 'pip', 'install', '--no-warn-script-location', '-r', '/home/jarrett/esp/esp-idf/requirements.txt']' returned non-zero exit status 1.

Other items if possible

I'm aware of this commit:

dc70cd1

Which solves a similar issue for Mac OS, here: #6532 . I manually patched my current IDF with that update, but that didn't work, and I also tried the current 4.1 dev branch. Have also tried 4.3.

Also not fixed by #6471

[dobairoland]

Hi @JarrettR. I'm sorry for the inconvenience. Thank you for taking the time by patching your branch and providing lot of information. I'm going to reproduce this on a clear Ubuntu 20.04.

Have you installed python3-pip package after you've run install.sh the first time?

Could you please provide the output of the following command?

python --version ; python -m virtualenv /tmp/py ; rm -r /tmp/py

Could you please run also the following? Does it fix your issue?

python $IDF_PATH/tools/idf_tools.py install-python-env --reinstall

[JarrettR]

That fixed it, thank you!

For future reference, virtualenv wasn't installed on my system, so that command failed at first. After installing it, I got this:

/esp/esp-idf$ python --version ; python -m virtualenv /tmp/py ; rm -r /tmp/py
Python 3.8.5
created virtual environment CPython3.8.5.final.0-64 in 329ms
  creator CPython3Posix(dest=/tmp/py, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/jarrett/.local/share/virtualenv)
    added seed packages: pip==21.0.1, setuptools==52.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

And then I also tried installing the IDF again from zip. Same issue.

The install-python-env --reinstall worked, though.

[dobairoland]

Great! I'm happy that it is solved.

For future reference, virtualenv wasn't installed on my system, so that command failed at first.

That is weird. It is should have been installed automatically.

I'm adding some additional checks and the virtual environment will be reinstalled in similar cases. This will be closed with my patch.

[AxelLin]

Hi @dobairoland

I'm not sure if this is related, I got below error with master tree (v4.4-dev-744-g1cb31e50943b).

$ python -m pip install --user -r $IDF_PATH/requirements.txt -r $IDF_PATH/tools/ble/requirements.txt
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

[dobairoland]

@AxelLin You don't need (and cannot use) the --user argument after you've activated a virtual environment, i.e. you've sourced export.sh or other export scripts of ESP-IDF. You can just rerun the command without --user.

[AxelLin]

Hi @dobairoland
After removing "--user" argument I got below error: (I'm using Ubuntu 20.04.2 LTS)

configure: error: Package requirements (dbus-1 >= 1.8) were not met:

No package 'dbus-1' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

[dobairoland]

@AxelLin Thank you for reporting this. Could you please open a new issue for it? The person responsible for this will take care of it.

The issue is that the requirements for BLE are separated from the general requirements of ESP-IDF. This means also that the installed prerequisites of ESP-IDF won't be enough for the tools of BLE.

You can fix this by installing DBUS with apt.