[Ubuntu 24] Removal of Temporary Workaround for Pip Install on Python Default Environment from 2025-05-09

[vidyasagarnimmagaddi]

Breaking changes

Removal of Temporary Workaround for Pip Install on Python Default Environment for Ubuntu 24

• Reverting the PR

Target date

The rollout will start on 2025-05-05, and will take 3-4 days to propagate the image to all the environments

The motivation for the changes

From Python Version 3.12 , packages in the default environment are restricted from Modification.
More info: https://packaging.python.org/en/latest/specifications/externally-managed-environments/

Possible impact

Post implementation: Pip Install command will fail with Error Message: "Externally Managed"

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

Platforms affected

• Azure DevOps
• GitHub Actions

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022

Mitigation ways

Users are requested to use actions/setup-python action or can create a virtual env.

[MichaIng]

I just wanted to suggest to apply the same workaround for the macOS images, where we just ran into the same issue ... sadly "The motivation for the changes" does not reflect the motivation for having this pip lock on a GitHub Actions runner image, which is meant to exist temporarily for a particular test or build only. It just explains the change at Python side, at best containing a motivation for why Python implemented the option. So what is your motivation for preventing GitHub Action users to run pip install ... outside of a v(irtual)env?

And I want to point out that Python itself does NOT prevent pip from installing packages, but it is the distributions which added the marker file into their Python stdlib packages, to make use of that option. For end user distros this makes sense, since indeed Python packages installed via pip usually override those installed via distro package manager, which can cause theoretically dependency/compatibility issues. GitHub runners however come with their own Python instance, NOT installed via distro package manager, where any motivation for distros to implement the marker does not apply. It stands IMO a fact that this is a burden for GitHub Action users, over-complicating things by requiring a v(irtual)env in a temporary runner image, or applying the workaround by themselves.

[agray-ctm]

My agents are running DinD (sysbox)

./pip refers to:

#!/bin/bash
set -e

# Wrap "pip install" with --break-system-packages
if [ "$1" = "install" ]; then
  shift;
  pip.orig install "$@" --break-system-packages;
else
  pip.orig "$@";
fi

And in the Dockerfile

# This is a consequence of upgrading Ubuntu 22.04 -> 24.04
# Given the agents are tossed away each run, this is a reasonable tradeoff
RUN mv $(which pip3) /usr/bin/pip.orig
COPY --chmod=755 ./pip /usr/bin/pip3
COPY --chmod=755 ./pip /usr/bin/pip
RUN dos2unix /usr/bin/pip3 /usr/bin/pip

If you're not using DinD, just do the pip replacement manually.

[MichaIng]

As an universal workaround for Windows and macOS as well, the marker file can be removed this way, for whichever Python environment the python3 command invokes:

rm "$(python3 -c 'import sysconfig; print(sysconfig.get_path("stdlib", sysconfig.get_default_scheme()))')/EXTERNALLY-MANAGED"

This is the same method Python uses internally to check for the merker.

However, what is the point to let GitHub Actions users all potentially run into this error and searching for a solution themselves, when it can be applied to the worker images instead, to make everyone's life easier 🙂.