Update/Add pipx to M1 runners

[henryiii]

Tool name

pipx

Tool license

MIT

Add or update?

• Add
• Update

Desired version

1.4.3

Approximate size

No response

Brief description of tool

This is available on all other images, and is the basis for the GitHub Actions for cibuildwheel, nox, and a lot of other Python tooling. It's also pre-installed in lots of other places too, like the manylinux images for Python.

I think it was supposed to be present in #7599 (given the name of the PR), but it is not present currently for M1.

URL for tool's homepage

https://pipx.pypa.io

Provide a basic test case to validate the tool's functionality.

pipx run cowsay

Platforms where you need the tool

• Azure DevOps
• GitHub Actions

Runner images where you need the tool

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Can this tool be installed during the build?

No response

Tool installation time in runtime

No response

Are you willing to submit a PR?

Yes if that helps.

[Jackenmen]

Notably, there is no Python present on the image either, usage of https://github.com/actions/setup-python seems to be required to run any kind of Python code since macOS no longer even ships with it. More so, according to https://github.com/actions/runner-images/blob/37a171e44de1f134793d7f0cbff137ca481942ab/images/macos/macos-14-arm64-Readme.md there's no tool cache anymore and setup-python always has to download it? Either way, at least one Python install, along with pipx, would be nice since pipx can then be used with any interpreter that's later installed with actions/setup-python.

[erik-bershel]

Hey @henryiii
The macOS-14 image has been significantly streamlined compared to its predecessors. We removed some packages and programs that could be easily and quickly installed at runtime. Compatibility issues also influenced decisions of this nature. Regarding the stated need to add the pipx package, we will explore what can be done and provide additional information later.

UPD:
@Jackenmen, thank you for your investigation. We'll take it into account also.

[henryiii]

Okay, thanks for looking at it. Having pipx as a supported package manager was very important for composite actions (Python actually didn't matter, since that can be pulled by setup-python and targeted by any pipx, only having pipx actually mattered once activate-environment: false was supported by setup-python). Here's what an isolated action looks like with pipx:

Pipx installed isolated composite action:

runs:
  using: composite
  steps:
    # Set up a non-EOL, cibuildwheel & pipx supported Python version
    - uses: actions/setup-python@v5
      id: python
      with:
        python-version: "3.8 - 3.12"
        update-environment: false
    - run: >
        pipx run
        --python '${{ steps.python.outputs.python-path }}'
        --spec '${{ github.action_path }}'
        myapp
      shell: bash

Now, if pipx is not installed, the simple workaround initially seemed to be:

Pipx not installed incorrectly isolated composite action:

runs:
  using: composite
  steps:
    - uses: actions/setup-python@v5
      id: python
      with:
        python-version: "3.8 - 3.12"
        update-environment: false

    # macos-14 (M1) may be missing pipx (due to it not having CPython)
    - run: |
        "${{ steps.python.outputs.python-path }}" -m pip install pipx
      shell: bash

    # Note that different syntax is required for powershell
    - run: >
        "${{ steps.python.outputs.python-path }}" -m pipx run
        --spec '${{ github.action_path }}'
        myapp
      shell: bash

However, this permanently changes the setup-python provide environment. This makes the composite action surprising (if you try to install something later that is incompatible with pipx's dependencies after running a compatible setup-python call, it will break), and it can affect self hosted runner's environments. That's why you should use pipx instead of pip, but if you have to use pip to get pipx, you are still using pip!

Here's the correct workaround:

Pipx not installed isolated composite action:

runs:
  using: composite
  steps:
    - uses: actions/setup-python@v5
      id: python
      with:
        python-version: "3.8 - 3.12"
        update-environment: false

    - id: myapp
      run: |
        "${{ steps.python.outputs.python-path }}" -u << "EOF"
        import os
        import sys
        import venv
        from pathlib import Path
        from subprocess import run
        class EnvBuilder(venv.EnvBuilder):
            def __init__(self):
                super().__init__()
            def setup_scripts(self, context):
                pass
            def post_setup(self, context):
                super().post_setup(context)
                self.python_path = context.env_exe
                run([sys.executable, "-m", "pip", "--python", context.env_exe, "install", r"${{ github.action_path }}"], check=True)
        print("::group::Install myapp")
        builder = EnvBuilder()
        builder.create(Path(r"${{ runner.temp }}") / "cibw")
        with open(os.environ["GITHUB_OUTPUT"], "at") as f:
            f.write(f"python-path={builder.python_path}\n")
        print("::endgroup::")
        EOF

    # Note that even more different syntax is required for powershell
    # Also, this workaround requires a __main__ module to run it via -m
    - run: >
        "${{ steps.myapp.outputs.python-path }}" -m myapp
      shell: bash

cibuildwheel has to use powershell on Windows, which makes the workaround even a bit more messy.

Having pipx as a package manager alongside the native package managers like brew was very helpful! (if setup python included pipx by default or via an option, that would also work, but has different implications & effects).

[henryiii]

Pipx is present now, but permissions aren'y set up correctly:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/opt/pipx/venvs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/bin/pipx", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipx/1.4.3/libexec/lib/python3.12/site-packages/pipx/main.py", line 916, in cli
    setup(parsed_pipx_args)
  File "/opt/homebrew/Cellar/pipx/1.4.3/libexec/lib/python3.12/site-packages/pipx/main.py", line 863, in setup
    mkdir(constants.PIPX_LOCAL_VENVS)
  File "/opt/homebrew/Cellar/pipx/1.4.3/libexec/lib/python3.12/site-packages/pipx/util.py", line 75, in mkdir
    path.mkdir(parents=True, exist_ok=True)
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/pathlib.py", line 1315, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/usr/local/opt/pipx'

[giarc3]

While waiting for a fix for this, I was able to work around the permissions issue in my workflow just by adding sudo. Maybe it will work for others as well.

[mikhailkoliada]

Pipix is added, permissions issue is tracked here: #9607 and should be rolled out soon too.

[henryiii]

Thanks!