`RUNNER_TOOL_CACHE` and/or `AGENT_TOOLSDIRECTORY` not picked up on self hosted runner

[nvincent-vossloh]

Description:
I have exported RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY in the shell running the self-hosted runner, however a pip install in the workflow still uses /opt/hostedtoolcache directory.
job-log.txt

2024-07-23T09:20:33.0492190Z       [1/41] Compiling C object gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0493604Z       FAILED: gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o
2024-07-23T09:20:33.0502062Z       cc -Igi/_gi.cpython-310-x86_64-linux-gnu.so.p -Igi -I../gi -I. -I.. -I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/
usr/include/gobject-introspection-1.0 -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -DPY_SSIZE_T_CLEAN -Wall -Warray-bounds -Wcast-align -Wduplicated-branches -Wextra -Wform
at=2 -Wformat-nonliteral -Wformat-security -Wimplicit-function-declaration -Winit-self -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wmissing-prototypes -Wne
sted-externs -Wnull-dereference -Wold-style-definition -Wpacked -Wpointer-arith -Wrestrict -Wreturn-type -Wshadow -Wsign-compare -Wstrict-aliasing -Wstrict-prototypes -Wswitch-default -Wundef -Wunused-but-set-variable -Wwrite-strings -W
no-missing-field-initializers -Wno-unused-parameter -Wno-discarded-qualifiers -Wno-sign-conversion -Wno-cast-function-type -Wno-int-conversion -fno-strict-aliasing -fvisibility=hidden -MD -MQ gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -MF gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o.d -o gi/_gi.cpython-310-x86_64-linux-gnu.so.p/pygboxed.c.o -c ../gi/pygboxed.c
2024-07-23T09:20:33.0511885Z       cc1: warning: /opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10: No such file or directory [-Wmissing-include-dirs]
2024-07-23T09:20:33.0513997Z       ../gi/pygboxed.c:23:10: fatal error: Python.h: No such file or directory
2024-07-23T09:20:33.0515317Z          23 | #include <Python.h>
2024-07-23T09:20:33.0516190Z             |          ^~~~~~~~~~
2024-07-23T09:20:33.0517073Z       compilation terminated.

Action version:
actions/setup-python@v5

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:
python-version: '3.10'

Repro steps:

name: setup python on shr    
# Test pip install which requires compilation on host where include path is not correct    
# https://github.com/actions/setup-python/issues/485    
# https://github.com/actions/setup-python/issues/489    
    
on:    
  # Allows you to run this workflow manually from the Actions tab    
  workflow_dispatch:    
  push:    
    branches:    
      - setup-python-pip-install    
    
jobs:    
  test-python:    
    name: Test Python pip install    
    runs-on: [self-hosted, linux, test-python]    
    steps:    
      - name: Github Actions Context Information    
        uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7    
              
      - uses: actions/setup-python@v5    
        with:    
          python-version: '3.10'    
    
      - name: show env    
        run: |    
          env    
    
      - name: python pkg install    
        run: |    
          pip install PyGObject

From the Self-hosted runner directory:

export RUNNER_TOOL_CACHE="$(pwd)/_work/_tool"
export AGENT_TOOLSDIRECTORY="$(pwd)/_work/_tool"
./run.sh

Expected behavior:
I would expect the pip install step to set properly the -I flags without /opt/hostedtoolcache

Actual behavior:
The logs (see above) show that it tried to add an include path (-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which do not exist on the self hosted runner.

Workaround I am not fond of

export CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10
./run.sh

With this workaround I manage to install the lib, however it looks like that pip cached some stuff in the default cache directory:

  pip install PyGObject
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    PKG_CONFIG_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib/pkgconfig
    Python_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    Python2_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    Python3_ROOT_DIR: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64
    LD_LIBRARY_PATH: /home/nicolas/work/siema/be/shr/_work/_tool/Python/3.10.14/x64/lib
Collecting PyGObject
  Using cached pygobject-3.48.2.tar.gz (715 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pycairo>=1.16 (from PyGObject)
  Using cached pycairo-1.26.1-cp310-cp310-linux_x86_64.whl
Building wheels for collected packages: PyGObject
  Building wheel for PyGObject (pyproject.toml): started
  Building wheel for PyGObject (pyproject.toml): finished with status 'done'
  Created wheel for PyGObject: filename=pygobject-3.48.2-cp310-cp310-linux_x86_64.whl size=259354 sha256=fc1421f9adae4077d60198ee04309773969c33b2372d1cc93f2733967c3f3f96
  Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97
Successfully built PyGObject
Installing collected packages: pycairo, PyGObject
Successfully installed PyGObject-3.48.2 pycairo-1.26.1

(Stored in directory: /home/nicolas/.cache/pip/wheels/94/02/70/6ba5403853459b4afcbb7c0348deece9bbc5853d9e0889dc97)
even though the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY have been set up.

I took the idea of this workaround from #489 and #485
However I do not understand why the RUNNER_TOOL_CACHE and AGENT_TOOLSDIRECTORY are not used.

Beside, if i run

grep -R /opt/hostedtoolcache *

from _work/_tool I get a lot of match:

...
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIGURE_LDFLAGS=      -Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:prefix=         /opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile:CONFIG_ARGS=     '--prefix=/opt/hostedtoolcache/Python/3.10.14/x64' '--enable-shared' '--enable-optimizations' '--enable-loadable-sqlite-extensions' 'LDFLAGS=-Wl,--rpath=/opt/hostedtoolcache/Python/3.10.14/x64/lib'
Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/python-config.py:#!/opt/hostedtoolcache/Python/3.10.14/x64/bin/python3.10
grep: Python/3.10.14/x64/lib/python3.10/config-3.10-x86_64-linux-gnu/libpython3.10.a: binary file matches
Python/3.10.14/x64/lib/pkgconfig/python-3.10-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python-3.10.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
Python/3.10.14/x64/lib/pkgconfig/python3-embed.pc:prefix=/opt/hostedtoolcache/Python/3.10.14/x64
...

I don't really know where pip gets the include path from, but I guess from one of the above files which clearly points to an invalid folder.

Thanks in advance for your help, let me know if you need more information.

[mahabaleshwars]

Hello @nvincent-vossloh,
Thank you for creating this issue. We will investigate it and provide feedback as soon as we have some updates.

[aparnajyothi-y]

Hello @nvincent-vossloh, Thank you once again for creating this issue. We have investigated and found that the problem is due to environment variables not being available in the context where the action is running. The given logs indicate the inclusion of a path (-I/opt/hostedtoolcache/Python/3.10.14/x64/include/python3.10) which does not exist on the self-hosted runner. You need to ensure that the actions/setup-python action respects the custom tool cache directory by setting the environment variables directly in the workflow, as shown in the job below:

jobs:
  test-python:
    name: Test Python pip install
    runs-on: [self-hosted, linux, test-python]
    env:
      RUNNER_TOOL_CACHE: ${{ github.workspace }}/_work/_tool
      AGENT_TOOLSDIRECTORY: ${{ github.workspace }}/_work/_tool
    steps:
      - name: Github Actions Context Information
        uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Show environment variables
        run: env
      - name: Python package install
        run: pip install PyGObject

you can use any custom directory to install python on your self-hosted runner using AGENT_TOOLSDIRECTORY environment variable as per the setup-python documentation.
Pip gets the include path from the environment variables set up during the installation of Python. Specifically, the paths such as PKG_CONFIG_PATH, Python_ROOT_DIR, Python2_ROOT_DIR, and Python3_ROOT_DIR are configured to point to the respective directories during the setup process.

[aparnajyothi-y]

Hello @nvincent-vossloh, Please let us know in case of any further concerns on the above information.

[nvincent-vossloh]

Thanks for your feedback, I will try your solution and let you know if that fixed the issue on my side.

[nvincent-vossloh]

The documentation you link to, states the the environment variable could be set in the same shell where the runner is running. Unfortunately that does not work (anymore ?), that is why I opened the issue in the first place. Maybe the runner software does not use the env variable from the shell anymore (did not work with the .env file either).

Anyway, unfortunately setting the env variable in the workflow does not fix the issue, the behavior is still the same. On my first try it succeeded, but that was because the pip install step used a cache version of PyGObject, after deleting it the failure to compile still happens.

Only setting CPATH seems to work.

[gowridurgad]

Hi @nvincent-vossloh, After further investigation, we found that setting the environment variable in the workflow successfully works when the cached version of PyGObject is available. However, after clearing the cache, the compilation failure reoccurs. Given these circumstances, It appears that only setting CPATH directly works. This is because CPATH explicitly sets the include path for the compiler, ensuring the correct headers are found during the compilation of PyGObject. The default paths set by pip or setup-python may not align with the directory structure on the self-hosted runner, leading to the observed compilation errors.

Setting CPATH ensures that the compiler can locate the necessary Python header files, avoiding the "No such file or directory" error for Python.h. This approach provides a direct and reliable way to specify include directories, bypassing any inconsistencies in environment variable handling between the workflow and the self-hosted runner.

Thank you for your understanding and patience as we work through this issue.

[nvincent-vossloh]

Do you plan on updating the setup-python action in order to add the include directory to the CPATH env var ?

[gowridurgad]

Hi @nvincent-vossloh, Thank you for suggesting to include directory to the CPATH environment variable in the setup-python action. We acknowledge the importance of this feature and will explore the possible ways to implement it. We will consider this for a future enhancement.

In the meantime, here is an alternative solution to include CPATH in a GitHub Actions workflow:
Modify the workflow to set the CPATH environment variable: The CPATH environment variable can be set directly in the workflow file using the env key.

jobs:
  test-python:
    name: Test Python pip install
    runs-on: self-hosted 
    steps:
      - name: Github Actions Context Information
        uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Set CPATH environment variable
        run: echo "CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10" >> $GITHUB_ENV
      - name: Python package install
        run: pip install PyGObject

[nvincent-vossloh]

Hi @nvincent-vossloh, Thank you for suggesting to include directory to the CPATH environment variable in the setup-python action. We acknowledge the importance of this feature and will explore the possible ways to implement it. We will consider this for a future enhancement.

Thank you for taking my suggestion into account.

In the meantime, here is an alternative solution to include CPATH in a GitHub Actions workflow: Modify the workflow to set the CPATH environment variable: The CPATH environment variable can be set directly in the workflow file using the env key.

jobs:
  test-python:
    name: Test Python pip install
    runs-on: self-hosted 
    steps:
      - name: Github Actions Context Information
        uses: SiemaApplications-attic/vossloh-gh-actions/debug/show-github-context@v7
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Set CPATH environment variable
        run: echo "CPATH=$(pwd)/_work/_tool/Python/3.10.14/x64/include/python3.10" >> $GITHUB_ENV
      - name: Python package install
        run: pip install PyGObject

In the end I used this:

env:
  pytest_python_version: '3.10'
[...]
      - uses: actions/setup-python@v5
        id: setup-python
        with:
          python-version: ${{env.pytest_python_version}}
      - name: setup CPATH
        run: |
          echo  "python version : ${{ steps.setup-python.outputs.python-version }}"
          python_version="${{ env.pytest_python_version }}"
          echo  "python path : ${{ steps.setup-python.outputs.python-path }}"
          CPATH=${pythonLocation}/include/python${python_version}
          echo CPATH=${CPATH} >> "${GITHUB_ENV}"

which leverage the various env variables set.

Thanks again.