Python Action not supported

[elleobrien]

I see in the README example how to install Python "manually", but there's a GH action setup-python already built for installing Python that seems recommended.

I was trying this out with the cml docker container, and there seems to be an odd interaction between the docker container and setup-python.

For example, if I'm not using our docker container and run this workflow:

name: your-workflow-name

on: [push]

jobs:
  run:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/dvc-cml:latest

    steps:
      - uses: actions/checkout@v2
      
      - name: Set up Python
        uses: actions/setup-python@v1
        with:
          python-version: '3.x'
          
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"
      
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip 
          pip install sklearn

Everything runs beautifully. As soon as I add container: docker://dvcorg/dvc-cml:latest to the .yaml file, though, I hit the following error:

Any ideas?

[DavidGOrtega]

Our image does not support that action out of the box (nor even their self-hosted runner). The actor has to implement a cache system to make that available. You can read more here

https://github.com/actions/setup-python#hosted-tool-cache

@dmpetrov as we discussed we should support this despite the image gets bigger. We then can create also actions for more popular ML stacks as also discussed.

[elleobrien]

Our image does not support that action out of the box (nor even their self-hosted runner). The actor has to implement a cache system to make that available.

Hm, I've definitely used the setup-python out of the box on the GH self-hosted runner. For example, last week I made a workflow using it like so:
https://github.com/andronovhopf/texts_from_guy/blob/master/.github/workflows/myci.yaml

I didn't have to implement a cache system on the self-hosted runner. Maybe I'm misunderstanding something?

[DavidGOrtega]

Did you used a self-hosted runner? Your workflow says the opposite. You run the workflow within a GH runner.

Self runner is where the runner is running on your end, maybe in your local machine.
Thats actually the only way to run GPU in CML, running your self-hosted runner

https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

[elleobrien]

ahhh i see. you're right I used a GitHub runner, not a self-hosted. Thanks for taking a look :)

[jbencook]

I got this working with the dvcorg/cml:latest image. By default, the container service mounts /opt/hostedtoolcache to /__t in the container, but that breaks the Python installs which are compiled to live in /opt/hostedtoolcache specifically.

But the container syntax lets you add mounted volumes. Here's a workflow file that worked for me:

name: test-setup-python

on: [push]

jobs:
  run:
    runs-on: [ubuntu-latest]
    container:
      image: docker://dvcorg/cml:latest
      volumes:
        - /opt/hostedtoolcache:/opt/hostedtoolcache

    steps:
      - uses: actions/checkout@v2
      
      - name: Set up Python
        uses: actions/setup-python@v1
        with:
          python-version: '3.x'
          
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"
      
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip 
          pip install sklearn

It also works with setup-python@v2 and it works on a self-hosted runner. I'm pretty sure the base image will need to be Ubuntu 18.04, but it looks like all your images are so that wouldn't be a problem.

IMO this is better than shipping with your own Python because matrix testing works and it would make CML workflows more similar to other GitHub Actions setups. I'm happy to help with PRs to update examples if you think this is useful.

NOTE: to get it to work in a self-hosted runner, I added AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache/ to my .env file.

[DavidGOrtega]

@jbencook that's fantastic!

[DavidGOrtega]

@jbencook I can't make it work with self-hosted runners

I added

ENV AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache/

in the image (my own image). I don't see how the image is knowing about that volume locally (I expect it to download it)

Additionally my preferred way to launch a self-hosted runner is using CML docker image and the runner is not generating it either

docker run --name runner3 --rm  --gpus all
  -e RUNNER_IDLE_TIMEOUT=1800     
  -e RUNNER_LABELS=gpu     
  -e RUNNER_REPO=$repo     
  -e repo_token=$repo_token
  dvcorg/cml:latest

[jbencook]

If I use setup-python@v2, then the following runner works for me on my local GPU box:

docker run --name runner3 --rm  --gpus all
  -e AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache/ \
  -e RUNNER_IDLE_TIMEOUT=1800     
  -e RUNNER_LABELS=gpu     
  -e RUNNER_REPO=$repo     
  -e repo_token=$repo_token
  dvcorg/cml:latest

That's just the latest image from your Docker Hub. I would expect adding ENV AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache/ to your Dockerfile to work if you switch to setup-python@v2.

It seems to be a problem in the semantic version matching in setup-python@v1. I've got debug logs turned on in my repo. You can see the broken Set up Python step with v1 here and the working step with v2 here.

It's not as good if it doesn't work with v1. It's strange to me that that would be different in the container runner vs a runner on the actual host. Maybe it's OK to say setup-python needs to be at least v2 to work? Not sure. Those actions are a pain to debug, but happy to help if you all think it's important.

Also, pretty cool that your runners de-register themselves :)

[DavidGOrtega]

@jbencook brilliant! It works like a charm!

it's OK to say setup-python needs to be at least v2 to work? Not sure

I would say Im pretty happy with that solution. Actually it's a bit like actions/checkout@v1 that gives you a detached HEAD that not anyone will find it good enough (specially DVC users).

Also, pretty cool that your runners de-register themselves :)

Thanks! As might noted they just need a TOKEN with actions API permissions generating the actions token themselves, then deregister but they also have the RUNNER_IDLE_TIMEOUT that GH runner does not have and the ability to cleanup themselves using docker-machine, have a look to the cloud demo.
We are working on that actually, to be able to be take the most from spot instances not having to have a third actor or orchestrator, just the workflow.

Why don't you do a PR with the Dockerfile including the env? I did it in mine.

ENV RUNNER_PATH=/home/runner
ENV RUNNER_ALLOW_RUNASROOT=1
ENV AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache/

I will be honoured to approve it ❤️

[jbencook]

Thanks @DavidGOrtega! I'm not super familiar with docker-machine, but I'll definitely check that out. I've been thinking about trying to setup autoscaling runners for our Applied ML team and this is a big improvement over how I was approaching it.