Testing cml-cloud-run

[elleobrien]

I'm working with the following .yaml:

name: train-my-model

on: [push]

jobs:
  deploy-cloud-runner:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/cml-cloud-runner

    steps:
      - name: deploy
        env:
          repo_token: ${{ secrets.REPO_TOKEN }} 
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          echo "Deploying..."
          MACHINE="CML-$(openssl rand -hex 12)"
          docker-machine create \
              --driver amazonec2 \
              --amazonec2-instance-type t2.micro \
              --amazonec2-region us-east-2 \
              --amazonec2-zone a \
              --amazonec2-vpc-id vpc-76f1f01e \
              --amazonec2-ssh-user ubuntu \
              $MACHINE
          eval "$(docker-machine env --shell sh $MACHINE)"
          ( 
          docker-machine ssh $MACHINE "sudo mkdir -p /docker_machine && sudo chmod 777 /docker_machine" && \
          docker-machine scp -r -q ~/.docker/machine/ $MACHINE:/docker_machine && \
          docker run --name runner -d \
            -v /docker_machine/machine:/root/.docker/machine \
            -e RUNNER_IDLE_TIMEOUT=120 \
            -e DOCKER_MACHINE=${MACHINE} \
            -e RUNNER_LABELS=cml \
            -e repo_token=$repo_token \
            -e RUNNER_REPO=https://github.com/iterative/cml_base_case \
           dvcorg/cml-cloud-runner && \
          sleep 20 && echo "Deployed $MACHINE"
          ) || (echo y | docker-machine rm $MACHINE && exit 1)
  train:
    needs: deploy-cloud-runner
    runs-on: [self-hosted,cml]
    
    steps:
      - uses: actions/checkout@v2

      - name: cml_run
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }} 
        run: |
          pip install -r requirements.txt
          python train.py
        
          cat metrics.txt >> report.md
          cml-publish confusion_matrix.png --md >> report.md
          cml-send-comment report.md

Note that the job train depends on deploy-cloud-runner, not deploy as in #108 (deploy threw an error). The first job, deploy-cloud-runner is running fine, but train is throwing an inscrutable error:

Project repo is here: https://github.com/andronovhopf/test_cloud

Any ideas?

[elleobrien]

lol i think it's that the machine turns itself off at the end ofdeploy-cloud-runner

[elleobrien]

OK, it seems the problem is that after deploy-cloud-runner the runner is turned off; so there's no machine for train to run on. But even if I remove docker-machine rm $MACHINE && exit 1 from the deploy-cloud-runner job, GitHub Actions will still turn stop & remove containers and the VPN.

So the question is... how do I make sure the runner is available to train?

[DavidGOrtega]

lol i think it's that the machine turns itself off at the end of deploy-cloud-runner

I think that you are speaking abut this

|| (echo y | docker-machine rm $MACHINE && exit 1)

this means that if you have ANY problem executing any of the instructions inside the parenthesis it will be executed to cleanup the deployed machine hence the || (acts as a try catch). You should NEVER remove this or you will end up with a runner that can not remove itself if fails inside the parenthesis.
The instructions inside the parenthesis are providing the remote machine the files needed to be able to remove itself.

If you see the first valid logs you will see that the machine was deployed without any issue so at this point GH can cleanup the job since it has been successful.
At this point you have a runner totally independent in AWS waiting for 2 minutes

-e RUNNER_IDLE_TIMEOUT=120 \

to handle jobs before it remove itself. The cleanup code that you removed later never executed!

What's the issue then?

You are telling the runner to register to another repo!

-e RUNNER_REPO=https://github.com/iterative/cml_base_case \

Runners register to repos.
You can see them in settings->actions

Right now you are creating a runner that registers for the repo https://github.com/iterative/cml_base_case while leaving the other job orphan. When job deploy-cloud-runner finishes, job train starts but no runner is ready! Thats why you have that non descriptive error.

Other considerations

After this fix another issue that you might run with is the $repo_token. The original $GITHUB_TOKEN that was automatically given by GH has not enough permissions to use the Actions API as stated in #108

[elleobrien]

Ah, beautiful- thanks for the detailed explanation. I made some changes to my workflow file like you suggested:

• set RUNNER_REPO to https://github.com/andronovhopf/test_cloud
• created REPO_TOKEN with repo privileges
• in the train job, I changed repo_token: ${{ secrets.GITHUB_TOKEN }} → repo_token: ${{secrets.REPO_TOKEN}}. I'm not positive if the environmental variable is passed from the first job to train, but didn't want to overwrite it

It's looking good! Now train runs, although the step cml_run fails because pip is not installed. It looks like the Docker container doesn't come with python installed; do we plan to add it? Or is this something we'll ask users to install manually?

[dmpetrov]

@andronovhopf there is another docker with python - cml-py3-cloud-runner

[elleobrien]

Still no dice- I am using container: docker://dvcorg/cml-py3-cloud-runner, and have tried pip & pip3. Doesn't seem to be installed

https://github.com/andronovhopf/test_cloud/runs/762835189?check_suite_focus=true

[DavidGOrtega]

@andronovhopf the change that @dmpetrov points needs to be done in the runner that you are launching

change

docker run --name runner -d \
            -v /docker_machine/machine:/root/.docker/machine \
            -e RUNNER_IDLE_TIMEOUT=120 \
            -e DOCKER_MACHINE=${MACHINE} \
            -e RUNNER_LABELS=cml \
            -e repo_token=$repo_token \
            -e RUNNER_REPO=https://github.com/andronovhopf/test_cloud \
           dvcorg/cml-cloud-runner && \
               sleep 20 && echo "Deployed $MACHINE"
          ) || (echo y | docker-machine rm $MACHINE && exit 1)

to

...
           dvcorg/cml-py3-cloud-runner && \
           sleep 20 && echo "Deployed $MACHINE"
...

I have seen that you added the install of pip here. Its not needed that code if you use the proper image for your runner

[elleobrien]

@DavidGOrtega got it, trying that now. So it isn't important to specify container: docker://dvcorg/cml-py3-cloud-runner in line 8 of my .yaml, because we don't need python for the job of launching the runner. Only in the docker run function. Think I got it 👍

Edit: yep this works now I'm running Python training jobs.

[DavidGOrtega]

@andronovhopf you got it.
Deploy job needs our runner image because its already loaded with docker-machine etc... Ideally if we finally wrap this inside a single command that command would be also available there.

Train job is the one actually that needs the proper python version.

To simplify things and not generating confusing I would have loved to just have one image only cml but in our internal discussions we decided that the image would be very large and too much a dark box.

[DavidGOrtega]

Closing this. Feel free to reopen it.