cml-cloud-run with gpus

[elleobrien]

I'm testing out using an EC2 GPU w/ the cloud container cml-gpu-py3-cloud-runner. I wanted to make sure I'm on the right track:

name: train-my-model

on: [push]

jobs:
  deploy-cloud-runner:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/cml-gpu-cloud-runner

    steps:
      - name: deploy
        env:
          repo_token: ${{ secrets.REPO_TOKEN }} 
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          echo "Deploying..."
          MACHINE="CML-$(openssl rand -hex 12)"
          docker-machine create \
              --driver amazonec2 \
              --amazonec2-instance-type g3s.xlarge \
              --amazonec2-region us-east-2 \
              --amazonec2-zone a \
              --amazonec2-vpc-id vpc-76f1f01e \
              --amazonec2-ssh-user ubuntu \
              $MACHINE
          eval "$(docker-machine env --shell sh $MACHINE)"
          ( 
          docker-machine ssh $MACHINE "sudo mkdir -p /docker_machine && sudo chmod 777 /docker_machine" && \
          docker-machine scp -r -q ~/.docker/machine/ $MACHINE:/docker_machine && \
          docker run --name runner -d \
            -v /docker_machine/machine:/root/.docker/machine \
            -e RUNNER_IDLE_TIMEOUT=120 \
            -e DOCKER_MACHINE=${MACHINE} \
            -e RUNNER_LABELS=cml \
            -e repo_token=$repo_token \
            -e NVIDIA_VISIBLE_DEVICES=all \
            -e RUNNER_REPO=https://github.com/andronovhopf/test_cloud \
           dvcorg/cml-gpu-py3-cloud-runner && \
               sleep 20 && echo "Deployed $MACHINE"
          ) || (echo y | docker-machine rm $MACHINE && exit 1)
  train:
    needs: deploy-cloud-runner
    runs-on: [self-hosted,cml]
    
    steps:
      - uses: actions/checkout@v2

      - name: cml_run
        env:
          repo_token: ${{ secrets.REPO_TOKEN }} 
        run: |
          nvidia-smi

This isn't working yet; looks to be issues getting the drivers setup on the self-hosted runner. I'm betting I have a flag wrong somewhere in the deploy job. I tried adding the flag --gpus all to docker run but that didn't work. Any ideas?

[DavidGOrtega]

Ok. You need to have installed NVIDIA and nvidia-docker in the remote host machine.

Using an already prepared ami

You could use an already prepared ami or buld it your own.

AWS has ami-08a5e4d75a3914234 and you would use it specifying it within docker-machine like this.

docker-machine create \
              --driver amazonec2 \
              --amazonec2-instance-type g3s.xlarge \
              --amazonec2-vpc-id vpc-76f1f01e \
              --amazonec2-ssh-user ubuntu \
              --amazonec2-ami "ami-08a5e4d75a3914234" \
              $MACHINE

DIY

The other way and Its the way we should enforce since discrepancies in other vendors would be installing it on your side.

echo "Deploying..."

          RUNNER_LABELS="cml,aws"
          RUNNER_REPO="https://github.com/DavidGOrtega/3_tensorboard"
          MACHINE="cml$(date +%s)"

          docker-machine create \
            --driver amazonec2 \
            --amazonec2-instance-type g4dn.xlarge \
            --amazonec2-vpc-id vpc-06bc773d85a0a04f7 \
            --amazonec2-ssh-user ubuntu \
            --amazonec2-ami ami-085925f297f89fce1 \
            $MACHINE

          eval "$(docker-machine env --shell sh $MACHINE)"

          (
          docker-machine ssh $MACHINE "sudo mkdir -p /docker_machine && sudo chmod 777 /docker_machine" && \
          docker-machine scp -r -q ~/.docker/machine/ $MACHINE:/docker_machine && \

          docker-machine ssh $MACHINE "sudo apt update && sudo apt install -y ubuntu-drivers-common && sudo ubuntu-drivers autoinstall" && \

          docker-machine restart $MACHINE && \
          eval "$(docker-machine env --shell sh $MACHINE)" && \

          docker-machine ssh $MACHINE "curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - && curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list && sudo apt update && sudo apt install -y nvidia-container-toolkit && sudo systemctl restart docker" && \

          docker run --name runner --gpus all -d \
            -v /docker_machine/machine:/root/.docker/machine \
            -e DOCKER_MACHINE=$MACHINE \
            -e repo_token=$repo_token \
            -e RUNNER_LABELS=$RUNNER_LABELS \
            -e RUNNER_REPO=$RUNNER_REPO \
            -e RUNNER_IDLE_TIMEOUT=120 \
            dvcorg/cml-cloud-runner:latest && \

          sleep 20 && echo "Deployed $MACHINE"
          ) || (docker-machine rm -y -f $MACHINE && exit 1)

Aside of minor differences with your code like the MACHINE id generator (please use this one the older is not supported in azure) its very similar to yours.

installs nvidia driver

 docker-machine ssh $MACHINE "sudo apt update && sudo apt install -y ubuntu-drivers-common && sudo ubuntu-drivers autoinstall" && \

restarts remote machine

          docker-machine restart $MACHINE && \
          eval "$(docker-machine env --shell sh $MACHINE)" && \

installs nvidia-docker

          docker-machine ssh $MACHINE "curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - && curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list && sudo apt update && sudo apt install -y nvidia-container-toolkit && sudo systemctl restart docker" && \

then when launching the runner you have to set nvidia-docker --gpus flag

docker run --name runner --gpus all -d \

We are working to provide dvcorg ami in every vendor. This will make this process cleaner and faster.

[elleobrien]

Thanks, @DavidGOrtega! Finally getting back to this....

It looks like the AMI specified isn't found:
Error creating machine: Error in driver during machine creation: Error launching instance: InvalidAMIID.NotFound: The image id '[ami-085925f297f89fce1]' does not exist

Should this be happening? I can try with any of the deep learning AMIs so not a huge problem, just wanted to check if this is expected

[elleobrien]

Also, it looks like git show origin/master isn't working on my GPU workflow- I'm guessing some mixup between GITHUB_TOKEN and REPO_TOKEN credentials?

Error here: https://github.com/andronovhopf/style/runs/797106414?check_suite_focus=true

Any ideas?

[DavidGOrtega]

It looks like the AMI specified isn't found:

Thats one of the most weird things I find on AWS, Azure... All AMIs are not available in every region. It's a bit weird but I actually used that. That's one of the reasons we should provide a proper GPU AMI.

Also, it looks like git show origin/master isn't working on my GPU workflow

It's a matter of setting the git user maybe?... I remember that when this project was a whole action we were authenticating as

git config --local user.email  'action@github.com'
git config --local user.name 'GitHub Action'

Im reviewing your repo...

[DavidGOrtega]

@andronovhopf were you able to fix it?

[elleobrien]

I did fix it, but to be honest I'm not sure what did the trick. For example, this .yaml with git show origin/master in the workflow is running fine:

https://github.com/iterative/cml_cloud_case/blob/experiment/.github/workflows/cml.yaml

I did experiment with different repo token permissions ( I am currently using one repo token throughout that has repository read/write and workflow privileges, but I'm not positive workflow privileges are really needed). But not sure if that's what fixed it.

[DavidGOrtega]

@andronovhopf feel free to open it again