[WIP] Add CircleCI for CI #1295
Conversation
| filters: | ||
| branches: | ||
| only: | ||
| - master |
There was a problem hiding this comment.
You probably don't want to actually do this, as it will prevent you from easily doing binary builds by opening PRs.
There was a problem hiding this comment.
if I want only a subset of the builds to run, how would you recommend me doing this, with an approach similar to PyTorch should_run_job.sh ?
There was a problem hiding this comment.
In a vacuum, what I would have expected to see is that by default there is some limited set of builds that run on master, but then there is an easy way to turn on all of the builds by editing CircleCI config, if you need it. That's what the {%- if True %-} block does today.
You can do a should_run_job.sh strategy, but if you enable too many builds by default on a PR, or too many builds which have exactly equal finish times, you'll hit GitHub rate limiting. We had exactly this problem. Another way to solve this problem is to use the parametrized jobs endpoint to trigger jobs yourself; ask @suo about it, he's been poking at it on main pytorch/pytorch repo.
| stable" | ||
|
|
||
| sudo apt-get update | ||
| sudo apt-get install docker-ce docker-ce-cli containerd.io |
There was a problem hiding this comment.
PyTorch CI pins docker-ce version and you should too. Whenever docker releases a new version it takes a day or so for nvidia to release updated nvidia-docker packages. Your CI will be broken in the meantime if you don't pin.
| set -e | ||
|
|
||
| curl -o conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh | ||
| sh conda.sh -b |
There was a problem hiding this comment.
If you're going to use soumith/conda-cuda image (which I'm not sure I condone, but I guess it's fine for now), you shouldn't need to reinstall conda; conda should already come on the image.
There was a problem hiding this comment.
This is a very good point. I originally tried using a basic nvidia/cuda:10.0 image because I wanted kind of minimal dependencies, conda-cuda was huge (20GB) and I wanted to understand a bit a few more things.
But it was in the end a bit more work and I just switched to soumith/conda-cuda, which I have now found where the original Dockerfile lives, so I will switch to using it and use the conda from there as well.
|
|
||
| /workspace/packaging/conda/switch_cuda_version.sh 10.0 | ||
|
|
||
| conda install -y pytorch cudatoolkit=10.0 -c pytorch-nightly |
There was a problem hiding this comment.
You'll be at the mercy of PyTorch not wantonly breaking vision if you do it this way ;)
There was a problem hiding this comment.
I know, but torchvision master currently requires PyTorch nightly (see e.g., #1262, so I think this is the simplest thing we can do?
| conda install -y pytest mock scipy | ||
|
|
||
| # torchvision dependency on av | ||
| conda install -y av -c conda-forge |
There was a problem hiding this comment.
Ideally, all of these dependencies should be preloaded on the Docker image, so you can just go straight to build and test immediately
|
Looks basically reasonable! |
|
Subsumed by #1298 |
This is PR is currently in a very raw state, but seems to work and CUDA tests pass.
Clean-up is required, but I'd requesting early feedback on the overall direction.