[WIP] Fork dask-jobqueue's testing suite images

[MilesCranmer]

This is the start of a fork of the dask-jobqueue testing suite for ClusterManagers.jl. ClusterManagers.jl seems to keep breaking due to the lack of a proper testing suite, so I want to start working on this fork of the dask-jobqueue suite which has working CI for the major cluster managers.

The testing suite has been copied to the ci folder. I also copied the license information from dask-jobqueue as well into the ci folder. Since this is only used in testing it will not get packaged with the distributed copy of ClusterManagers.jl so should not affect the license of the project.

However maybe it is preferable to make a full fork of dask-jobqueue in the JuliaParallel organization, and clone the repo during the CI test? At least that way it would preserve the git history. In any case we can worry about that later, as we can just apply the git patch from this PR.

Thanks to @lesteve for pointing this out in #105.

---

TODO (help requested)

• Change conda env to install juliaup rather than dask
• Using that juliaup, install a version of julia to all docker images
• Set up dockerhub
  • dask-jobqueue builds docker images on a cronjob, and uploads those images to dockerhub. Maintainers (@kescobo + @Moelf + @vchuravy + ?) could you please set up a dockerhub account and add DOCKER_USERNAME and DOCKER_PASSWORD as secrets to this repo?
• Fork the GitHub workflows from dask-jobqueue
• Decide whether to make a full fork of dask-jobqueue, or just copy the testing suite with local changes here.
• Update ci/slurm.sh to run Julia test
• Update ci/pbs.sh to run Julia test
  • (Write test to actually use?)
• Update ci/sge.sh to run Julia test
  • (Write test to actually use?)
• Update ci/htcondor.sh to run Julia test
  • (Write test to actually use?)

Maintainers can feel free to work directly on my branch.

I would also appreciate if we could merge this before #179 (and others) is fixed, as it will give us a way of testing it. Otherwise we would be fixing #179 without knowing whether it even works.

[Moelf]

I'd like to help but I don't have privilege to things.

Maybe you should hard-fork it.

[MilesCranmer]

I guess if you can't push directly, you could submit PRs to https://github.com/MilesCranmer/ClusterManagers.jl/tree/master (or I can just add you to that repo with push permissions)

[MilesCranmer]

Okay @Moelf you should have push permissions on my repo. Feel free to edit directly

[MilesCranmer]

We can probably remove the conda logic entirely and just install julia via juliaup in the docker images

[vchuravy]

Could we do https://docs.github.com/en/actions/publishing-packages/publishing-docker-images#publishing-images-to-github-packages instead of having t maintain a separate account?

[MilesCranmer]

Sure. Could you set up the permissions for that and set the relevant secrets? (if needed, I don't know how it works)

[vchuravy]

Sure. Could you set up the permissions for that and set the relevant secrets? (if needed, I don't know how it works)

I don't know how it works either, I invited you to the JuliaParallel org and if you create a new repo for the docker images I can make you admin both here and there

[Moelf]

@vchuravy I can also try to take a look into making a Docker repo it if you trust me xD

[giordano]

For what is worth, I can confirm publishing images to the github container registry using github actions is quite easy, and particularly nice because you don't need an extra account, as mentioned above. I've done that a few times, I could help with that (but I see there are plenty of other problems here).

[MilesCranmer]

Just to note – I unfortunately can't find time to push this forward. So if someone wants to try, please feel free to go ahead with it.