Use pre-built docker images to speed up CI #432

riedel · 2020-05-21T14:43:18Z

addressing #424, started with htcondor just to check travis, then will add others

… htcondor

riedel · 2020-05-21T17:51:16Z

This is the current autobuild config:

riedel · 2020-05-21T17:59:24Z

ci/htcondor.sh

@@ -3,26 +3,21 @@
 function jobqueue_before_install {
    docker version

-    docker run -d --name jobqueue-htcondor-mini htcondor/mini:el7 # might fail if called as script
+    docker run --rm -d --name jobqueue-htcondor-mini tillkit/dask-jobqueue-ci:htcondor #TODO: changeme on final


htcondor will be changed to docker-compose too, I did not want to put it into one PR, but same as compose: imagename needs to be adjusted once an official autobuild is there

riedel · 2020-05-21T18:00:26Z

ci/htcondor.sh

-	pip3 install -e .;
-	pip3 install pytest;
-	rm -f /var/log/condor/*
+	pip3 install --upgrade --upgrade-strategy eager -e .;


this is something that should probably done on the images to check changes in dask and distributed

riedel · 2020-05-21T18:01:23Z

ci/htcondor.sh

 	chown -R submituser:submituser /dask-jobqueue 
+	pip3 freeze


Also a bit unrelated but I could not help it :)

riedel · 2020-05-21T18:02:02Z

ci/htcondor.sh

@@ -40,3 +35,8 @@ function jobqueue_after_script {
   "
 }
 [[ "${BASH_SOURCE[0]}" != "${0}" ]] || jobqueue_after_script # excute if called as script
+
+function jobqueue_cleanup {
+    docker stop jobqueue-htcondor-mini


Probably this should also not be in the PR

riedel · 2020-05-21T18:03:15Z

ci/htcondor/Dockerfile

@@ -0,0 +1,9 @@
+# base image
+ARG  HTCONDOR_VERSION
+FROM htcondor/mini:${HTCONDOR_VERSION}${HTCONDOR_VERSION:+-}el7


needed to have a HTCondor image. Actually for HTCondor there is close to 0 speedup.

riedel · 2020-05-21T18:04:16Z

ci/pbs.sh

@@ -6,6 +6,7 @@ function jobqueue_before_install {

    # start pbs cluster
    cd ./ci/pbs
+    docker-compose pull


this seems to be needed to not build the docker files (I kept the build lines)

riedel · 2020-05-21T18:04:42Z

ci/pbs/docker-compose.yml

@@ -3,6 +3,7 @@ version: "2"
 services:

  master:
+    image: tillkit/dask-jobqueue-ci:pbs 


needs to be changed to the official

riedel · 2020-05-21T18:05:33Z

ci/sge/Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:14.04
+FROM ubuntu:14.04 as base


I made this a multistage build because otherwise this will generate independent layers and the pull time increases

riedel · 2020-05-21T18:06:12Z

ci/sge/Dockerfile

@@ -14,6 +14,11 @@ RUN conda install -c conda-forge python=$PYTHON_VERSION dask distributed pytest

 COPY ./*.sh /
 COPY ./*.txt /
+
+FROM base as slave


We tag this target with a hook later

riedel · 2020-05-21T18:06:38Z

ci/sge/docker-compose.yml

@@ -1,11 +1,12 @@
-version: "2.1"
+version: "3.4"


Needed to support targets for the builds

riedel · 2020-05-21T18:07:00Z

ci/sge/docker-compose.yml

    build:
      context: .
-      dockerfile: Dockerfile-master


removed and replaced with multistage target

riedel · 2020-05-21T18:07:25Z

ci/sge/hooks/post_build

@@ -0,0 +1,3 @@
+#!/bin/bash


this is automatically executed once the master image is build

This file seems related to autodocker IIUC (e.g. https://docs.docker.com/docker-hub/builds/advanced/#custom-build-phase-hooks)? Can we remove it?

riedel · 2020-05-21T18:14:35Z

Some speedup, but some are slower. The caching of travis is probably relevant here. The package caches might be much "warmer" that the docker image speed.

before:

after:

riedel · 2020-05-21T18:20:13Z

Actually slurm is still building. damn

riedel · 2020-05-21T18:46:18Z

There is probably a lot of room for optimization, but now I see at least either same speed or speed up:

riedel · 2020-05-21T20:12:14Z

@lesteve , it works now. I have 2 questions:

Is it OK if, remove all HTCondor stuff from this PR to make it look smaller and do it together with everything related to Make HTCondor CI a bit closer to the other CI #427
should the autobuilt docker images go into https://hub.docker.com/u/daskdev ? (@jacobtomlinson, would you be the right one to do this? )

lesteve · 2020-05-22T04:38:52Z

Very nice to see this progress!

Is it OK if, remove all HTCondor stuff from this PR to make it look smaller and do it together with everything related to #427

In general I prefer to keep PRs focused on one thing as much as possible because it makes it so much easier to review. If this is not too much work to remove the HTCondor stuff, that would be great!

should the autobuilt docker images go into hub.docker.com/u/daskdev ? (@jacobtomlinson, would you be the right one to do this? )

No strong opinion on this. I would say this would be probably faster to create our own project in dockerhub e.g. dask-jobqueue (or daskjobqueue maybe you can not have dashes in the name?). Another Dask project that has its own project on dockerhub it like this is dask-gateway (https://hub.docker.com/u/daskgateway). I am not sure what the daskdev images are used for to be honest. Also the potentiality for reuse of our docker images outside of Dask-Jobqueue is very small I would say.

This reverts commit 0097335.

guillaumeeb · 2020-05-22T09:13:26Z

If I'm not mistaken, this PR will fix #52. In this one @mrocklin offered to give daskdev credentials if needed. This could be a good place.

jacobtomlinson · 2020-05-22T09:13:38Z

I would say this would be probably faster to create our own project in dockerhub

I see no reason why we couldn't add folks to the existing project.

Another Dask project that has its own project on dockerhub it like this is dask-gateway (https://hub.docker.com/u/daskgateway).

There are a bunch of things in dask-gateway that I would like to consolidate back into mainstream dask. Consolidating the helm repo is one, docker images another.

I am not sure what the daskdev images are used for to be honest.

They are used by dask-kubernetes, dask-cloudprovider, the Dask Helm Chart and generally just folks wanting to use Dask in Docker.

Also the potentiality for reuse of our docker images outside of Dask-Jobqueue is very small I would say.

While true I see no harm in consolidating in a single location. Also helps share out maintenance with multiple people being able to access them. In the same way the dask-jobqueue repo lives under the dask org on GitHub.

riedel · 2020-05-22T09:26:16Z

In general I prefer to keep PRs focused on one thing as much as possible because it makes it so much easier to review. If this is not too much work to remove the HTCondor stuff, that would be great!

done (was luckily in a separate commit)

Also the potentiality for reuse of our docker images outside of Dask-Jobqueue is very small I would say.

I see some potential use for people to test their jobs (at home) before submitting to clusters (I like to do this sometimes without good internet). Sure would need a bit more docs

lesteve · 2020-05-22T10:04:31Z

Thanks @jacobtomlinson for your detailed answer!

I am a complete newbie regarding dockerhub, so I guess what we would need is something like:

necessary rights on dockerhub to push Docker images on daskdev. I am guessing this is done at an individual level? If that's the case, could you give the necessary rights to @guillaumeeb and me as a start? My username on dockerhub is lesteve.
some inputs (e.g. a link) as to how I enable autodocker on the dask-jobqueue/dask-jobqueue repo and configure it to push into daskdev/dask-jobqueue-ci on DockerHub

Is there is anything else I am missing?

jacobtomlinson · 2020-05-22T10:23:37Z

Sure. Happy to add you to the org on Docker Hub. I'll follow the standard Dask process of requesting another owner approval by email. I'll let you know when this is done.

I tend to avoid the auto builds on Docker Hub as they are often slow and you have limited control over them. They work by waiting for GitHub webhooks and trigger a build on their side when you push to master.

However we are likely already building the Docker image as part of CI on master in order to test it, so it makes sense to push that image up to Docker Hub instead.

Checkout the deploy script we have in dask/dask-docker. It needs you to configure a DOCKER_USERNAME and DOCKER_PASSWORD in the CI secrets but then pushes all images based on the Docker compose config.

riedel · 2020-05-22T10:43:08Z

it is a bit of a chicken and egg problem here as the goal was not rebuilding them too often, but I left the code for building in there. Maybe travis could check the age and rather use build instead of pull once a week. everything seems a bit hacky. Sure an option would be creating a separate repo like in dask-docker but this would quite a change. Maybe a branch with a different CI target?

…

________________________________ Von: Jacob Tomlinson <notifications@github.com> Gesendet: Freitag, 22. Mai 2020 12:23 An: dask/dask-jobqueue Cc: Till Riedel; Author Betreff: Re: [dask/dask-jobqueue] [WIP] use docker autobuilds to speed up CI (#432) Sure. Happy to add you to the org on Docker Hub. I'll follow the standard Dask process of requesting another owner approval by email. I'll let you know when this is done. I tend to avoid the auto builds on Docker Hub as they are often slow and you have limited control over them. They work by waiting for GitHub webhooks and trigger a build on their side when you push to master. However we are likely already building the Docker image as part of CI on master in order to test it, so it makes sense to push that image up to Docker Hub instead. Checkout the deploy script we have in dask/dask-docker. It needs you to configure a DOCKER_USERNAME and DOCKER_PASSWORD in the CI secrets but then pushes all images based on the Docker compose config. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

lesteve · 2020-05-22T11:24:39Z

Maybe travis could check the age and rather use build instead of pull once a week. everything seems a bit hacky.

I would be in favour favour of something like this:

a weekly Travis Cron job builds and pushes the Docker image
the "normal" CI would reuse the Docker image and not build it

jacobtomlinson · 2020-05-22T13:14:47Z

I've added @lesteve to Docker Hub, @guillaumeeb if you can share your Docker Hub username I'll add you too.

Feel free to ping me if you want any more input.

riedel · 2020-05-22T14:41:17Z

I will quickly add a build target to the ci/x.sh and you can configure travis. Ideally the travis job will run the push after running the other tests though to keep the CI from breaking. would be perfect if it could be triggered manually on PRs that contain dockerfile changes. But this is an extra and i am no travis guru...

________________________________ Von: Loïc Estève <notifications@github.com> Gesendet: Freitag, 22. Mai 2020 13:24 An: dask/dask-jobqueue Cc: Till Riedel; Author Betreff: Re: [dask/dask-jobqueue] [WIP] use docker autobuilds to speed up CI (#432)

Maybe travis could check the age and rather use build instead of pull once a week. everything seems a bit hacky.

I would be in favour favour of something like this: a weekly Travis Cron job builds and pushes the Docker image the "normal" CI would reuse the Docker image and not build it — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

jacobtomlinson · 2020-05-22T16:13:05Z

would be perfect if it could be triggered manually on PRs that contain dockerfile changes.

I know it's possible to do this kind of thing on GitHub Actions. Haven't tried on Travis.

guillaumeeb · 2020-05-22T16:58:57Z

@guillaumeeb if you can share your Docker Hub username I'll add you too.

It's also guillaumeeb! Thanks @jacobtomlinson.

riedel · 2020-05-22T20:18:36Z

It seems you cannot distinguish multiple cron jobs on one branch easily (travis-ci/travis-ci#8820).

I check the day of the week : [[ $TRAVIS_EVENT_TYPE == "cron" && $(date +%u) == 5 ]] && docker-compose build

I also looked at the deploy script, that looks really complicated: Is there any reason not to just use docker-compose push? (works for me locally).

If you set up a docker repo and set the DOCKER_USERNAME and DOCKER_PASSWORD in the CI secrets I can give it a try.

lesteve · 2020-05-23T07:45:38Z

I am going to push the docker images by hand to daskdev/dask-jobqueue (using this PR code) and make sure that we can reuse them in the CI with this PR CI.

One that is done I will merge this PR. We can work on building the Docker images automatically in a weekly Travis Cron job in a separate PR.

Let me know if you see a problem with this approach!

riedel · 2020-05-23T09:03:33Z

Thanks @lesteve for this very good suggestion! Learning a lot on how to do PR in small batches. (I am always worried that once the normal work week starts I will run out of time and thus I am really happy to get things merged early, so nobody needs to wait for me). I will then first be able to finally rework the htcondor bit. (Maybe we can then even hold the weekly schedule before the next PR :) )

lesteve · 2020-05-23T09:22:09Z

I removed what I thought was a autodocker-related file in the last commit. If this passes, I'll 'merge this one.

lesteve · 2020-05-23T09:25:16Z

OK the build is green, merging this one, thanks a lot @riedel!

About small PRs yeah as a maintainer this is so much easier to review ... as a PR submitter it is a bit of a pain sometimes ... so need to find a compromise in between (not that easy).

To be fair, in this particular case, the fact that you don't have permissions to set-up things yourself adds some friction.

jacobtomlinson · 2020-05-27T12:52:14Z

Just a note I have found creating CI jobs with various cron schedules is much easier in GitHub Actions. Don't know if you were thinking about migrating things there, but it is nice to configure and use.

lesteve · 2020-05-27T13:26:19Z

It seems that some projects have migrated to Github Actions, I have to say I was not plannning to do it in the near future ... but if someone wants to have a look and do a PR that would be more than welcome !

using autobuild for htcondor

0097335

riedel force-pushed the docker_autobuild branch from 974c558 to 2e17146 Compare May 21, 2020 17:09

patch other compose files and fix error when updating dependencies in…

6e37196

… htcondor

riedel force-pushed the docker_autobuild branch from 2e17146 to 6e37196 Compare May 21, 2020 17:48

riedel changed the title ~~[WIP] Try docker autobuilds to speed up CI~~ [WIP] use docker autobuilds to speed up CI May 21, 2020

riedel commented May 21, 2020

View reviewed changes

ci/sge/docker-compose.yml

@@ -1,11 +1,12 @@

version: "2.1"

version: "3.4"

Copy link

Member Author

riedel May 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed to support targets for the builds

riedel commented May 21, 2020

View reviewed changes

ci/sge/docker-compose.yml

build:

context: .

dockerfile: Dockerfile-master

Copy link

Member Author

riedel May 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed and replaced with multistage target

riedel commented May 21, 2020

View reviewed changes

keep slurm from force building

0c6ac22

riedel added the all job schedulers label May 21, 2020

riedel requested a review from lesteve May 21, 2020 18:53

Revert "using autobuild for htcondor"

eaf1a6f

This reverts commit 0097335.

Move to daskdev/dask-jobqueue

a9312de

riedel marked this pull request as ready for review May 23, 2020 08:52

riedel changed the title ~~[WIP] use docker autobuilds to speed up CI~~ use docker autobuilds to speed up CI May 23, 2020

riedel changed the title ~~use docker autobuilds to speed up CI~~ use docker builds to speed up CI May 23, 2020

riedel changed the title ~~use docker builds to speed up CI~~ use pre-built docker images to speed up CI May 23, 2020

riedel mentioned this pull request May 23, 2020

Periodically build images for CI #433

Closed

Remove autodocker hook.

741c11b

lesteve changed the title ~~use pre-built docker images to speed up CI~~ Use pre-built docker images to speed up CI May 23, 2020

lesteve merged commit b2a5ac8 into dask:master May 23, 2020

lesteve mentioned this pull request May 23, 2020

Pooling Docker test images #52

Closed

ocaisa mentioned this pull request May 25, 2020

Update to CI in jobqueue E-CAM/jobqueue_features#56

Closed

andersy005 mentioned this pull request Jul 27, 2020

Add GH action to periodically build docker images #455

Merged

guillaumeeb mentioned this pull request Aug 30, 2022

Check if CI really don't build Docker image if not needed #566

Closed

Use pre-built docker images to speed up CI #432

Use pre-built docker images to speed up CI #432

riedel commented May 21, 2020 •

edited

riedel commented May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

riedel May 21, 2020

lesteve May 23, 2020

riedel commented May 21, 2020 •

edited

riedel commented May 21, 2020

riedel commented May 21, 2020 •

edited

riedel commented May 21, 2020 •

edited

lesteve commented May 22, 2020

guillaumeeb commented May 22, 2020

jacobtomlinson commented May 22, 2020

riedel commented May 22, 2020

lesteve commented May 22, 2020 •

edited

jacobtomlinson commented May 22, 2020

riedel commented May 22, 2020 via email

lesteve commented May 22, 2020

jacobtomlinson commented May 22, 2020

riedel commented May 22, 2020 via email

jacobtomlinson commented May 22, 2020

guillaumeeb commented May 22, 2020

riedel commented May 22, 2020

lesteve commented May 23, 2020

riedel commented May 23, 2020

lesteve commented May 23, 2020

lesteve commented May 23, 2020 •

edited

jacobtomlinson commented May 27, 2020

lesteve commented May 27, 2020

Use pre-built docker images to speed up CI #432

Use pre-built docker images to speed up CI #432

Conversation

riedel commented May 21, 2020 • edited

riedel commented May 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

riedel commented May 21, 2020 • edited

riedel commented May 21, 2020

riedel commented May 21, 2020 • edited

riedel commented May 21, 2020 • edited

lesteve commented May 22, 2020

guillaumeeb commented May 22, 2020

jacobtomlinson commented May 22, 2020

riedel commented May 22, 2020

lesteve commented May 22, 2020 • edited

jacobtomlinson commented May 22, 2020

riedel commented May 22, 2020 via email

lesteve commented May 22, 2020

jacobtomlinson commented May 22, 2020

riedel commented May 22, 2020 via email

jacobtomlinson commented May 22, 2020

guillaumeeb commented May 22, 2020

riedel commented May 22, 2020

lesteve commented May 23, 2020

riedel commented May 23, 2020

lesteve commented May 23, 2020

lesteve commented May 23, 2020 • edited

jacobtomlinson commented May 27, 2020

lesteve commented May 27, 2020

riedel commented May 21, 2020 •

edited

riedel commented May 21, 2020 •

edited

riedel commented May 21, 2020 •

edited

riedel commented May 21, 2020 •

edited

lesteve commented May 22, 2020 •

edited

lesteve commented May 23, 2020 •

edited