Automate release process

[jcrist]

Currently the release process is manual, and has several error-prone steps. It would be good to automate much of this, both for ease of maintenance and to prevent mistakes. The artifacts from a release are:

• dask-gateway and dask-gateway-server packages on pypi and conda-forge
• daskgateway/dask-gateway and daskgateway/dask-gateway-server images on docker hub
• A new helm chart packaged to https://dask.org/dask-gateway-helm-repo/

Releasing the packages

• Update the versions for both dask-gateway and dask-gateway-server. These are currently stored in _version.py files in each package.

• Commit and tag the release

git commit -m "Release $VERSION"
git tag -a $VERSION -m "Release $VERSION"

• Build dask-gateway as an sdist and a bdist_wheel, and upload to pypi:

cd dask-gateway
python setup.py sdist
python setup.py bdist_wheel
twine upload dist/*

• Build dask-gateway-server as an sdist and a bdist_wheel, and upload to pypi. Since this package has a binary component, this is a bit more involved:

cd dask-gateway-server
python setup.py sdist
python build_tools/build_macos.py
python build_tools.build_manylinux.py
twine upload dist/*

• Update the dask-gateway-feedstock on conda-forge, and merge. Usually the conda-forge automation takes care of this for us.

Releasing the helm chart and docker images

• Build the dask-gateway and dask-gateway-server docker images and upload to docker hub. If it's been a bit between releases and miniconda has updated a lot, we may also need to build the dask-gateway-base image that's shared between those, updating miniconda to a newer release (that dockerfile lives here). Generally when releasing I also bump the dependencies in the provided docker images to the latest stable version. We advise users to build their own dask-gateway images (since those run user code), but the dask-gateway-server image should work for all users and should be thoroughly tested. There's almost certainly a better system than this.

docker build dask-gateway-server/ --tag daskgateway/dask-gateway-server:$VERSION
docker tag dask-gateway-server:$VERSION daskgateway/dask-gateway-server:latest
docker build dask-gateway/ --tag daskgateway/dask-gateway:$VERSION
docker tag dask-gateway:$VERSION daskgateway/dask-gateway:latest
docker push daskgateway/dask-gateway-server:$VERSION
docker push daskgateway/dask-gateway-server:latest
docker push daskgateway/dask-gateway:$VERSION
docker push daskgateway/dask-gateway:latest

• Update the helm chart with the new version info. We need to bump the version and apiVersion fields in the Chart.yaml, and update the image versions for the dask-gateway and dask-gateway-server images in the values.yaml file.

• Package the helm chart and push it to helm chart repo. I've been doing this manually, which isn't terrible, but has a few steps. Chartpress does this automatically, but I scripting our own process wouldn't be to hard either, and we don't need all of chartpress's functionality. The general process is:

# Package the helm chart
cd resources/helm
mkdir temp
helm package dask-gateway --destination temp

# Update the repo index
helm repo index temp --url https://dask.org/dask-gateway-helm-repo/ --merge /path/to/dask-gateway-helm-repo/index.yaml

# Copy over the new files
cp -r temp/ /path/to/dask-gateway-helm-repo

# Commit and push daks-gateway-helm-repo
git add -a
git commit -m "Release $VERSION"
git push origin gh-pages

[fanshi118]

Are there any tools you'd recommend for this? (i.e. bash, python, or makefile)

[consideRatio]

Python package related automation

I would suggest that building and pushing the packages are automated via GitHub workflow jobs like in this github Workflow if a tag is pushed, and that a tag should only be pushed if one has updated the version in _version.py manually first for both packages.

Docker image and and helm chart related automation

I suggest this is automated via a github workflow either by a mix of using scaffold and chartpress or solely by using chartpress. I'm very experienced in using chartpress but very inexperienced in using skaffold. I think though that its probably best to let a single GitHub Workflow with one or more coupled jobs (via needs) do image and chart build/push as they are quite related.

[consideRatio]

Closed by:

• ci, maint: build/publish to PyPI, build linux/mac and amd64/arm64 wheels #538
• ci: use chartpress to build/test/publish images and Helm chart #514