Add workflow / recipe to generate Dask/distributed pre-releases by charlesbluca · Pull Request #5636 · dask/distributed

charlesbluca · 2022-01-04T18:33:33Z

As part of dask/community#76 and following up on dask/dask#8469, this PR adds:

Conda recipes in continuous_integration to build distributed and dask pre-release packages from the Git repo
A GHA workflow to build the pre-release packages as a check for PRs, and to upload these packages to the Dask channel under the dev label for pushes to main
Closes #xxxx
Tests added / passed
Passes pre-commit run --all-files

jakirkham

Looks good Charles! 😄

Had a few comments below.

Also have we tested this locally?

jakirkham · 2022-01-11T19:21:00Z

+          export DASK_VERSION=`conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | awk '{print $2}'`
+          export DASK_BUILD=`conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | awk '{print $3}'`


Wonder if we can use bash arrays to compute this once and extract each piece from that result

Perhaps something like this would work

Suggested change

export DASK_VERSION=`conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | awk '{print $2}'`

export DASK_BUILD=`conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | awk '{print $3}'`

export DASK_INFO=( $(conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | tr " " "\n") )

export DASK_VERSION="${DASK_INFO[1]}"

export DASK_BUILD="${DASK_INFO[2]}"

Thanks @jakirkham 😄 was playing around with this myself, think we can get away with removing the tr command

Was thinking we should be able to avoid tr, but wasn't seeing how to do that on my Mac. Maybe Linux is better behaved 😄

jakirkham · 2022-01-11T19:33:35Z

+          && github.ref == 'refs/heads/main'
+          && github.repository == 'dask/distributed'
+        env:
+          ANACONDA_API_TOKEN: ${{ secrets.DASK_CONDA_TOKEN }}


Just a note, we already set this up in issue ( dask/dask#8504 ). So it should now just work here

Co-authored-by: jakirkham <jakirkham@gmail.com>

charlesbluca

I did test locally to ensure the packages built properly, will run another build and report the resulting packages (including the results of conda convert); that actually reminded me of a relevant question:

charlesbluca · 2022-01-12T15:01:22Z

+          cd build && conda convert linux-64/*.tar.bz2 -p osx-64 \
+                                                       -p osx-arm64 \
+                                                       -p linux-ppc64le \
+                                                       -p linux-aarch64 \
+                                                       -p win-64


Should we include this in the testing that runs on PRs?

…uted into distributed-nightly

charlesbluca · 2022-01-12T15:25:34Z

Ran the commands locally, we end up getting the following packages in build:

$ find -type f -name "*.tar.bz2"
./win-64/distributed-2021.12.0a220112-py310_g3abe94e4_28.tar.bz2
./linux-ppc64le/distributed-2021.12.0a220112-py310_g3abe94e4_28.tar.bz2
./osx-64/distributed-2021.12.0a220112-py310_g3abe94e4_28.tar.bz2
./noarch/dask-2021.12.0a220112-py310_g3abe94e4_28.tar.bz2
./osx-arm64/distributed-2021.12.0a220112-py310_g3abe94e4_28.tar.bz2
./linux-aarch64/distributed-2021.12.0a220112-py310_g3abe94e4_28.tar.bz2
./linux-64/distributed-2021.12.0a220112-py310_g3abe94e4_28.tar.bz2

Anything we're missing?

EDIT:

I notice that we're not building Distributed for 3.7-3.9 - this is unexpected to me, as I would think that specifying architecture in the build would be enough to build packages for these other versions; is there something else we need to specify here?

EDIT 2:

Completely forgot that we need a build config to control the python versions we build 😅

charlesbluca · 2022-01-12T16:22:54Z

Looks like we should be building all the required packages now:

$ find -type f -name "*.tar.bz2"
./win-64/distributed-2021.12.0a220112-py310_g1d4bd481_31.tar.bz2
./win-64/distributed-2021.12.0a220112-py39_g1d4bd481_31.tar.bz2
./win-64/distributed-2021.12.0a220112-py37_g1d4bd481_31.tar.bz2
./win-64/distributed-2021.12.0a220112-py38_g1d4bd481_31.tar.bz2
./linux-ppc64le/distributed-2021.12.0a220112-py310_g1d4bd481_31.tar.bz2
./linux-ppc64le/distributed-2021.12.0a220112-py39_g1d4bd481_31.tar.bz2
./linux-ppc64le/distributed-2021.12.0a220112-py37_g1d4bd481_31.tar.bz2
./linux-ppc64le/distributed-2021.12.0a220112-py38_g1d4bd481_31.tar.bz2
./osx-64/distributed-2021.12.0a220112-py310_g1d4bd481_31.tar.bz2
./osx-64/distributed-2021.12.0a220112-py39_g1d4bd481_31.tar.bz2
./osx-64/distributed-2021.12.0a220112-py37_g1d4bd481_31.tar.bz2
./osx-64/distributed-2021.12.0a220112-py38_g1d4bd481_31.tar.bz2
./noarch/dask-2021.12.0a220112-py310_g1d4bd481_31.tar.bz2
./osx-arm64/distributed-2021.12.0a220112-py310_g1d4bd481_31.tar.bz2
./osx-arm64/distributed-2021.12.0a220112-py39_g1d4bd481_31.tar.bz2
./osx-arm64/distributed-2021.12.0a220112-py37_g1d4bd481_31.tar.bz2
./osx-arm64/distributed-2021.12.0a220112-py38_g1d4bd481_31.tar.bz2
./linux-aarch64/distributed-2021.12.0a220112-py310_g1d4bd481_31.tar.bz2
./linux-aarch64/distributed-2021.12.0a220112-py39_g1d4bd481_31.tar.bz2
./linux-aarch64/distributed-2021.12.0a220112-py37_g1d4bd481_31.tar.bz2
./linux-aarch64/distributed-2021.12.0a220112-py38_g1d4bd481_31.tar.bz2
./linux-64/distributed-2021.12.0a220112-py310_g1d4bd481_31.tar.bz2
./linux-64/distributed-2021.12.0a220112-py39_g1d4bd481_31.tar.bz2
./linux-64/distributed-2021.12.0a220112-py37_g1d4bd481_31.tar.bz2
./linux-64/distributed-2021.12.0a220112-py38_g1d4bd481_31.tar.bz2

jakirkham · 2022-01-14T16:29:20Z

@jrbourbeau would you be able to take a look at this? JFYI not needed before the release

jakirkham · 2022-01-28T01:23:09Z

Friendly nudge @jrbourbeau 🙂

jakirkham · 2022-01-31T18:38:41Z

@dask/maintenance could someone please review? 🙂

jsignell

This looks good to me.

jakirkham · 2022-01-31T20:39:22Z

Thanks Julia! 😄

charlesbluca · 2022-01-31T20:41:49Z

Looks like the upload itself is failing:

https://github.com/dask/distributed/runs/5012056353?check_suite_focus=true

Is secrets.DASK_CONDA_TOKEN available to this repo?

EDIT:

Just checked and it looks like it isn't - maybe we should opt to make DASK_CONDA_TOKEN an organization level secret instead of repo-specific (think only the Dask repo has access to it)?

jakirkham · 2022-01-31T20:52:17Z

We set it at the org level. It is here

Edit: Oh though I do note that despite being set at the org level it is only visible to one repo (dask). Have just updated it to include distributed. Let's try again. It is running here

jakirkham · 2022-01-31T21:02:04Z

Looks like we have another issue uploading

Using "dask" as upload username
Error:  File "**.tar.bz2" does not exist
Error:  File "**.tar.bz2" does not exist
Error: Process completed with exit code 1.

charlesbluca · 2022-01-31T21:09:18Z

Yeah it looks like anaconda upload doesn't allow for double asterisk wildcards - a simple (though somewhat annoying to maintain) solution to fix this now is to just upload each package directory in its command, such as:

anaconda upload linux-64/*.tar.bz2
anaconda upload windows-64/*.tar.bz2
...

I'll do this now and play around with uploading locally to see if there's a better solution

jakirkham · 2022-01-31T22:10:27Z

JFYI this was done in PR ( #5741 ) and appears to be working. Thanks Charles for the hard work here! 😄

…#5636) As part of dask/community#76 and following up on dask/dask#8469, this PR adds: - Conda recipes in `continuous_integration` to build `distributed` and `dask` pre-release packages from the Git repo - A GHA workflow to build the pre-release packages as a check for PRs, and to upload these packages to the Dask channel under the `dev` label for pushes to `main`

First pass at distributed nightly builds

89705f1

charlesbluca commented Jan 4, 2022

View reviewed changes

Comment thread continuous_integration/recipe/meta.yaml Outdated

charlesbluca added 5 commits January 10, 2022 11:25

Pin dask-core dep to latest nightly build

f3d86a0

Add dask metapackage recipe

1ade91a

Try to pin distributed dep by build

b479c7f

Pin dask dep by version and build

55f9109

Minor changes to anaconda upload step

3abe94e

jakirkham reviewed Jan 11, 2022

View reviewed changes

Update doc source URL

9dafda2

Co-authored-by: jakirkham <jakirkham@gmail.com>

charlesbluca commented Jan 12, 2022

View reviewed changes

charlesbluca added 2 commits January 12, 2022 10:23

Only run conda search command once

0ab4191

Merge branch 'distributed-nightly' of github.com:charlesbluca/distrib…

1d4bd48

…uted into distributed-nightly

Add conda build config

75af34e

charlesbluca marked this pull request as ready for review January 12, 2022 16:36

Add newline to build config

004813c

jakirkham reviewed Jan 13, 2022

View reviewed changes

Comment thread continuous_integration/recipes/distributed/conda_build_config.yaml Outdated

jakirkham requested a review from jrbourbeau January 14, 2022 01:18

Don't build Python 3.10 nightlies

4d01682

jsignell approved these changes Jan 31, 2022

View reviewed changes

jsignell merged commit 788005f into dask:main Jan 31, 2022

charlesbluca mentioned this pull request Feb 1, 2022

Publishing nightly packages dask/community#76

Closed

charlesbluca deleted the distributed-nightly branch July 20, 2022 03:00

		export DASK_VERSION=`conda search --override-channels -c dask/label/dev dask-core \| tail -n 1 \| awk '{print $2}'`
		export DASK_BUILD=`conda search --override-channels -c dask/label/dev dask-core \| tail -n 1 \| awk '{print $3}'`

-          export DASK_VERSION=`conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | awk '{print $2}'`
-          export DASK_BUILD=`conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | awk '{print $3}'`
+          export DASK_INFO=( $(conda search --override-channels -c dask/label/dev dask-core | tail -n 1 | tr " " "\n") )
+          export DASK_VERSION="${DASK_INFO[1]}"
+          export DASK_BUILD="${DASK_INFO[2]}"

Uh oh!

Conversation

charlesbluca commented Jan 4, 2022

Uh oh!

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkham Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

jakirkham Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charlesbluca Jan 12, 2022

Choose a reason for hiding this comment

Uh oh!

jakirkham Jan 12, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakirkham Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

charlesbluca left a comment

Choose a reason for hiding this comment

Uh oh!

charlesbluca Jan 12, 2022

Choose a reason for hiding this comment

Uh oh!

charlesbluca commented Jan 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charlesbluca commented Jan 12, 2022

Uh oh!

Uh oh!

jakirkham commented Jan 14, 2022

Uh oh!

jakirkham commented Jan 28, 2022

Uh oh!

jakirkham commented Jan 31, 2022

Uh oh!

jsignell left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkham commented Jan 31, 2022

Uh oh!

charlesbluca commented Jan 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham commented Jan 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham commented Jan 31, 2022

Uh oh!

charlesbluca commented Jan 31, 2022

Uh oh!

jakirkham commented Jan 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jakirkham Jan 11, 2022 •

edited

Loading

charlesbluca commented Jan 12, 2022 •

edited

Loading

charlesbluca commented Jan 31, 2022 •

edited

Loading

jakirkham commented Jan 31, 2022 •

edited

Loading