ci: Add nightly upload of wheels to Anaconda Cloud

[matthewfeickert]

Resolves #3011

• Add nightly schedule job for Packaging Tests workflow at 01:23 UTC to generate wheels.
• Add nightly wheels workflow that uses the gh command line tool to search and find the artifacts that were uploaded, downlod them, and then upload them to the scikit-hep-nightly-wheels Anaconda Cloud org using the scientific-python/upload-nightly-action GitHub Action.

I'm putting this in draft mode so that I can clean it up before this gets approved and merged.

[jpivarski]

I'm definitely in favor. This is a great thing to do; thanks!

Don't worry about the pypy test. It's allowed to fail as a non-required test to remind us about #2971.

[matthewfeickert]

I'm definitely in favor. This is a great thing to do; thanks!

@jpivarski Great! I think beyond a review here, what would be good to discuss is also updates to the other workflows that are doing the wheel building to make sure that all desired wheels are built each night to get uploaded. At the moment #3011 (comment) makes it seems like only a subset are being built in the test workflows.

Also the

If this sounds good, I'll PR what I have and I'll need at least 1 (perferably 2) maintainers to create Anaconda Cloud (https://anaconda.org/) accounts and make new API tokens with permissions described in https://scientific-python.org/specs/spec-0004/#process-for-adding-new-projects

from #3011 (comment) is needed before this can go in.

[jpivarski]

Is this only for compiled wheels or for pure Python packages as well? (I'm thinking about Uproot.) Pure Python packages are less in need of pre-built wheels because pip+git would work.

[matthewfeickert]

Is this only for compiled wheels or for pure Python packages as well? (I'm thinking about Uproot.) Pure Python packages are less in need of pre-built wheels because pip+git would work.

Yeah, good point. It is totally up to us, as this is a package index that Scikit-HEP controls, so for single point of contact simplicity we could add pure-Python wheels, but for pyhf's HEAD of dependencies testing workflow simply doing

...
python -m pip uninstall --yes uproot
python -m pip install --upgrade git+https://github.com/scikit-hep/uproot5.git

has worked fine. So I'd defer to you and the rest of the uproot team on this.

[codecov-commenter]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (b749e49) 81.90% compared to head (cec028e) 81.91%.
Report is 12 commits behind head on main.

Additional details and impacted files

Files	Coverage Δ
src/awkward/_connect/cuda/__init__.py	0.00% <ø> (ø)
src/awkward/operations/ak_enforce_type.py	81.97% <ø> (ø)
src/awkward/operations/ak_to_dataframe.py	90.76% <100.00%> (+0.14%)	⬆️
src/awkward/_nplikes/typetracer.py	74.85% <0.00%> (ø)

[matthewfeickert]

@jpivarski Okay, I cleaned up a few things in the PR and gave it nicer commit messages for the history. Once you have done the following:

• Created an account on Anaconda Cloud (oh, is this you: https://anaconda.org/jpivarski ?)
• Told me the name of your account
• Made a new Anaconda Cloud API token with permissions described in https://scientific-python.org/specs/spec-0004/#process-for-adding-new-projects
• Saved the API token as a repo GitHub Actions secret named ANACONDA_ORG_UPLOAD_TOKEN
• Verified that I have added you to the Awkward team account on https://anaconda.org/scikit-hep-nightly-wheels

This can get merged in and should start working.

https://github.com/scikit-hep/awkward/blob/dd4753b97949e4098aa6008a189624ba7a72fda6/.github/workflows/packaging-test.yml only builds a handful of wheels, and so isn't quite providing true nightly wheels for all targets yet, but this is a solid first step and I think the Awkward team can exapnd on this to meet whatever goals they want from here.

[agoose77]

Right now we don't publish any wheels built on main, unless I am misremembering something! We do build PR wheels, which you rightly spot in packaging-test.yml.

Given that we test quite a lot of platform-specific stuff in our main PR test suite, I'd be tempted to change packaging-test.yml to test only main via a CRON job (or maybe on PUSH). @jpivarski thoughts?

[matthewfeickert]

@agoose77 here's updated logic. @jpivarski These revisions also look good to you?

[matthewfeickert]

@jpivarski @agoose77 Reverted things, but added runs on push events on main until you can tell me if you want this or just the schedule events added here. At the moment, the logic for downloading is only checking for schedule events, but it could look for schedule or push and just get whatever is the latest.

[jpivarski]

I'm still in favor of this; it looks like a good thing to do. I wonder what motivates the odd time, though. It's late enough to be evening, even in the U.S. (We'll soon have a new collaborator in Mountain time, but it's still okay for that—not likely to take away task-runners when we want feedback on a PR.)

[matthewfeickert]

I wonder what motivates the odd time, though

Ah that is easy to explain. Running jobs at 0:00 UTC gets slowed way down because many people are using that as a default and so GitHub Actions gets swamped with runner usage right then. Running at 1:23 UTC then gives you a little offset (and the numbers are just silly placeholders) and then running the uploader at 2:34 UTC means that you give it a huge amount of time to finish in the event that anything ever goes wrong and things get delayed by many minutes.

[matthewfeickert]

To make testing of this easier in advance of merge, I just triggered a workflow dispatch run of .github/workflows/packaging-test.yml on Awkward which finished and then I pushed a change that looks for both schedule and workflow_dispatch event triggers. So this shell script

#!/bin/bash

rm -rf awkward-cpp-wheels-* awkward-wheel dist runs.json

PROJECT_REPO="scikit-hep/awkward"
BRANCH="main"
WORKFLOW_NAME="packaging-test.yml"
ARTIFACT_PATTERN="awkward*wheel*"  # awkward-wheel and awkward-cpp-wheels-*

gh run --repo "${PROJECT_REPO}" \
   list --branch "${BRANCH}" \
        --workflow "${WORKFLOW_NAME}" \
        --json event,status,conclusion,databaseId > runs.json
RUN_ID=$(
  jq --compact-output \
    '[
      .[] |
      # Filter on "schedule" and "workflow_dispatch" events to main (nightly build) ...
      select(.event == "schedule" or .event == "workflow_dispatch") |
      # that have completed successfully ...
      select(.status == "completed" and .conclusion == "success")
     ] |
    # and get ID of latest build of wheels.
    sort_by(.databaseId) | reverse | .[0].databaseId' runs.json
)
gh run --repo "${PROJECT_REPO}" view "${RUN_ID}"
gh run --repo "${PROJECT_REPO}" \
   download "${RUN_ID}" --pattern "${ARTIFACT_PATTERN}"

mkdir dist
mv ${ARTIFACT_PATTERN}/*.whl dist/
ls -l dist/

is the logic I implemented in the PR. If you run it

$ bash debug.sh 

✓ main Packaging Tests · 7836312788
Triggered via workflow_dispatch about 16 minutes ago

JOBS
✓ Build awkward-cpp: windows-latest in 6m41s (ID 21383661203)
✓ Build awkward in 45s (ID 21383661437)
✓ Build awkward-cpp: macos-latest in 8m33s (ID 21383661690)
✓ Build awkward-cpp: ubuntu-latest in 7m1s (ID 21383661957)

ARTIFACTS
awkward-sdist
awkward-wheel
awkward-cpp-sdist
awkward-cpp-wheels-windows-latest
awkward-cpp-wheels-ubuntu-latest
awkward-cpp-wheels-macos-latest

For more information about a job, try: gh run view --job=<job-id>
View this run on GitHub: https://github.com/scikit-hep/awkward/actions/runs/7836312788
total 3844
-rw-r--r-- 1 feickert feickert  761870 Feb  8 15:57 awkward-2.6.1-py3-none-any.whl
-rw-r--r-- 1 feickert feickert  706425 Feb  8 15:57 awkward_cpp-29-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 feickert feickert  707671 Feb  8 15:57 awkward_cpp-29-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 feickert feickert 1214783 Feb  8 15:57 awkward_cpp-29-cp38-cp38-macosx_10_9_universal2.whl
-rw-r--r-- 1 feickert feickert  533474 Feb  8 15:57 awkward_cpp-29-cp39-cp39-win_amd64.whl

it gets the wheels that were just built. 👍 That's the whole wheel gathering logic that I added and then the rest of the PR is just the uploader action.

[matthewfeickert]

So as the wheels all have the same name each time (they don't have vcs information associated with them) they currently need some manual rerunning to deal with anaconda/anaconda-client#702, but after that, the workflow runs fine.