Stubs upload actions #4861
Stubs upload actions #4861
Conversation
|
Following today's stream I've been thinking more about this and researching PEP440 and think the following changes should be made. diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 4a478c381..47813766b 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -63,14 +63,6 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install setuptools wheel twine
- - name: Publish stubs
- if: contains(steps.need-pypi.outputs.setup-py, 'setup.py')
- env:
- TWINE_USERNAME: ${{ secrets.pypi_username }}
- TWINE_PASSWORD: ${{ secrets.pypi_password }}
- run: |
- python setup.py sdist
- twine upload dist/*
- name: Test Documentation Build (HTML)
run: sphinx-build -E -W -b html -D version=${{ env.CP_VERSION }} -D release=${{ env.CP_VERSION }} . _build/html
- uses: actions/upload-artifact@v2
@@ -126,6 +118,13 @@ jobs:
name: mpy-cross.static-x64-windows
path: mpy-cross/mpy-cross.static.exe
- name: Upload stubs and mpy-cross builds to S3
+ if: github.event_name == 'push' || (github.event_name == 'release' && (github.event.action == 'published' || github.event.action == 'rerequested'))
+ env:
+ AWS_PAGER: ''
+ AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+ AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+ TWINE_USERNAME: ${{ secrets.pypi_username }}
+ TWINE_PASSWORD: ${{ secrets.pypi_password }}
run: |
[ -z "$AWS_ACCESS_KEY_ID" ] || aws s3 cp mpy-cross/mpy-cross.static-aarch64 s3://adafruit-circuit-python/bin/mpy-cross/mpy-cross.static-aarch64-${{ e
nv.CP_VERSION }} --no-progress --region us-east-1
[ -z "$AWS_ACCESS_KEY_ID" ] || aws s3 cp mpy-cross/mpy-cross.static-raspbian s3://adafruit-circuit-python/bin/mpy-cross/mpy-cross.static-raspbian-${{ env.CP_VERSION }} --no-progress --region us-east-1
@@ -133,11 +132,10 @@ jobs:
[ -z "$AWS_ACCESS_KEY_ID" ] || aws s3 cp mpy-cross/mpy-cross.static.exe s3://adafruit-circuit-python/bin/mpy-cross/mpy-cross.static-x64-windows-${{ env.CP_VERSION }}.exe --no-progress --region us-east-1
zip -9r circuitpython-stubs.zip circuitpython-stubs
[ -z "$AWS_ACCESS_KEY_ID" ] || aws s3 cp circuitpython-stubs.zip s3://adafruit-circuit-python/bin/stubs/circuitpython-stubs-${{ env.CP_VERSION }}.zip --no-progress --region us-east-1
- env:
- AWS_PAGER: ''
- AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
- AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- if: github.event_name == 'push' || (github.event_name == 'release' && (github.event.action == 'published' || github.event.action == 'rerequested'))
+ if [[ -z "${{steps.need-pypi.outputs.setup-py}}" && -z "$TWINE_PASSWORD" ]]; then
+ python setup.py sdist
+ twine upload dist/*
+ fi
mpy-cross-mac:
diff --git a/setup.py b/setup.py
index 8c1f27e40..2c898a88b 100644
--- a/setup.py
+++ b/setup.py
@@ -20,8 +20,7 @@ version = git_out.strip().decode("utf-8")
# Detect a development build and mutate it to be valid semver and valid python version.
pieces = version.split("-")
if len(pieces) > 2:
- # Merge the commit portion onto the commit count since the tag.
- pieces[-2] += "+" + pieces[-1]
+ # Dump the github hash part
pieces.pop()
# Merge the commit count and build to the pre-release identifier.
pieces[-2] += ".dev." + pieces[-1]
The changes to The change in |
|
I do think it would run on every push currently. I'm still very green with github actions, I'm not certain how it specifies what the trigger should be. I think most libraries have a I'm also not sure about how to separate out the stable version from the pre-release one if they execute the |
It pushes to PyPI every time there is an upload to S3, so I believe every time a PR gets merged. |
.github/workflows/build.yml
Outdated
| AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| if: github.event_name == 'push' || (github.event_name == 'release' && (github.event.action == 'published' || github.event.action == 'rerequested')) | ||
|
|
||
| if [[ -z "${{steps.need-pypi.outputs.setup-py}}" && -z "$TWINE_PASSWORD" ]]; then |
There was a problem hiding this comment.
I am thinking this if statement logic might need to be tweaked. it might also be good to add an else clause that prints a message about being skipped if it does evaluate to false.
I tried merging this over to foamyguy/main (with my own PyPi project name) but as far as I can tell from the actions output seems like it did not do the upload. My best guess is that this came out false and caused it to skip. tonight or tomorrow I will try to tinker some more to add some debugging output to try to make it easier to tell what is going on.
Here are the actions output: https://github.com/FoamyGuy/circuitpython/runs/2769102086?check_suite_focus=true
Note that currently local branches commits (pushes) pass the
I'm new to that too, but I believe that at the very start of the file this makes all jobs trigger on all pushes and PRs (and releases). (Incidentally it also causes some builds to be doubled, but that doesn't impact the upload). While the |
I think you'll want to split the S3 upload from the pypi upload and have a different |
…load steps in actions.
# Conflicts: # setup.py
|
The latest commits separate out the pypi upload the mpy-cross s3 upload and use setuptools_scm to generate the version string which is the same utility used by circuit python libraries. I have successfully tested both dev and stable uploads using this branch: https://github.com/FoamyGuy/circuitpython/tree/foamyguy_test_stubs which had a modified package name that included my username. I think once pypi credentials are added to this repo these actions should begin working to upload the stubs to PyPi. |
I think they are set at the organizational level so they'll already apply. |
|
looks like we got 403 from PyPi: Maybe it was created with a different account? I noticed on PyPi the current On a different circuitpython library PyPi listing it shows author as Adafruit |
|
A side note, I'm a bit surprised that the stubs are nearly 9MB. Do they have extra content they shouldn't have? Are they more than just some ".pyi" files? |
| STD_PACKAGES = set(('array', 'math', 'os', 'random', 'struct', 'sys', 'ssl', 'time')) | ||
|
|
||
| STD_PACKAGES = set(('array', 'math', 'os', 'random', 'struct', 'sys', 'ssl', 'time')) |
There was a problem hiding this comment.
Ah yep I think this got doubled up, it's likely I misclicked it while working through changes to merge from main. I'll make a new PR to remove the dupe
|
When it runs it does print a bunch of other unrelated files, many things that are in the circuitpython repo, but not stubs specifically. If you unfold the stubs step here: https://github.com/adafruit/circuitpython/runs/2879766210?check_suite_focus=true it seems to take a second to load. For me it's a giant blank section for a while but eventually gets filled in if I scroll around a bit. It does list many files that are unrelated to stubs: As far as I could tell these did not seem to be getting included when the stubs are installed via pip though. Looking in the site-packages directory of the virtaul env I've been testing with I see only relavent dirs and pyi files, none of these extras that get printed during the action run. Perhaps they are in there somewhere but in a less obvious location. |
This adds some steps in the actions workflows to upload the generated stubs to pypi.
It seems to fail for something about an invalid version number when it runs on my fork. But I was able to successfully test it by changing the version string to be hard coded instead of generated dynamically inside of
setup.pyI'm hopeful this issues is related to being a separate fork on the repo and won't be a problem on this real main branch.The test pypi listing for the successful test can be found here for now:
This is the error message from my fork when it does fail using the generated version string: