create release artifacts of this repo

[mattip]

This repo is a little different that the other MacPython wheel-producing ones. While the wheels may find a home on a PyPI-like index, the tarballs here (and perhaps the mingw ones?) I think will need a different solution. One relatively simple thing would be to create releases and host them here. Thoughts?

[tylerjereddy]

I don't know much about GitHub releases so not sure I can comment on the drawbacks. Do we have a free extension with rackspace for a while now?

[mattip]

Someone is doing the releases on scipy/scipy. Those artifacts are available via a URL like https://github.com/scipy/scipy/releases/download/v1.2.3/scipy-1.2.3-cp27-cp27mu-manylinux1_x86_64.whl which is pretty easy to parse

[tylerjereddy]

Oh right, that's me. I thought you meant the new Workflows stuff

[ogrisel]

Here are the docs to deploy release artifacts to github:

• for travis: https://docs.travis-ci.com/user/deployment/releases/
• for appveyor: https://www.appveyor.com/docs/deployment/github/

[ogrisel]

For information, I discovered that it's possible to upload arbitrary files (that are neither wheels nor conda packages) to anaconda.org.

I started changing the CI scripts to upload anaconda artifacts for openblas-libs to anaconda.org (in the shared multibuild-wheels-staging organization):

https://github.com/ogrisel/openblas-libs/tree/anaconda-upload

I am waiting for appveyor to complete but if it works I will open a PR.

@matthew-brett you might be interested in the above as well.

[tylerjereddy]

Sounds interesting. I've been working on the anaconda.org transition for scipy-wheels initially: MacPython/scipy-wheels#68

What happens if/when we hit the 3GB free limit? For the forced replacements of wheels that may be less likely, but with a progressive accumulation of artifacts maybe it could happen, not sure.

[ogrisel]

For nightly builds we should not hit the 3GB limit if we delete older wheels on a regular basis (we can do a cron job somewhere to do this).

For release staging, e.g. project can create its own organization on anaconda to get its own 2GB limit. The shared staging area is just a convenience for starting but it does not really need to be shared.

Also I was I contact with people at anaconda and they will probably raise our storage quota if we ask them as long as the bandwidth stays similar to way we had on rackspace.

[ogrisel]

About release staging: nowadays it's possible to use a single CI config for the 3 main platforms (e.g. travis and azure pipelines can do Windows, macOS and Linux). On Azure Pipelines in particular it's possible to copy release artifacts to a shared folder visible to all the workers of the same build and have a dag structured dependency to collect all the wheels together and maybe publish them all at once to pypi.org if all the tests for all the platforms passed. See we would actually not need anaconda.org for release staging.

Also on Azure and Travis it's possible to use regular bash on windows for all the steps which makes it much simpler to share the same CI scripts for the 3 platforms.

[mattip]

Your branch seems quite straightforward. I do wonder if releases here would not serve our purpose better though. What would the URL look like to download the openblas-lib for use in other projects?

[ogrisel]

I updated the travis secret, added more comments on the token configuration and open #17.

The release would work as well. Here is a simulated URL:

https://github.com/MacPython/openblas-libs/releases/download/v0.3.5-605-gc815b8fb/openblas64_-v0.3.5-605-gc815b8fb-manylinux1_x86_64.tar.gz

On anaconda it looks like (actual URL):

https://anaconda.org/multibuild-wheels-staging/openblas/v0.3.5-605-gc815b8fb/download/openblas64_-v0.3.5-605-gc815b8fb-manylinux1_x86_64.tar.gz

[charris]

We (NumPy) want to control the wheel uploads to PyPI because we generate hashes for the released wheels.

[ogrisel]

We (NumPy) want to control the wheel uploads to PyPI because we generate hashes for the released wheels.

I am curious about more details about your release process. Are the hashes you mentions the same as those computed by PyPI?

https://pypi.org/project/numpy/#copy-hash-modal-f160f956-126a-4ccc-981d-c19883a3ae48

Do you keep an independent copy of the hash you generate before uploading to pypi to check that the binaries have not been corrupted after the release?

If your remark is about my suggested single-CI workflow described above It should be possible to compute those hashes in the final stage of the Azure pipeline configuration and those hashes could be published to an independent append-only log (e.g. a dedicated twitter feed) and it would also be possible to have a scheduled job that would periodically download past released wheels, recompute their hashes and compare those to the values published on the append-only log.

[charris]

We don't use the PyPI hashes, but rather publish our own in the release notes on Github. That said, I'm not convinced the hashes are actually useful. @rgommers Thoughts?

[tylerjereddy]

I guess the idea is the release manager uses their own secure signing mechanism with a published public portion of their key, or something like that. Wouldn't be surprised if it could be streamlined with some careful thought I suppose. The README gets updated with the newly-downloaded file hashes after staging, but in theory I suppose you might be able to automate that.