Support for python 3.9

[pipermerriam]

See my comment here: ethereum/web3.py#1774 (comment)

It looks like we might need a new build published to support python 3.9? Not 100% sure this is the problem but at first glance... 🤷

[cburgdorf]

@davesque if you could give @g-r-a-n-t and me access to the library (including pypi) we could handle this as well as future maintenance work.

[MatthiasLohr]

Would be nice if anyone would take over work here. I'm also willing to support here.

I'm using py-evm, which depends on this library, and it somehow drives me crazy that I need to have complete rust toolchain installed to get my dependencies up and running. I'm also willing to invest more time here.

[davesque]

@cburgdorf @g-r-a-n-t Let me know your pypi usernames and I'll add you as collaborators on the pypi package.

[cburgdorf]

Thanks, that would be great. I'm cburgdorf on pypi

[g-r-a-n-t]

I'm g-r-a-n-t

[davesque]

@cburgdorf Added you.

@g-r-a-n-t I'm getting this error when trying to add you on pypi:

[g-r-a-n-t]

Ah, sorry about this @davesque. I've just verified my account and it should be good now.

[cburgdorf]

I looked into this today. If I understand the release process correctly, it goes roughly like this:

1. Create Linux and Mac wheels using Circle CI
2. Create Win wheels via Appveyor
3. Collect the wheels via custom collectwheels.py script

This requires obtaining authentication token for Github, CircleCI and Appveyor.

It seems the project also uses Traves CI but I'm not sure if that is involved in creating the wheels.

I have recently dealt with a similar issue (shipping a python library written in Rust to Linux, Mac and Win users) and came to enjoy Github Actions (which I believe wasn't available when this project was set up).

With GitHub Actions, shipping wheels for all major platforms can be setup in 50 loc and doesn't involve any manual work with authentication tokens...ok, except for the one needed to automatically push to pypi.

I propose that we migrate away from CircleCI, Travis and Appveyor and consolidate everything into one Github Actions configuration.

@davesque Would you be ok with that?

[davesque]

Yeah, go for it! Sounds fine with me.
@cburgdorf

[ytrezq]

Why not add a setup.py to the source version on PyPi so it can be built from pip?

[pipermerriam]

I believe it is because the actual implementation is in rust and so it needs special tooling to build, thus providing the pre-built wheels.

[ytrezq]

@pipermerriam this also applies to some C wrappers. In that case, why not do the same and add the required additional scripts along with the required tools in order to support Python versions installed in virtualenv along all the architectures supported by llvm?

[pipermerriam]

If someone would like to do this 👍

[ytrezq]

@pipermerriam this would be the real way forward in order to support future python distributions.

[MatthiasLohr]

Since there is still no commit in this repository... is there any official succeeding repository, where you do the work? Talked to the maturin guys, which now have support for ARM/Raspberry Pi (PyO3/maturin#364 (comment)), which would also be great to be added (see #1).

[cburgdorf]

@MatthiasLohr would you be up to take the task that I suggested here #2 (comment)

If we migrate that to Github Actions, adding support for new Python versions becomes trivial without having to rely on manually uploading binaries etc. Regarding ARM/Rasperry Pi, that would continue to be an issue because I don't think GitHub Actions does support it yet.

You seem to care strongest about this issue so you might be the best candidate to put in the work 😉

[MatthiasLohr]

Sure, why not. But this repository's last commit was in 2019 - so, where is the right place to push my code to?

Any maybe I'm wrong, but it seems that there is still no Python3.9 package for blake2b-py available on PyPI?

[MatthiasLohr]

In the meantime, I also compiled some of the packages by my own to get one of my projects running, (including 3.9/x64 and some ARM packages). Until everything is migrated to GitHub actions, would you mind to manually upload missing artifacts? This would really help me with the portability of my project, since right now I have to install blake2b-py manually using my self-prepared packages...

Currently, I'm preparing armv7l packages.

[cburgdorf]

Sure, why not. But this repository's last commit was in 2019 - so, where is the right place to push my code to?

This is still the right place for now.

Any maybe I'm wrong, but it seems that there is still no Python3.9 package for blake2b-py available on PyPI?

No, but this would become easy to add with the Github Actions setup

Until everything is migrated to GitHub actions, would you mind to manually upload missing artifacts?

I'm afraid, I can't. I don't have the capacity to validate these and uploading trusted binaries would be a big no-no.

I would recommend that you fork the project and publish your own binaries on pypi and setup your project to use those.

[MatthiasLohr]

Would it be ok if I provide you the commands I used for building the binaries? It's not much, so maybe you can use it for a manual build (assuming curl, python 3.9 and virtualenv is installed):

git clone https://github.com/davesque/blake2b-py.git && cd blake2b-py
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
rustup default nightly
virtualenv -p python3.9 venv
source ./venv/bin/activate
pip install maturin
maturin build -i python3.9

These all are official packages from official sources, so it should be possible to trust them. Target wheels are then located in target/wheels/. Btw, the exact same command sequence works for Raspberry Pi, too.

[MatthiasLohr]

I would recommend that you fork the project and publish your own binaries on pypi and setup your project to use those.

Sorry, but that sounds like a really ugly, bad and unreliable workaround. If would strongly suggest not to split efforts for the same goal into different projects...

[cburgdorf]

Sorry, but that sounds like a really ugly, bad and unreliable workaround. If would strongly suggest not to split efforts for the same goal into different projects...

I understand but I'm just trying to help you find a solution to your problem.

Let me try to explain my point of view. This is not my project. It's also not an official repository from the Ethereum Foundation. It is a privat repository from @davesque that he created a while ago but stopped maintaining. He added me as a maintainer because we know us IRL and he trusts me. I have never worked on or used this project directly.

I understand that Python 3.9 as well as Rasperry Pi support are good features but it seems that they are currently most valuable to you and hence I'm kindly asking you to put in the work or alternatively maintain your own fork (even if just temporary).

I don't have Python 3.9 on my system (in fact, I haven't written any Python in a while) and I don't want to mess with my system now to create these builds.

What I can promise to you is that if you open up a PR that adds a way to transparently create and update these binaries (preferable using Github Actions) than this will be reviewed, merged and I'll be happy to trigger builds and releases.

Unfortunately, that's as much as I can help you right now.

[MatthiasLohr]

Pull Request with (beginnings of) GitHub actions is there: #3. This is my very first contact with GitHub actions, so not sure about the process of replacing the currently active CI. At least with my forked repository, the action definitions seems to work: https://github.com/MatthiasLohr/blake2b-py/actions/runs/614582761

[fubuloubu]

It's also not an official repository from the Ethereum Foundation.

It is a downstream dependency of an EF-maintained library, and in my opinion should be in the interests of EF to maintain:
https://github.com/ethereum/py-evm/blob/db9ae3c1aa617e28525344db159db2a312a03033/setup.py#L8