Fix Builds (python_gnupg-0.5.2-py2.py3-none-any.whl.asc 404 not found)

[maltfield]

Describe the bug
Currently it's not possible to build the BusKill app due to gpg signing issues with the python-gnupg dependency

To Reproduce
I've tested this on Linux, but it should be an issue on all platforms

Steps to reproduce the behavior:

Execute the following on Debian 11

sudo su -
sudo apt-get install git
git clone https://github.com/BusKill/buskill-app.git
cd buskill-app/
build/linux/debianWrapper.sh

You'll get the following error

user@buskill:~/sandbox/buskill-app$ build/linux/debianWrapper.sh 
...
Successfully installed Kivy-1.11.1 Kivy-Garden-0.1.4 certifi-2020.6.20 chardet-3.0.4 docutils-0.16 idna-2.10 pygments-2.6.1 requests-2.24.0 urllib3-1.25.9
++ mktemp -d
+ tmpDir=/tmp/tmp.TxQFtmxBry
+ chown _apt:root /tmp/tmp.TxQFtmxBry
+ chmod 0770 /tmp/tmp.TxQFtmxBry
+ pushd /tmp/tmp.TxQFtmxBry
+ /bin/su _apt -s /bin/bash -c '/tmp/kivy_appdir/AppRun -m pip download python-gnupg'
/tmp/tmp.TxQFtmxBry ~/buskill-app
WARNING: The directory '/nonexistent/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting python-gnupg
  Downloading python_gnupg-0.5.2-py2.py3-none-any.whl (20 kB)
  Saved ./python_gnupg-0.5.2-py2.py3-none-any.whl
Successfully downloaded python-gnupg
WARNING: You are using pip version 20.1.1; however, version 24.0 is available.
You should consider upgrading via the '/tmp/kivy_appdir/opt/python3.7/bin/python3.7 -m pip install --upgrade pip' command.
++ head -n1
++ ls -1
+ filename=python_gnupg-0.5.2-py2.py3-none-any.whl
++ /bin/su _apt -s /bin/bash -c 'curl -s https://pypi.org/simple/python-gnupg/'
++ grep -oE 'https://.*python_gnupg-0.5.2-py2.py3-none-any.whl#'
++ sed s/#/.asc/
+ signature_url=https://files.pythonhosted.org/packages/5f/86/18938c2378a8c8284910308b93e2bc521a965ced4ca028947e1ffa6e98dc/python_gnupg-0.5.2-py2.py3-none-any.whl.asc
+ /bin/su _apt -s /bin/bash -c 'wget "https://files.pythonhosted.org/packages/5f/86/18938c2378a8c8284910308b93e2bc521a965ced4ca028947e1ffa6e98dc/python_gnupg-0.5.2-py2.py3-none-any.whl.asc"'
--2024-02-26 01:35:40--  https://files.pythonhosted.org/packages/5f/86/18938c2378a8c8284910308b93e2bc521a965ced4ca028947e1ffa6e98dc/python_gnupg-0.5.2-py2.py3-none-any.whl.asc
Resolving files.pythonhosted.org (files.pythonhosted.org)... 199.232.44.223, 2a04:4e42:48::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|199.232.44.223|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-02-26 01:35:42 ERROR 404: Not Found.

+ mkdir gnupg
+ chmod 0700 gnupg
+ popd
+ gpg --homedir /tmp/tmp.TxQFtmxBry/gnupg --import build/deps/python-gnupg.asc
~/buskill-app
gpg: keybox '/tmp/tmp.TxQFtmxBry/gnupg/pubring.kbx' created
gpg: /tmp/tmp.TxQFtmxBry/gnupg/trustdb.gpg: trustdb created
gpg: key 9147B477339A9B86: public key "Vinay Sajip (CODE SIGNING KEY) <vinay_sajip@yahoo.co.uk>" imported
gpg: Total number processed: 1
gpg:               imported: 1
+ gpgv --homedir /tmp/tmp.TxQFtmxBry/gnupg --keyring /tmp/tmp.TxQFtmxBry/gnupg/pubring.kbx /tmp/tmp.TxQFtmxBry/python_gnupg-0.5.2-py2.py3-none-any.whl.asc /tmp/tmp.TxQFtmxBry/python_gnupg-0.5.2-py2.py3-none-any.whl
gpgv: can't open '/tmp/tmp.TxQFtmxBry/python_gnupg-0.5.2-py2.py3-none-any.whl.asc': No such file or directory
gpgv: verify signatures failed: No such file or directory
ERROR: Invalid PGP signature!
+ [[ 2 -ne 0 ]]
+ echo 'ERROR: Invalid PGP signature!'
+ exit 1
+ chown -R user dist
+ exit 0
+ exit 0
user@buskill:~/sandbox/buskill-app$

Expected behavior
The build should be able to verify the authenticity of the python_gnupg package and proceed with building the BusKill app

[maltfield]

In the past, we've had build issues due to the verification of this same python_gnupg python module, but only on MacOS

1. Fix MacOS Builds (curl dyld Library not loaded for libunistring.2.dylib) #70
2. Fix MacOS Builds #72

In the above issues, the issue was a tooling issue with the availability of curl/wget on MacOS.

But in this case, it appears that the file actually has disappeared from the server. I cannot fetch it, for example, on my local machine

user@buskill:~$ wget https://files.pythonhosted.org/packages/5f/86/18938c2378a8c8284910308b93e2bc521a965ced4ca028947e1ffa6e98dc/python_gnupg-0.5.2-py2.py3-none-any.whl
--2024-02-25 22:08:59--  https://files.pythonhosted.org/packages/5f/86/18938c2378a8c8284910308b93e2bc521a965ced4ca028947e1ffa6e98dc/python_gnupg-0.5.2-py2.py3-none-any.whl
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.124.223, 2a04:4e42:1e::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.124.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20964 (20K) [binary/octet-stream]
Saving to: ‘python_gnupg-0.5.2-py2.py3-none-any.whl’

python_gnupg-0.5.2- 100%[===================>]  20.47K  --.-KB/s    in 0.02s   

2024-02-25 22:09:00 (868 KB/s) - ‘python_gnupg-0.5.2-py2.py3-none-any.whl’ saved [20964/20964]

user@buskill:~$ 

user@buskill:~$ wget https://files.pythonhosted.org/packages/5f/86/18938c2378a8c8284910308b93e2bc521a965ced4ca028947e1ffa6e98dc/python_gnupg-0.5.2-py2.py3-none-any.whl.asc
--2024-02-25 22:09:03--  https://files.pythonhosted.org/packages/5f/86/18938c2378a8c8284910308b93e2bc521a965ced4ca028947e1ffa6e98dc/python_gnupg-0.5.2-py2.py3-none-any.whl.asc
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.124.223, 2a04:4e42:1e::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.124.223|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-02-25 22:09:04 ERROR 404: Not Found.

user@buskill:~$ 

[maltfield]

I checked the package's page on the PyPI project's website. If there's a newer version, then it might make sense that the archived version somehow got orphaned from its signature? If so, it's a bug that we should report

• https://pypi.org/project/python-gnupg/#files

Anyway, it appears that the version hasn't changed since the one we pegged as a dependency. The latest version is still v0.5.2

[maltfield]

I was digging through my notes and found that my old link to hash checking in PyPI is stale

• https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode

The above link points to a new location which, after some digging, brings you here

• https://pip.pypa.io/en/stable/topics/secure-installs/

Woah, that's dangerous. The above link suggests that hash checking in pip somehow adds security to verify packages are not malicious when turned-on. This is dangerous misinformation. pip doesn't have any built-in mechanism to cryptographically verify the authenticity of packages (although they've been working on this since 2019).

The whole reason we have this awkward manual gpg check is because pip doesn't have a secure way to download software. Fortunately, twine does have PGP signing support, as documented here:

• https://twine.readthedocs.io/en/latest/#why-should-i-use-this

Anyway, I opened a ticket with PyPI to fix their documentation to move the hashing section outside of the Secure Install section and to add a big warning indicating to users that hashes does not provide any secure verification checks

• Documentation Error: Hashes are for integrity, not security pypa/pip#12541

[maltfield]

I opened an upstream bug about this here:

• Fix Release Signatures on PyPI (404 Not Found) vsajip/python-gnupg#234

[maltfield]

I switched to fetching the files from GitHub, but now I have an issue with verifying the signature of the libusb1 dependency

Successfully installed Kivy-1.11.1 Kivy-Garden-0.1.4 certifi-2020.6.20 chardet-3.0.4 docutils-0.16 idna-2.10 pygments-2.6.1 requests-2.24.0 urllib3-1.25.9
++ mktemp -d
+ tmpDir=/tmp/tmp.hTMYgQLAMP
+ chown _apt:root /tmp/tmp.hTMYgQLAMP
+ chmod 0770 /tmp/tmp.hTMYgQLAMP
+ pushd /tmp/tmp.hTMYgQLAMP
/tmp/tmp.hTMYgQLAMP ~/buskill-app
+ file_url=https://github.com/vsajip/python-gnupg/releases/download/0.5.2/python_gnupg-0.5.2-py2.py3-none-any.whl
+ signature_url=https://github.com/vsajip/python-gnupg/releases/download/0.5.2/python_gnupg-0.5.2-py2.py3-none-any.whl.asc
+ /bin/su _apt -s /bin/bash -c 'wget "https://github.com/vsajip/python-gnupg/releases/download/0.5.2/python_gnupg-0.5.2-py2.py3-none-any.whl"'
--2024-02-27 00:20:12--  https://github.com/vsajip/python-gnupg/releases/download/0.5.2/python_gnupg-0.5.2-py2.py3-none-any.whl
Resolving github.com (github.com)... 20.248.137.48
Connecting to github.com (github.com)|20.248.137.48|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/436952016/770eba9a-6d0e-489e-beb3-660efc8abc55?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240227%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240227T002015Z&X-Amz-Expires=300&X-Amz-Signature=d6d333e9827488296005b9563b5060663602719b4323be9e940fc2e4019f76de&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=436952016&response-content-disposition=attachment%3B%20filename%3Dpython_gnupg-0.5.2-py2.py3-none-any.whl&response-content-type=application%2Foctet-stream [following]
--2024-02-27 00:20:14--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/436952016/770eba9a-6d0e-489e-beb3-660efc8abc55?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240227%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240227T002015Z&X-Amz-Expires=300&X-Amz-Signature=d6d333e9827488296005b9563b5060663602719b4323be9e940fc2e4019f76de&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=436952016&response-content-disposition=attachment%3B%20filename%3Dpython_gnupg-0.5.2-py2.py3-none-any.whl&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20964 (20K) [application/octet-stream]
Saving to: 'python_gnupg-0.5.2-py2.py3-none-any.whl'

     0K .......... ..........                                 100% 2.51M=0.008s

2024-02-27 00:20:17 (2.51 MB/s) - 'python_gnupg-0.5.2-py2.py3-none-any.whl' saved [20964/20964]

++ ls -1
++ head -n1
+ filename=python_gnupg-0.5.2-py2.py3-none-any.whl
+ /bin/su _apt -s /bin/bash -c 'wget "https://github.com/vsajip/python-gnupg/releases/download/0.5.2/python_gnupg-0.5.2-py2.py3-none-any.whl.asc"'
--2024-02-27 00:20:17--  https://github.com/vsajip/python-gnupg/releases/download/0.5.2/python_gnupg-0.5.2-py2.py3-none-any.whl.asc
Resolving github.com (github.com)... 20.248.137.48
Connecting to github.com (github.com)|20.248.137.48|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/436952016/d1d4da38-0a74-42b8-aa0f-120f13479a30?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240227%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240227T002020Z&X-Amz-Expires=300&X-Amz-Signature=6b0dbdfe514fea43d2a4db573cf3ba5b44c16abcf510ed005368e91f22427422&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=436952016&response-content-disposition=attachment%3B%20filename%3Dpython_gnupg-0.5.2-py2.py3-none-any.whl.asc&response-content-type=application%2Foctet-stream [following]
--2024-02-27 00:20:20--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/436952016/d1d4da38-0a74-42b8-aa0f-120f13479a30?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240227%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240227T002020Z&X-Amz-Expires=300&X-Amz-Signature=6b0dbdfe514fea43d2a4db573cf3ba5b44c16abcf510ed005368e91f22427422&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=436952016&response-content-disposition=attachment%3B%20filename%3Dpython_gnupg-0.5.2-py2.py3-none-any.whl.asc&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 801 [application/octet-stream]
Saving to: 'python_gnupg-0.5.2-py2.py3-none-any.whl.asc'

     0K                                                       100% 38.5M=0s

2024-02-27 00:20:22 (38.5 MB/s) - 'python_gnupg-0.5.2-py2.py3-none-any.whl.asc' saved [801/801]

+ mkdir gnupg
+ chmod 0700 gnupg
+ popd
+ gpg --homedir /tmp/tmp.hTMYgQLAMP/gnupg --import build/deps/python-gnupg.asc
~/buskill-app
gpg: keybox '/tmp/tmp.hTMYgQLAMP/gnupg/pubring.kbx' created
gpg: /tmp/tmp.hTMYgQLAMP/gnupg/trustdb.gpg: trustdb created
gpg: key 9147B477339A9B86: public key "Vinay Sajip (CODE SIGNING KEY) <vinay_sajip@yahoo.co.uk>" imported
gpg: Total number processed: 1
gpg:               imported: 1
+ gpgv --homedir /tmp/tmp.hTMYgQLAMP/gnupg --keyring /tmp/tmp.hTMYgQLAMP/gnupg/pubring.kbx /tmp/tmp.hTMYgQLAMP/python_gnupg-0.5.2-py2.py3-none-any.whl.asc /tmp/tmp.hTMYgQLAMP/python_gnupg-0.5.2-py2.py3-none-any.whl
gpgv: Signature made Tue Dec 12 08:14:22 2023 UTC
gpgv:                using RSA key 9147B477339A9B86
gpgv: Good signature from "Vinay Sajip (CODE SIGNING KEY) <vinay_sajip@yahoo.co.uk>"
+ [[ 0 -ne 0 ]]
+ /tmp/kivy_appdir/AppRun -m pip install --ignore-installed --upgrade --cache-dir build/deps/ --no-index --find-links file:////tmp/tmp.hTMYgQLAMP /tmp/tmp.hTMYgQLAMP/python_gnupg-0.5.2-py2.py3-none-any.whl
WARNING: The directory '/root/buskill-app/build/deps' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Looking in links: file://tmp/tmp.hTMYgQLAMP
Processing /tmp/tmp.hTMYgQLAMP/python_gnupg-0.5.2-py2.py3-none-any.whl
Installing collected packages: python-gnupg
Successfully installed python-gnupg-0.5.2
+ rm -rf /tmp/tmp.hTMYgQLAMP
++ mktemp -d
+ tmpDir=/tmp/tmp.TMbcMBjmCN
+ chown _apt:root /tmp/tmp.TMbcMBjmCN
+ chmod 0770 /tmp/tmp.TMbcMBjmCN
/tmp/tmp.TMbcMBjmCN ~/buskill-app
+ pushd /tmp/tmp.TMbcMBjmCN
+ /bin/su _apt -s /bin/bash -c '/tmp/kivy_appdir/AppRun -m pip download libusb1'
WARNING: The directory '/nonexistent/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting libusb1
  Downloading libusb1-3.1.0-py3-none-any.whl (62 kB)
  Saved ./libusb1-3.1.0-py3-none-any.whl
Successfully downloaded libusb1
WARNING: You are using pip version 20.1.1; however, version 24.0 is available.
You should consider upgrading via the '/tmp/kivy_appdir/opt/python3.7/bin/python3.7 -m pip install --upgrade pip' command.
++ ls -1
++ head -n1
+ filename=libusb1-3.1.0-py3-none-any.whl
++ /bin/su _apt -s /bin/bash -c 'curl -s https://pypi.org/simple/libusb1/'
++ grep -oE 'https://.*libusb1-3.1.0-py3-none-any.whl#'
++ sed s/#/.asc/
+ signature_url=https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl.asc
+ /bin/su _apt -s /bin/bash -c 'wget "https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl.asc"'
--2024-02-27 00:20:34--  https://files.pythonhosted.org/packages/85/5c/9169aea7690df382b677d9f725accc3ec864849c5ab49991e3823a942392/libusb1-3.1.0-py3-none-any.whl.asc
Resolving files.pythonhosted.org (files.pythonhosted.org)... 151.101.164.223, 2a04:4e42:27::223
Connecting to files.pythonhosted.org (files.pythonhosted.org)|151.101.164.223|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-02-27 00:20:36 ERROR 404: Not Found.

+ mkdir gnupg
+ chmod 0700 gnupg
~/buskill-app
+ popd
+ gpg --homedir /tmp/tmp.TMbcMBjmCN/gnupg --import build/deps/libusb1.asc
gpg: keybox '/tmp/tmp.TMbcMBjmCN/gnupg/pubring.kbx' created
gpg: /tmp/tmp.TMbcMBjmCN/gnupg/trustdb.gpg: trustdb created
gpg: key CAC936914257B0C1: public key "Vincent Pelletier <vincent.plr@wanadoo.fr>" imported
gpg: Total number processed: 1
gpg:               imported: 1
+ gpgv --homedir /tmp/tmp.TMbcMBjmCN/gnupg --keyring /tmp/tmp.TMbcMBjmCN/gnupg/pubring.kbx /tmp/tmp.TMbcMBjmCN/libusb1-3.1.0-py3-none-any.whl.asc /tmp/tmp.TMbcMBjmCN/libusb1-3.1.0-py3-none-any.whl
gpgv: can't open '/tmp/tmp.TMbcMBjmCN/libusb1-3.1.0-py3-none-any.whl.asc': No such file or directory
gpgv: verify signatures failed: No such file or directory
ERROR: Invalid PGP signature!
+ [[ 2 -ne 0 ]]
+ echo 'ERROR: Invalid PGP signature!'
+ exit 1
+ chown -R user dist
+ exit 0
+ exit 0
user@buskill:~/sandbox/buskill-app$ 

[maltfield]

I opened an upstream bug about this here:

• Fix Release Signatures on PyPI (404 Not Found) vpelletier/python-libusb1#96

[maltfield]

Per the above ticket, the python-libusb1 project is now uploading their releases (including their release signatures) as GitHub releases, which should enable us to fix this bug by switching from downloading from PyPI to GitHub.

[maltfield]

I've replaced the download URL for the signed python deps from PyPI to GitHub in all three platform's build scripts, but it looks like MacOS is still having some distinct issues finding the path to pip after upgrade

2024-03-07T23:52:10.4864930Z + /usr/local/Cellar/python@3.11/3.11.7_1/Frameworks/Python.framework/Versions/3.11/bin/pip3 install --ignore-installed --upgrade --cache-dir build/deps/ --no-index --find-links file:///Users/runner/work/buskill-app/buskill-app/build/deps/ build/deps/pip-20.1.1-py2.py3-none-any.whl
2024-03-07T23:52:10.9370650Z Looking in links: file:///Users/runner/work/buskill-app/buskill-app/build/deps/
2024-03-07T23:52:10.9683600Z Processing ./build/deps/pip-20.1.1-py2.py3-none-any.whl
2024-03-07T23:52:11.0374420Z Installing collected packages: pip
2024-03-07T23:52:12.3176550Z Successfully installed pip-23.3.1
2024-03-07T23:52:12.4074210Z ++ sort -n
2024-03-07T23:52:12.4102550Z ++ uniq
2024-03-07T23:52:12.4104960Z ++ find /usr/local/Cellar/python -type f -wholename '*bin/pip3*'
2024-03-07T23:52:12.4107930Z ++ head -n1
2024-03-07T23:52:12.4108430Z find: /usr/local/Cellar/python: No such file or directory
2024-03-07T23:52:12.4115640Z + PIP_PATH=
2024-03-07T23:52:12.4125740Z ++ pwd
2024-03-07T23:52:12.4131170Z + install --ignore-installed --upgrade --cache-dir build/deps/ --no-index --find-links file:///Users/runner/work/buskill-app/buskill-app/build/deps/ build/deps/setuptools-49.1.0-py3-none-any.whl
2024-03-07T23:52:12.4196800Z install: illegal option -- -