pip: python generator produces empty sources

[muelli]

I run the python generator against this requirements file: https://github.com/gnome-keysign/gnome-keysign/blob/1c5b04ac0b265491ab65b651e3bbbe0f4a310a18/requirements.txt ) and it produces empty sources:

$ flatpak-pip-generator --requirements-file=requirements.txt
========================================================================
Downloading sources
========================================================================
Running: "pip3 download --exists-action=i --dest /tmp/pip-generator-python3-requirements7hbolm42 -r /home/muelli/vcs/gnome-keysign/requirements.txt --require-hashes"
/usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
Collecting attrs==21.4.0
  Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting autobahn[twisted]==22.3.2
  Using cached autobahn-22.3.2.tar.gz (376 kB)

...
...

Collecting zope.interface==5.4.0
  Using cached zope.interface-5.4.0-cp38-cp38-manylinux2010_x86_64.whl (259 kB)
Collecting setuptools
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools from https://files.pythonhosted.org/packages/0c/3c/548d361162702df85a0301f0cd0c47d05176b20bb086077a0fda740daf41/setuptools-62.0.0-py3-none-any.whl#sha256=a65e3802053e99fc64c6b3b29c11132943d5b8c8facbcc461157511546510967 (from autobahn[twisted]==22.3.2->-r requirements.txt (line 11))
Failed to download
Please fix the module manually in the generated file
========================================================================
Downloading arch independent packages
========================================================================
========================================================================
Obtaining hashes and urls
========================================================================
========================================================================
Generating dependencies
========================================================================
Generating dependencies for attrs

$ head python3-requirements.json
{
    "name": "python3-requirements",
    "buildsystem": "simple",
    "build-commands": [],
    "modules": [
        {
            "name": "python3-attrs",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"attrs==21.4.0\" --no-build-isolation"
            ],
            "sources": []
        },
        {
            "name": "python3-autobahn",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"autobahn[twisted]==22.3.2\" --no-build-isolation"
            ],
            "sources": []
        },

I see the error there, but I remember that setuptools requires some special care. Maybe it's not f-p-generator who ought to take that special care. I'll happily forward this to autobahn, if necessary.

[RyuzakiKK]

I investigated a bit this issue. The minimal reproducer is a requirements.txt file with just one dependency, e.g.:

cffi==1.15.1 --hash=sha256:dd86c085fae2efd48ac91dd7ccffcfc0571387fe1193d33b6394db7ef31fe2a4

In order to remove flatpak-pip-generator from the equation we can launch pip3 download directly as f-p-g would do:

$ pip3 download --exists-action=i --dest /tmp/aa -r requirements.txt --require-hashes

Collecting cffi==1.15.1
  File was already downloaded /tmp/aa/cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Collecting pycparser
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    pycparser from https://files.pythonhosted.org/packages/62/d5/5f610ebe421e85889f2e55e33b7f9a6795bd982198517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl (from cffi==1.15.1->-r requirements.txt (line 1))

--require-hashes can also be omitted because pip3 install will automatically add it if there is a --hash in the requirements file.

pip complains about the dependencies not being pinned because the hashes option is: "Require a hash to check each requirement against, for repeatable installs".
One way to fix this is to explicitly add all dependencies in the requirements.txt, e.g. in this case:

cffi==1.15.1 --hash=sha256:dd86c085fae2efd48ac91dd7ccffcfc0571387fe1193d33b6394db7ef31fe2a4
pycparser==2.21 --hash=sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9

would be enough to fix pip3 download.

Alternatively you should avoid using the hashes altogether, e.g.:

cffi==1.15.1

Regarding setuptools, it doesn't seem to require anything special with pip download. It required --allow-unsafe if you wanted to use pip install, but not with pip download.

If I got all the info correctly, I don't think there is much that could be done in flatpak-pip-generator.

[muelli]

hm. well. I think having the pip generator produce an empty file is suboptimal, at least. So I believe that there is at least something, that could be done.

Are you saying that the pip generator calls out to "pip3 download" which in turn errors out? Would it help to propagate this error?

And am I understanding it correctly that a full list of dependencies with their hashes makes things work? How do you generate such full list then? Maybe that information can be given back to the user.

[RyuzakiKK]

hm. well. I think having the pip generator produce an empty file is suboptimal, at least. So I believe that there is at least something, that could be done.

Are you saying that the pip generator calls out to "pip3 download" which in turn errors out? Would it help to propagate this error?

Yes definitely. A more clear error should definitely help.

And am I understanding it correctly that a full list of dependencies with their hashes makes things work? How do you generate such full list then? Maybe that information can be given back to the user.

Yes, and pip-compile supposedly already includes the complete list of dependencies by default.

In your example though it seems like pip3 download gets confused when we bring twisted[tls] as a dependency.

This is another minimal reproducer for the specific issue that you reported:

$ touch requirements.in
$ echo "magic-wormhole>=0.10.2" > requirements.in

$ pip-compile --generate-hashes --resolver=backtracking requirements.in

At the end of the file we will have the following warning:

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools

So let's use --allow-unsafe:

$ pip-compile --generate-hashes --allow-unsafe --resolver=backtracking requirements.in

This time setuptools is included in the requirements.txt output and no warnings are generated.

Then if we launch flatpak-pip-generator:

$ ./flatpak-pip-generator --requirements-file=requirements.txt
[...]
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    twisted>=20.3.0 from https://files.pythonhosted.org/packages/ac/63/b5540d15dfeb7388fbe12fa55a902c118fd2b324be5430cdeac0c0439489/Twisted-22.10.0-py3-none-any.whl (from autobahn[twisted]==23.1.2->-r requirements.txt (line 16))

In the generated requirements.txt we indeed don't have a pinned twisted, but instead we have twisted[tls]. AFAICT this is either an issue on the pip-compile side, because it should list both twisted[tls] and twisted as dependencies, or it is an issue with pip download where it should not complain about the missing twisted because twisted[tls] replaces it.

[RyuzakiKK]

FTR the upstream issue with pip seems to be pypa/pip#9644