Hashes not supported in requirements.txt

[carlcsaposs-canonical]

Bug Description

#1157 introduced a regression—hashes in requirements.txt cannot be used since PYDEPS requirements do not use hashes, and "hash-checking is an all-or-nothing proposition" (from https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode)

To Reproduce

Create requirements.txt file with --hash and install a charm lib with PYDEPS

Example: https://github.com/canonical/zookeeper-operator/actions/runs/5651535680

Environment

Ubuntu 22.04

charmcraft.yaml

# Copyright 2023 Canonical Ltd.
# See LICENSE file for licensing details.

type: charm
parts:
  charm:
    charm-binary-python-packages:
      - setuptools
    build-packages:
      # The following are needed for tls-certificates-interface
      - build-essential
      - python3-dev
      - pkg-config
      - libffi-dev
      - libssl-dev
      - rustc
      - cargo
bases:
  - build-on:
      - name: "ubuntu"
        channel: "22.04"
    run-on:
      - name: "ubuntu"
        channel: "22.04"

Relevant log output

2023-07-25 01:24:55.067 :: 2023-07-25 01:24:48.341 :: Installing dependencies
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:48.341 :: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:51.875 :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip3', '--version']
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:52.200 ::    :: pip 22.0.2 from /root/parts/charm/build/staging-venv/lib/python3.10/site-packages/pip (python 3.10)
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:52.240 :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip3', 'install', '--upgrade', 'setuptools']
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:52.679 ::    :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0)
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.117 ::    :: Collecting setuptools
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.200 ::    ::   Downloading setuptools-68.0.0-py3-none-any.whl (804 kB)
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.264 ::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 804.0/804.0 KB 13.0 MB/s eta 0:00:00
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.372 ::    :: Installing collected packages: setuptools
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.373 ::    ::   Attempting uninstall: setuptools
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.373 ::    ::     Found existing installation: setuptools 59.6.0
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.471 ::    ::     Uninstalling setuptools-59.6.0:
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.481 ::    ::       Successfully uninstalled setuptools-59.6.0
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.928 ::    :: Successfully installed setuptools-68.0.0
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:53.998 :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip3', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt', 'cosl', 'pydantic']
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.595 ::    :: Collecting cosl
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.595 ::    :: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.595 ::    ::     cosl from https://files.pythonhosted.org/packages/47/33/01443388eb758a558ea7de71375a63c9cf31d810f6d83501e93bc669f52f/cosl-0.0.5.tar.gz#sha256=31c131d1f04c061d3fbef49a4e0a175d4cb481deeb06d0cb3c7b242e4c5416be
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.651 :: Traceback (most recent call last):
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.651 ::   File "/snap/charmcraft/1429/lib/charmcraft/charm_builder.py", line 432, in <module>
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.652 ::     main()
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.652 ::   File "/snap/charmcraft/1429/lib/charmcraft/charm_builder.py", line 427, in main
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.652 ::     builder.build_charm()
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.652 ::   File "/snap/charmcraft/1429/lib/charmcraft/charm_builder.py", line 102, in build_charm
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.653 ::     self.handle_dependencies()
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.653 ::   File "/snap/charmcraft/1429/lib/charmcraft/charm_builder.py", line 314, in handle_dependencies
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.653 ::     self._install_dependencies(staging_venv_dir)
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.653 ::   File "/snap/charmcraft/1429/lib/charmcraft/instrum.py", line 152, in _f
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.654 ::     return func(*args, **kwargs)
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.654 ::   File "/snap/charmcraft/1429/lib/charmcraft/charm_builder.py", line 274, in _install_dependencies
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.654 ::     _process_run(cmd)
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.654 ::   File "/snap/charmcraft/1429/lib/charmcraft/charm_builder.py", line 368, in _process_run
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.654 ::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.654 :: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip3', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt', 'cosl', 'pydantic'] execution failed with retcode 1
2023-07-25 01:24:55.067 :: 2023-07-25 01:24:54.770 Parts processing error: Failed to run the build script for part 'charm'.

[marceloneppel]

It works with the previous revision of charmcraft (1424).

[lengau]

Thanks for the report! I'm going to relate #1140 to this too since this is a use case we want to ensure doesn't break when we change the dependency management.

This was a use case I didn't consider (for and which we probably need a regression test as we tackle future dependency issues). I'll have to think through options with this.

[taurus-forever]

Dear @lengau , is it possible to release a fix for this regression only?
The #1140 looks huge with unpredictable release time (hope I am wrong).

At the moment we have to workaround it in many places (see all the linked commits):

• pin old charmcraft 2.3.0 (oblivious negative side-effects)
• remove hashes from requirements.txt => security concerns.

[lengau]

@taurus-forever and @carlcsaposs-canonical — could you test the latest charmcraft on edge? This should be fixed there, and if so I can possibly cherry-pick the changes and do a quick release

[carlcsaposs-canonical]

@lengau Does that fix #1135 and #1179 simultaneously? From my understanding of the fallback mechanism in #1233, it looks like we can pick 1 of 2 of the issues to be solved, but not both.

If my understanding ^ is correct, I believe we are more concerned about #1135 than #1179, so if we have to pick only 1 of 2 issues to be solved, we'd pick #1135—so we wouldn't be using hashes