Cannot replicate build behaviour from charmcraft v2 to v3

[kian99]

Bug Description

When building a charm that contains the python cryptography lib in requirements.txt, I'm unable to replicate the behaviour from v2 to v3.

A portion of my requirements.txt file:

cryptography == 42.0.7

My charmcraft.yaml file contains:

type: "charm"
parts:
  charm:
    charm-python-packages: [setuptools]
    charm-binary-python-packages:
      - cryptography

When building with charmcraft v2 I see the following output:

pip', 'install', '--no-binary=Jinja2,charmhelpers,cosl,hvac,ops,pip,pydantic,pytest-interface-tester,requests,setuptools,wheel', '--requirement=requirements.txt', 'PyYAML', 'attrs', 'importlib-resources', 'pip', 'pydantic', 'pydantic-core', 'pydantic<2.0', 'setuptools', 'urllib3', 'wheel', 'zipp']
2024-08-02 17:33:06.516 ::    :: Collecting cryptography==42.0.7 (from -r requirements.txt (line 5))
2024-08-02 17:33:06.520 ::    ::   Downloading cryptography-42.0.7-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (5.3 kB)

When building with charmcraft v3 I see:

2024-08-08 18:54:21.960 :: 2024-08-08 18:54:06.235 :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', 'PyYAML', 'attrs', 'cryptography', 'importlib-resources', 'jsonschema', 'pydantic-core', 'urllib3', 'zipp']
2024-08-08 18:54:21.960 :: 2024-08-08 18:54:08.623 ::    :: Successfully installed PyYAML-6.0.2 attrs-24.2.0 cffi-1.17.0 cryptography-43.0.0 importlib-resources-6.4.0 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 pycparser-2.22 pydantic-core-2.21.0 referencing-0.35.1 rpds-py-0.20.0 typing-extensions-4.12.2 urllib3-2.2.2 zipp-3.19.2
2024-08-08 18:54:21.960 :: 2024-08-08 18:54:08.626 ::    ::
...
2024-08-08 18:54:21.960 :: 2024-08-08 18:54:09.317 :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=:all:', '--requirement=requirements.txt', 'pydantic', 'pydantic<2.0']
...
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:39.147 ::    :: Collecting cryptography==42.0.7 (from -r requirements.txt (line 5))
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:39.150 ::    ::   Using cached cryptography-42.0.7.tar.gz (671 kB)
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:39.308 ::    ::   Installing build dependencies: started
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.391 ::    ::   Installing build dependencies: finished with status 'error'
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::   error: subprocess-exited-with-error
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::   × pip subprocess to install build dependencies did not run successfully.
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::   │ exit code: 1
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::   ╰─> [74 lines of output]
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::       Collecting setuptools>=61.0.0
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::         Using cached setuptools-72.1.0-py3-none-any.whl
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.397 ::    ::       Collecting wheel
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Using cached wheel-0.44.0-py3-none-any.whl
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::       Collecting cffi>=1.12
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Using cached cffi-1.17.0.tar.gz (516 kB)
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Installing build dependencies: started
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Installing build dependencies: finished with status 'done'
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Getting requirements to build wheel: started
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Getting requirements to build wheel: finished with status 'done'
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Preparing metadata (pyproject.toml): started
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Preparing metadata (pyproject.toml): finished with status 'done'
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::       Collecting setuptools-rust>=1.7.0
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::         Using cached setuptools_rust-1.10.1-py3-none-any.whl
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.398 ::    ::       Collecting pycparser (from cffi>=1.12)
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::         Using cached pycparser-2.22-py3-none-any.whl
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::       Collecting semantic-version<3,>=2.8.2 (from setuptools-rust>=1.7.0)
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::         Using cached semantic_version-2.10.0-py2.py3-none-any.whl
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::       Building wheels for collected packages: cffi
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::         Building wheel for cffi (pyproject.toml): started
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::         Building wheel for cffi (pyproject.toml): finished with status 'error'
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::         error: subprocess-exited-with-error
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::
2024-08-08 18:16:43.327 :: 2024-08-08 18:16:42.399 ::    ::         × Building wheel for cffi (pyproject.toml) did not run successfully.

It seems with the new additional pip install --no-binary=:all: --requirement=requirements.txt command, that cryptography's build dependencies are attempting to be installed but I already specified it as a binary.

Is there any way to replicate the v2 behavior?

To Reproduce

Pack a charm with cryptography in requirements.txt and include it in charm-binary-python-packages in charmcraft.yaml,

Environment

Building in a LXD 22.04 container.

charmcraft.yaml

type: "charm"
parts:
  charm:
    charm-python-packages: [setuptools]
    charm-binary-python-packages:
      - cryptography
      - jsonschema
      - PyYAML
      - attrs
      - importlib-resources
      - urllib3
      - zipp
      - pydantic-core
bases:
  - build-on:
      - name: "ubuntu"
        channel: "22.04"
        architectures:
        - amd64
    run-on:
      - name: "ubuntu"
        channel: "22.04"
        architectures:
        - amd64

Relevant log output

Added above.

[lengau]

Thanks for the report. Unfortunately this is intentional, as the other mechanism caused several other types of build failures.

If you're using a full requirements.txt file to list both direct and indirect dependencies though, you could use strict dependencies by setting charm-strict-dependencies: true in the charm part.

This is a preferred way of building the charm, as it will use only a single pip command and allow pip to figure out the inconsistencies rather than potentially building a broken charm.