Replacing heavy integration test fixtures with minimal purpose-built packages

[Harsh-2005d]

Opening this discussion as a follow-up to #1592, where the goal is to replace cryptography in the pip/e2e_rust_extensions integration test with something lighter.

Problem

The current pip/e2e_rust_extensions test uses cryptography built from a GitHub archive URL. It works, but it's the slowest integration test by far and produces an SBOM of ~7,600 lines, mostly transitive Cargo crates . A real-world alternative with a Rust git dep was explored but introduced Perl as a system requirement, which adds its own overhead. (hermetoproject/integration-tests#106)

Proposed approach

A purpose-built minimal fixture hosted under the hermetoproject org, a small maturin/pyo3 package. This gives full control, keeps the SBOM minimal, requires no extra system dependencies, and deliberately covers the transitive Rust git dep case.

I've put together a proof of concept here: https://github.com/Harsh-2005d/hello-rust-ext - a small maturin/pyo3 package with rand as a git-sourced Rust dependency.

The bigger question

If this becomes the pattern for other backends too, a few options worth discussing:

• One repo per fixture — easy to reason about individually, but maintenance overhead grows with every new backend
• Single repo under the org — single source of truth, cleaner org. But pip's subdirectory git URL support is not straightforward and it's unclear how Hermeto handles that path.
• Automation — Dependabot on the fixture repo to keep deps current without manual effort.
• Versioning — Pin via git tags in the integration test, or point to main? Pointing to main risks silent breakage whenever the fixture changes, so tagged releases seem safer

Happy to move forward once there's a consensus on direction.

[Harsh-2005d]

After thinking on it further, I want to update the "single repo" option with more detail since it has two distinct variants worth separating out.

On the subdirectory approach — I looked at how Hermeto handles downloaded packages in pip backend. The extraction logic in filter_packages_with_rust_code assumes the package lives at the root of the archive, and there's no #subdirectory= support in the VCS URL path. So a monorepo where fixtures live in subdirectories doesn't work with the current code without a change to the VCS handling pipeline. I'd consider this a separate issue worth addressing on its own.

On the branch-per-fixture approach — this actually works today since a branch archive URL gives Hermeto a clean root-level package. More importantly, hermetoproject/integration-tests already uses exactly this pattern (pip/e2e_rust_extensions, yarn/e2e, etc.), so a test-fixtures repo organized the same way would be consistent with existing org conventions.

On scope — I checked the line counts across all bom.json files in the integration test suite. maven_e2e (8932 lines) and several gomod_* tests (2982–3271 lines) are comparable in size to pip_e2e_rust_extensions (7677 lines). The pip/rust case is different: the bloat is incidental to what the test actually validates, which is just that Hermeto can handle a maturin-built package. So this fixture strategy is really a pip-specific concern for now, not something that needs to scale to every backend immediately.

So the two realistic options as I see them:

• Polyrepo — one dedicated repo per fixture, each a proper standalone package. Simple, works, isolated. Slight org clutter if fixtures grow, but maintenance overhead should be minimal for dummy projects.
• Monorepo with branches — a single hermetoproject/test-fixtures repo with one branch per fixture, following the same convention as integration-tests. Single place to maintain, consistent with existing org patterns, technically works with current Hermeto code.

Both options work. Happy to hear other's opinion.

[eskultety]

Polyrepo

The worst case scenario with ^this one is what we had before we consolidated all our integrations tests to a single monolith we have today. The current setup has flaws, but arguably what having to remember and maintain N repositories per-test case was a nightmare. I don't expect things to be that bad here, especially since the main motivation is to just write git dependencies for various backends' integration tests. I'd still be probably more inclined towards...

Monorepo with branches

...and the reason is quite simple - whether we'd have a unique fixture repo or a branched setup, the effort to contribute a change would be equally high. Neither is really straightforwards from contributor's perspective, but the latter IMO poses significantly less maintenance burden and I'd personally not like to go back to unique fixture repositories. Maybe my vision here is skewed and I'm simply wrong in the long run, but I simply don't want to deal with multiple repositories that a contributor would have to clone, it's inconvenient and were I a contributor I might got discouraged by such practice myself :).

Automation — Dependabot on the fixture repo to keep deps current without manual effort.

I'm not sure how ^this would work with the branched setup, one could only ever point to a particular commit digest, but IIUC dependabot would not be able to update that even if we maintain linear history on the branches?

It works, but it's the slowest integration test by far and produces an SBOM of ~7,600 lines, mostly transitive Cargo crates

Yes, this is highly motivating - we're still a small project, i.e. it feels unacceptable to me that a single e2e test takes 4-6 minutes to execute in a local dev environment (not talking about restricted cloud resources).

We might need to consider transitive deps scenarios with these fixture dep setups, but I guess we can cross that bridge when we get there.

[slimreaper35]

I'd still be probably more inclined towards Monorepo with branches

+1. Note that we also use the same approach for https://github.com/hermetoproject/doc-examples.

Automation — Dependabot on the fixture repo to keep deps current without manual effort.

We don't use dependabot for integration tests anyway, so I would skip this topic here. Each update for the test fixture -> integration tests repo update -> quick update for Hermeto test SBOMs to avoid test failures. Just a nightmare.

[Harsh-2005d]

I experimented further with fixture

• fetch-deps completed roughly ~30 seconds faster compared to the existing cryptography-based fixture in my local testing.
• However, the generated bom.json was still around ~6000 lines, compared to the existing ~7000 lines :(

I also attempted to run the pip/e2e_rust_extensions E2E workflow locally to validate the fixture end-to-end, but ran into an RPM repository issue during the image build step. The generated RPM lockfile appeared to reference UBI10 repositories while the test Containerfile uses UBI9, which caused package resolution failures during the offline build.

error logs:

FAILED tests/integration/test_pip.py::test_e2e_pip[pip_e2e_rust_extensions] - RuntimeError: Building image failed. Output:
STEP 1/7: FROM registry.access.redhat.com/ubi9/ubi@sha256:c2bc4ee4ec4f200af9d78fd1307b7ff39f5db2c27166e52fb37650dee68a5ffd
STEP 2/7: RUN if getent hosts www.google.com; then echo "Has network access!"; exit 1; fi
--> f2e4ec01d04b
STEP 3/7: RUN dnf -y install     cargo     python3     python3-pip     python3-setuptools     python3-wheel     python3-devel     libffi-devel     openssl-devel
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Repository 'ubi-10-baseos-rpms' is missing name in configuration, using id.
Repository 'ubi-10-appstream-rpms' is missing name in configuration, using id.
ubi-10-baseos-rpms                               10 MB/s |  18 kB     00:00    
ubi-10-appstream-rpms                           106 MB/s | 109 kB     00:00    
Package python3-3.9.21-1.el9_5.x86_64 is already installed.
Package python3-setuptools-53.0.0-13.el9.noarch is already installed.
No match for argument: python3-wheel
Error: Unable to find a match: python3-wheel
Error: building at STEP "RUN dnf -y install     cargo     python3     python3-pip     python3-setuptools     python3-wheel     python3-devel     libffi-devel     openssl-devel": while running runtime: exit status 1

Then i removed the python3-wheel.

2026-06-11 13:40:52,809 INFO Reading RPM lockfile: /tmpb8pgo1mu.hermeto-source-copy/rpms.lock.yaml
  2026-06-11 13:40:52,842 INFO Downloading files for 'x86_64' architecture.
  2026-06-11 13:44:54,343 ERROR Unsuccessful download: https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi10/10/x86_64/appstream/os/Packages/p/python3-pip-23.3.2-11.el10_2.noarch.rpm
  2026-06-11 13:44:54,348 ERROR FetchError: exception_name: ClientPayloadError, details: Response payload is not completed: <ContentLengthError: 400, message='Not enough data to satisfy content length header.'>
  Error: FetchError: exception_name: ClientPayloadError, details: Response payload is not completed: <ContentLengthError: 400, message='Not enough data to satisfy content length header.'>
    The error might be intermittent, please try again.
    If the issue seems to be on the hermeto side, please contact the maintainers.
  
assert 15 == 0
 +  where 0 = <ExitError.ERR_OK: 0>.value
 +    where <ExitError.ERR_OK: 0> = TestParameters(branch='pip/e2e_hello_extension', packages=({'type': 'pip'}, {'type': 'rpm'}), check_output=True, expected_error=<ExitError.ERR_OK: 0>, expected_output='', global_flags=[], flags=[], repo_url='https://github.com/Harsh-2005d/integration-tests.git', hermeto_env={}, unset_hermeto_env=set(), netrc_content=None).expected_error

[Harsh-2005d]

integration-test repo.
SBOM generated by test-data maybe broken coz that also failed but the test-data did overwrite, the sbom i generated during local fetch-deps was around ~6000 lines.

[Harsh-2005d]

@eskultety @slimreaper35 can you please take a look? Thanks.

[slimreaper35]

@Harsh-2005d I replaced UBI9 for UBI10 in hermetoproject/integration-tests@434a02f, so the python3-wheel rpm is not needed.

However, the generated bom.json was still around ~6000 lines, compared to the existing ~7000 lines :(

Well, I think the main motivation here is the overall build time. The SBOM output is in the "nice to have" bucket.

[Harsh-2005d]

Thanks, the bugs were fixed.

the overall build time.

So i ran both the test end to end.

# new test
passed in 237.64s (0:03:57)

real	4m7.600s
user	13m1.849s
sys	1m2.301s

# old test
passed in 377.23s (0:06:17) 

real	6m27.429s
user	16m45.661s
sys	2m6.601s

[Harsh-2005d]

To summarize:

• Consensus was monorepo with branches (hermetoproject/test-fixtures, one branch per fixture, following the same pattern as doc-examples/integration-tests).
• Validated the approach end-to-end with the maturin/pyo3 fixture build time down from 6m27s to 3m40s .
• Fixture: hello-rust-ext
• Integration test setup: integration-tests

@eskultety @slimreaper35 any concerns remaining, and how should we proceed?