Working Group Proposal - Binary Reproducibility

[Ladicek]

Objective

Ensure that Quarkus application builds are binary reproducible. That is, building the same source with the same toolchain produces bit-for-bit identical output.

The Problem

Binary reproducibility (the property that building the same source code with the same tools produces identical artifacts) is a cornerstone of supply chain security and build trust. It allows independent parties to verify that a published binary was genuinely built from its claimed source, and it simplifies debugging, caching, and compliance audits.

Quarkus performs significant build-time work: it runs deployment processors, generates bytecode via recorders or Gizmo, and assembles the final application artifact. Each of these steps is a potential source of non-determinism. When iteration order over sets, maps, or annotation metadata is not stable, the generated JARs can differ between builds, even though the inputs are identical.

Known sources of non-determinism

Several sources of non-determinism have already been identified and fixed:

• Non-deterministic Jandex index creation: Jandex indices used during augmentation were not created in a deterministic order and so could not guarantee stable ordering, causing downstream build steps to process classes in varying order across builds.
• Missing hashCode() in Jandex ClassInfo: the absence of a proper hashCode() implementation led to unpredictable behavior when ClassInfo instances were stored in hash-based collections, propagating non-determinism into any build step that iterated over such collections.
• Unstable iteration in build steps: multiple extension processors used HashMap or HashSet where insertion order matters for bytecode generation, or used non-comparable MultiBuildItems as an input to bytecode generation.
• Non-deterministic bytecode generation: Gizmo function counters and recorder-generated bytecode could vary between augmentation runs.

The Proposed Solution

Finish fixing remaining sources of non-determinism

Continue the work already in progress to identify and fix non-deterministic behavior in Quarkus core and extensions. This includes stabilizing bytecode generation, ensuring deterministic ordering in build items, and fixing any remaining issues in the augmentation pipeline.

Establish reproducibility verification criteria

Define what "reproducible" means in precise, testable terms for Quarkus applications. The criteria are conceptually obvious (same input -> same output), but the details matter: which artifacts are compared, how dev-services-related bytecode is handled, what constitutes "same toolchain", and how to account for legitimate differences (e.g., build timestamps in Maven metadata).

Build automated reproducibility testing into CI

Add CI infrastructure that performs reproducibility checks (building the same application multiple times and comparing the output) to catch regressions early. This is already underway with the reproducibility test mechanism introduced in #54420 and the proposed nightly CI workflow in #54895.

Produce guidelines for extension developers

Extensions contribute bytecode and resources to the final application. Extension authors need clear guidance on how to write build steps that preserve reproducibility: for example, using sorted collections, stable comparators, and deterministic code generation patterns. These guidelines will help both Quarkus core extensions and Quarkiverse extensions.

It is my personal belief that for the foreseeable future, we should be able to rely on stable iteration of HashSets and HashMaps, even if these collections explicitly document that their iteration order is undefined, provided that the keys have deterministic hashCode(). They almost always have or can have; one class that is often used as a key and doesn't have deterministic hashCode() is java.lang.Class.

Definition of Done

• Quarkus core JVM-mode builds produce bit-for-bit identical application artifacts when built from the same source with the same JDK and Maven version
• Automated reproducibility tests run in CI to prevent regressions
• Existing Quarkus core extensions pass reproducibility checks
• Reproducibility verification criteria and reproducibility guarantees are documented
• Guidelines for extension developers on writing reproducible build steps are published

Scope of Work

In Scope

• Fixing remaining sources of non-determinism in Quarkus core and extensions (JVM mode)
• Defining reproducibility verification criteria and tooling
• Documentation of guarantees and known limitations
• Developer guidelines for writing reproducibility-safe extensions
• CI infrastructure for automated reproducibility testing

Out of Scope (for now)

• Gradle builds: the initial focus is on Maven-based builds. It is possible Gradle builds will be reproducible out of the box, or with very minor additional work, but this is currently unclear.
• Native image reproducibility: GraalVM native image determinism is a separate and significantly more complex problem. It may be addressed in a follow-up working group, depending on progress.
• Reproducible builds of Quarkus itself: making the Quarkus framework build reproducible is a separate effort.

Organizing the Work

Communication

• Zulip channel: #dev > WG - Binary reproducibility
  • Previously used Zulip channels:
  • #dev > Reproducible builds
  • #dev > Build reproducibility progress
  • #dev > Build reproducibility in Jandex/ArC
• Points of contact: @reaver585, @gsmet
  • These people are also involved: @Ladicek, @mkouba, @manovotn

Timeline

This working group is not tied to a specific Quarkus release. The work is incremental (each fix independently improves the situation) and will continue until the definition of done is satisfied.

Existing Work

Significant progress has already been made. Notable PRs include:

• Introduce a mechanism to avoid generating dev services related bytecode for reproducibility checks #54897
• Add nightly scheduled CI workflow to run reproducibility tests #54895
• Reset global Gizmo function counters between augmentations when running reproducibility tests #54754
• Jandex: order Jandex input deterministically #54456
• Add generated bytecode reproducibility checks #54420
• Stabilize generated bytecode in Arc #54371
• Use output timestamp for deterministic info build time #54369

And many more fixes across individual extensions.

[cescoffier]

WG board ready: WG - Binary Reproducibility

[dmlloyd]

It's possible to prove that builds are not reproducible, but it is really hard if not impossible to prove that they are. What are the acceptance criteria?

[Ladicek]

The definition of how we verify is a big part of the scope of this WG.

[reaver585]

The approach we try follow right now is:

• Take all the tests in the upstream repo that run the augmentation phase, and adapt them s.t. the augmentation is reran multiple times.
• Compare the bytecode across these runs.
• If bytecode mismatches, find the source of the mismatch and fix it.

Even after having done that, we can't say that Quarkus gives reproducibility guarantees.
This is the best effort to make the builds as reproducible as we possibly can with what we have right now.

[dmlloyd]

I'm wondering if we want to formalize some things about code generation and things like that, to try to take a more proof-oriented approach.

[reaver585]

I think if that's possible, that would be great..

I think a full proof over arbitrary code generation is not achievable with JVM, is it? Perhaps we could think of some advanced version of enforcing input-determinism: flagging the known impure sources in augmentation code rather than catching them after the fact by diffing. That could be "more proof-oriented".

@Ladicek recently merged a change with something like that (for testing scenarios only), and @gsmet has a similar idea for quite some time now.

Does that go into the direction you're thinking about, @dmlloyd?

[dmlloyd]

Yeah that makes some sense I think. But we'd also be proactive about it, laying down a general set of rules about member ordering, how constant pools should be arranged, and things like that (maybe). But maybe this is all infeasible dreaming.