Maven CI Optimization Techniques — Cross-Project Comparison (44 OSS Projects)

[paoloantinori]

Maven CI Optimization Techniques — Cross-Project Comparison

Exhaustive research across 44 major open-source Java projects to identify CI build optimization techniques, who uses them, and what Apicurio Registry should adopt next. All claims verified against actual repository source code (June 2026).

Related: Discussion #8364 — Timing Baselines

---

Projects Surveyed (44 total)

Original 17 Projects

Project	Build Tool	Modules	Notes
Apicurio Registry	Maven + Quarkus	~30	Our project
Quarkus	Maven	~600	Most advanced Maven CI. Scalpel, Develocity, PTS.
Debezium	Maven + Quarkus	~40	-Dquick profile pioneer
Keycloak	Maven + Quarkus	~50	Hit same assembly OOM we fixed
Apache Camel	Maven	~800+	Largest Maven project
Apache Flink	Maven	~30+	-Dfast profile, timestamp manipulation, watchdog
ActiveMQ Artemis	Maven	~40+	-Pfast-tests, source JARs gated behind -Prelease
WildFly	Maven	~100+	Large app server
Strimzi	Maven	~20+	K8s operator, docker save/load, comment-triggered tests
Infinispan	Maven	~50+	6-layer flaky test system, zstd compression
SmallRye Mutiny	Maven	~10	Auto-activated CI profile, GCS mirror
SmallRye Reactive Msg	Maven	~15	Build-once fan-out to 30-cell matrix, JUnit tag quarantine
RESTEasy	Maven	~20	Negative path exclusions, SNAPSHOT tarball
Mandrel	Maven + native	~10	Memory partitioning (Maven 1g, native-image 13g)
Hibernate ORM	Gradle	~20	Layered cache rotation (daily + monthly)
Apache Kafka	Gradle	~50	Container isolation breakthrough
Micronaut	Gradle	~50	Develocity PTS, remote build cache

Expanded Survey (27 additional projects)

Major Maven/Gradle Projects (verified):

Project	Build Tool	CI Platform	Matrix Strategy	Key Optimizations
Spring Boot	Gradle	GitHub Actions composite actions	7-cell (2 OS × 4 Java − 1 exclusion)	Develocity (ge.spring.io), Liberica JDK, free-disk-space action, cache-read-only
Apache Druid	Maven	GitHub Actions workflow_call	18-cell (2 JDK × 9 grouped pattern shards)	-Dmaven.test.failure.ignore, quidem split 4-way, Jacoco gated by PR label
gRPC Java	Gradle	GitHub Actions	Multi-axis	Google Maven Central mirror ("less flaky"), ErrorProne, jApiCmp, compiler fork for OOM
OpenTelemetry Java	Gradle	GitHub Actions	15-cell (3 OS × 5 JDK)	config-cache, HTTP timeout tuning (120s/10 retries), Develocity, jApiCmp
Apache CXF	Maven	Jenkins declarative	3-JDK (17/21/25)	-Peverything, timeout(140), quietPeriod(30), skipStagesAfterUnstable()
Apache ZooKeeper	Maven	Jenkins declarative	2-JDK (17/25)	-Pfull-build, surefire-forkcount=4, spotbugs+checkstyle, @daily cron, timeout(2h)
Vert.x	Maven	GitHub Actions workflow_call	Parameterized (branch/jdk/os/profile)	custom maven-ci-settings.xml, io_uring memlock tuning

K8s Operator Testing (24 operators analyzed):

Operator	Language	Test Framework	E2E Framework	Cluster
cert-manager	Go	envtest	Ginkgo + kind	kind
OTEL operator	Go	envtest	Ginkgo + Kyverno Chainsaw	kind
strimzi	Java	JUnit 5 (Jupiter)	kubetest4j + kind	kind
keycloak-operator	Java	JUnit 5 + Fabric8 + Surefire	minikube	minikube
(20 more Go operators)	Go	envtest	Ginkgo variants	kind (96%)

---

Technique Comparison Matrices

Plugin Skip Techniques

Technique	Apicurio	Quarkus	Debezium	Flink	Artemis	Strimzi	SmallRye
Skip javadoc in CI	Y	Y	Y	Y	Y	Y	Y
Skip source JARs in CI	Y	Y	Y	Y	Y (release-only)	N	Y
Skip checkstyle in test shards	Y	Y	Y	Y	N	N	Y
Unified quick profile	N	Y (-Dquickly)	Y (-Dquick)	Y (-Dfast)	P	N	Y
COMMON_TEST_MAVEN_ARGS env	N	Y	?	?	N	N	N

Build Parallelism

Technique	Apicurio	Quarkus	Flink	SmallRye RM	ZooKeeper
-T in build step	Y (-T0.5C)	Y (-T1C)	Y (-T1C)	Y (-T1C)	?
Surefire forkCount > 1	N (1)	?	Y (4 UT, 2 IT)	Y (3)	Y (4)
Maven Daemon (mvnd)	N	N	N	N	N

Affected-Module Detection

Technique	Apicurio	Quarkus	Flink	Strimzi	RESTEasy
Path-based CI triggers	Y	Y	Y	Y	Y
Semantic affected-module (Scalpel)	N	Y	N	N	N
Dynamic job generation	N	Y	N	N	N
Comment-triggered expensive tests	N	N	N	Y (/gha run)	N

Caching and Artifact Transfer

Technique	Apicurio	Quarkus	Flink	Strimzi	Infinispan
Tiered cache rotation	N	Y (weekly/monthly)	N	N	Y (weekly)
Split restore/save (main-only write)	N	Y	N	Y	N
Tarball artifact (project JARs only)	N	Y	Y	N	Y (zstd)
GCS Maven Central mirror	N	Y	N	N	N
Cache pre-warm job	N	Y (weekly)	N	N	N
Google Maven Central mirror	N	N	N	N	N

New: gRPC Java uses maven-central.storage-download.googleapis.com — "less flaky than mavenCentral()"

Reliability and Flaky Tests

Technique	Apicurio	Strimzi	Infinispan	SmallRye RM	Flink	Druid
Surefire rerunFailingTestsCount	N	Y (5 UT, 2 IT)	Y (2)	N	N	N
JUnit tag quarantine	N	N	Y (unstable)	Y (slow, flaky)	N	N
Flaky test auto-issue creation	N	N	Y	N	N	N
Watchdog (jstack on timeout)	N	N	Y	N	Y	N
-Dmaven.test.failure.ignore	N	N	N	N	N	Y
Jacoco gating by PR label	N	N	N	N	N	Y

Gradle-Specific Patterns (for reference)

Technique	gRPC Java	OTEL Java	Spring Boot
Google Maven Central mirror	Y	N	N
HTTP timeout tuning (120s/10 retries)	N	Y	N
Develocity build scans	N	Y (v4.4.3)	Y (ge.spring.io)
Configuration cache	N	Y	?
jApiCmp API diff	Y (v0.4.2)	Y	N
Compiler fork for OOM	Y	N	N
free-disk-space action	N	N	Y (v1.3.1)

Observability

Technique	Apicurio	Quarkus	Micronaut	Camel	Spring	OTEL Java
Develocity Build Scans	N	Y	Y	Y	Y	Y
Predictive Test Selection	N	P (disabled)	Y (PR-only)	N	N	N
Remote build cache	N	Y	Y	Y	N	Y

K8s Operator CI Patterns

Pattern	kind	minikube	envtest	Ginkgo	KUTTL/Chainsaw
Adoption rate	96% (23/24)	12% (3/24)	100% Go ops	75%	25%
Apicurio operator	N	Y	N	N	N

Key finding: Apicurio's operator is one of only 3/24 operators still using minikube. 96% have moved to kind for faster, more reliable CI.

---

Gap Analysis

What we already do that others don't

• Manual CI timing baselines — systematic pre/post measurement (no other project does this)
• CI decision summary artifacts — our verify.yaml saves decision JSON
• Lifecycle-aware CI gating — lifecycle/wip vs lifecycle/ready-for-review

What most projects do that we don't yet

• Surefire retry — Strimzi (5x), Infinispan (2x) retry flaky tests; we re-trigger manually
• Unified quick profile — Quarkus, Debezium, Flink, SmallRye all have one; PR open ([CI Optimization] Add -Dquick profile for fast builds #8356)
• GCS Maven Central mirror — Quarkus, SmallRye, gRPC Java use Google CDN
• Develocity — Quarkus, Micronaut, Camel, Spring, OTEL Java all use build scans
• Surefire forkCount > 1 — Flink (4), SmallRye (3), ZooKeeper (4); we use default 1

Techniques only one project does (innovation opportunities)

• Scalpel (Quarkus) — semantic affected-module detection via Maven extension
• Timestamp manipulation (Flink) — prevent recompilation after artifact unpack
• Flaky test auto-issue creation (Infinispan) — auto-create GitHub issues for flaky tests
• Kyverno Chainsaw (OTEL Operator) — next-gen manifest-driven CRD testing
• Druid's Jacoco PR-label gating — skip coverage on trivial PRs

Techniques nobody does yet

• mvnd in CI — no major project uses Maven Daemon despite 2-4x benchmarks
• Combining Scalpel with per-shard path detection — Quarkus uses one, we use the other

---

What We Have Already Applied

PR	Date	Optimization	Measured Impact
#8341	Jun 23	Skip checkstyle in UT shards	-5% to -15% per shard
#8343	Jun 23	Skip git-commit-id in UT shards	~5s x 7 shards
#8353	Jun 23	Skip source/javadoc + equalize MAVEN_OPTS	Build: -40%, non-app: -20%
#8347	Jun 24	Skip javadoc in CLI workflow	CLI-only
#8375	Jun 25	Fix Debezium MySQL CI flake (ClusterIP + timeouts)	Reliability improvement
#8377	Jun 25	Add integration-tests to java path filter	CI coverage fix
#8369	Jun 25	Skip operator tests for CI-only changes	Reliability improvement

Cumulative: 26% reduction in total shard time (59m55s → 44m19s, 15m36s saved per CI run)

Still Open

PR	Optimization	Status
#8098	OTel TracingResponseFilter	Awaiting review

What We Plan to Do Next

Priority	Technique	Source	Expected Impact
HIGH	Scalpel affected-module extension	Quarkus	30-60% fewer test modules per PR
HIGH	Surefire rerunFailingTestsCount	Strimzi, Infinispan	Eliminate manual flaky retriggers
MEDIUM	Split cache restore/save	Strimzi, Quarkus	Prevent PR cache pollution
MEDIUM	GCS Maven Central mirror	Quarkus, SmallRye, gRPC Java	10-30% faster downloads
MEDIUM	Surefire forkCount tuning	Flink, ZooKeeper	Better CPU utilization
LOW	Develocity OSS sponsorship	Micronaut, Quarkus, OTEL Java	Free build scans + remote cache
LOW	Maven Daemon (mvnd) evaluation	Benchmarks	2-4x potential
LOW	kind migration for operator tests	96% of K8s operators	Faster, more reliable operator CI

---

Known Issues & FAQ — Maven CI Pitfalls

Issues we encountered (and resolved) that affect any Maven project. Documented here so other teams don't have to rediscover them.

Maven 3.9+ Lock Contention with -T Parallel Builds

Symptom: Random Could not acquire lock(s) / java.lang.IllegalStateException: Could not acquire lock(s) failures. Non-deterministic — may pass on retry.

Affected versions: Maven 3.9.0+ (introduced maven-resolver-named-locks)

Root cause: Maven 3.9+ added file-based resolver locking (maven-resolver-named-locks) to prevent concurrent modification of the local .m2/repository. Under -T (parallel module builds), multiple threads within a single Maven process contend for these file locks. The lock acquisition itself can deadlock — the locking was designed for multi-process access (e.g., two separate mvn invocations sharing a repo), not multi-thread within one process.

Fix: Add -Daether.syncContext.named.factory=noop to Maven commands that use -T. This disables the file-based locking entirely.

Why it's safe in CI: Each CI runner has its own .m2/repository. There's no cross-process contention — the scenario the locking was designed to protect against doesn't exist. Disabling it in CI is a no-op from a safety perspective.

Where NOT to apply: Local development environments where a developer might run two Maven builds in parallel against the same ~/.m2/repository. Don't add this to .mvn/maven.config — keep it in CI workflow commands only.

Our evidence: PR #8435 hit this in 2 of 3 CI runs (failed in different modules each time — Registry :: Operator, Registry :: Common). Main branch also affected. Fix validated in PRs #8417, #8430, and #8439.

Reference: MRESOLVER-392, Maven Resolver Named Locks docs

GCS Maven Central Mirror — Repository ID Cache Thrashing

Symptom: Build time increases dramatically (+274%) after adding a Maven Central mirror.

Root cause: Maven caches artifact metadata per repository ID. Changing the repository ID from central to a mirror-specific ID (e.g., google-maven-central) forces Maven to re-verify ALL cached artifacts against the new ID. Every artifact gets a fresh HTTP HEAD request.

Who it affects: Projects adopting a mirror mid-flight with an existing Maven cache. Projects that use the mirror from day one (e.g., gRPC Java) are fine — their cache is built with the mirror's ID from the start.

Workaround: If you must add a mirror, use <mirrorOf>central</mirrorOf> with <id>central</id> to preserve the repository ID. Or accept the one-time cache rebuild cost.

Surefire forkCount — Not Always Faster

Symptom: Increasing forkCount from 1 to 2+ makes tests slower, not faster.

Root cause: JVM fork startup (~300ms per fork) dominates when individual tests are fast (~1ms). The parallelism gain only exceeds fork overhead when avg_test_duration × test_count > fork_startup × forkCount.

Rule of thumb: forkCount > 1 helps when tests take seconds (like Flink, ZooKeeper). For millisecond-range tests (like Apicurio's unit tests), stick with forkCount=1.

Maven Daemon (mvnd) — Local Dev Only, Not CI

Symptom: mvnd is slower than plain mvn on CI first builds.

Root cause: mvnd's speed comes from JVM warmup, JIT retention, and classloader caching across repeated builds. CI containers are ephemeral — no daemon survives between runs. Cold daemon start adds ~20% overhead vs plain mvn.

Who it helps: Developers iterating locally (10-26x speedup on warm daemon). Apache Camel Quarkus (1,336 modules) is the flagship user.

Co-creator's take: Peter Palaga (mvnd co-creator): "I see little potential for mvnd in the area of continuous integration."

GitHub Actions Lifecycle Bot Race — CI Silently Skips on PR Creation

Symptom: PR CI shows all jobs as "skipping" despite correct labels and path filters. Decide step passes but every downstream job is skipped.

Root cause: When a PR is created with a label like orchestrator/disabled, two events fire in rapid succession: (1) opened event triggers Verify correctly, (2) lifecycle bot adds lifecycle/new label → labeled event triggers a second Verify run. The second run's Decide step sees a lifecycle label and skips everything. GitHub overwrites the first run's results with the second run's "all skipped" results.

Fix: After creating a PR, wait 15 seconds and check gh pr checks <number>. If everything shows "skipping", force-push (git commit --amend --no-edit && git push --force-with-lease). The synchronize event from the force-push triggers a clean run.

How to detect lifecycle-skip vs path-skip: Check the Decide step logs. lifecycle-ready=false means the bot race happened. lifecycle-ready=true with specific run-*=false means path filters correctly skipped irrelevant jobs.

Our evidence: Hit on PRs #8375, #8345, #8350, #8435, #8439 — every single PR creation. We now have a skill (pr-lifecycle) and hook to catch this automatically.

GitHub Actions Silently Skips PRs with Merge Conflicts

Symptom: PR's Verify workflow never triggers — no run appears at all, not even a "skipped" one. Force-pushes, label changes, close/reopen all fail to trigger CI.

Root cause: GitHub Actions silently drops pull_request events for PRs in CONFLICTING state. No error, no log, no "skipped" run — it simply doesn't fire.

Fix: Before any CI debugging, check gh pr view <number> --json mergeable --jq '.mergeable'. If CONFLICTING, rebase the branch. Only investigate labels, queue, or workflow config if the PR is MERGEABLE.

Our evidence: Spent hours debugging #8353 — tried label changes, force pushes, empty commits, close/reopen, investigated zombie queued runs. The actual problem was a one-line merge conflict in verify-unit-tests.yaml. A simple rebase fixed it instantly.

CI Service Readiness — Retry Loops, Not Fixed Sleep

Symptom: CI steps that start a Docker container then immediately test against it fail intermittently with connection refused or timeout.

Root cause: sleep N followed by a single healthcheck is a race condition. On slow runners, the fixed sleep isn't enough. On fast runners, you're wasting time waiting for a service that's already up.

Fix: Always use a retry loop:

for i in $(seq 1 20); do
  curl -sf http://localhost:PORT/healthcheck >/dev/null 2>&1 && break
  echo "Waiting... ($i/20)"
  sleep 0.5
done
curl -sf http://localhost:PORT/healthcheck || (echo "Not ready after 10s"; exit 1)

This is both faster (succeeds immediately when ready) and more reliable (waits longer if needed).

CI-Only Flags — Don't Pollute Shared Config

Symptom: Developers report broken local builds after a CI optimization merges.

Root cause: Adding CI-specific flags to .mvn/maven.config or pom.xml profiles affects ALL builds — including local developer environments. Example: adding -s .mvn/settings.xml to maven.config silently overrides every developer's ~/.m2/settings.xml, breaking custom repos, credentials, proxies, and private mirrors.

Rule: CI-only changes go in workflow YAML files (command-line flags, env: variables). Shared config (.mvn/maven.config, pom.xml profiles) must be safe for all environments. If a change has developer-facing tradeoffs, document them explicitly in the PR and get team buy-in.

Scalpel Affected-Module Detection — Test-Scope Dependencies Limit Savings

Symptom: Scalpel (or GIB) is integrated for affected-module test skipping, but most PRs still run nearly all tests because the main application module test-depends on many library modules.

Root cause: In Quarkus-based projects, @QuarkusTest integration tests often live in the app module and import library modules (serdes, SDKs, schema utilities) in <scope>test</scope>. Scalpel correctly treats test-scope dependencies as "affected" — a change to avro-serde means app's serdes tests might behave differently, so app tests must run.

Impact on savings:

PR scope	Typical savings
app/src/ only (leaf module)	Biggest — 1/N tested
SDK or tooling modules only	Large — if app doesn't test-depend on them
Library modules that app test-depends on	Modest — app shards still run
Foundation modules (common/)	Moderate — some independent modules still skip

Could you restructure? Moving @QuarkusTest serdes tests from app/src/test/ to a separate module or to integration-tests/ would decouple app from serdes — but you lose fast embedded testing (seconds, no Docker). This trades developer productivity for CI savings, which is usually the wrong tradeoff.

Rule: Before integrating Scalpel, audit your app module's test-scope dependencies. The savings are real but concentrated in leaf-module and independent-module PRs, not in library PRs that the app tests against. Set expectations accordingly.

Our evidence: Stress-tested 8 scenarios. app/src/ change → 1/41 tested. serdes/ change → 2/41 in non-app shard but all app shards still run. common/ change → 30/41 tested (11 independent modules correctly skipped). See PR #8442 for full results.

Parallel Maven Builds Break Operator Integration Tests — Resource Exhaustion

Symptom: Adding -T 1C (parallel threads) to .mvn/maven.config causes Kubernetes operator integration tests to fail with ConditionTimeoutException — deployments never become ready, tests time out after ~120 seconds.

Root cause (corrected after investigation): The failure is caused by resource exhaustion, not shared Java memory state. The operator Makefile runs -pl controller -am — only ONE module's tests execute, so cross-module test parallelism is not the issue. Under -T 1C, 8 parallel Maven threads perform Quarkus augmentation and compilation while Minikube pods are also running. The CPU/memory pressure on the CI runner causes pods to start slower, exceeding the awaitility timeout.

Initial (incorrect) hypothesis: We initially attributed the failure to shared static fields in the test base class (ITBase) and CDI singleton conflicts between concurrent test modules. While these ARE code quality issues worth fixing, they're not the cause of the CI failure — the modules don't actually run tests concurrently under -pl controller -am.

Fix: Override -T 1C with -T1 in the operator Makefile to force single-threaded operator builds. This reduces resource pressure during test execution. The compilation speedup from parallelism on 3 operator modules (~2-3 seconds) isn't worth the resource contention.

Code quality improvement (separate): We also prototyped converting ITBase from static to @TestInstance(PER_CLASS) instance lifecycle (PR #8452). This eliminates unnecessary static mutable state and future-proofs for intra-module test parallelism, but it does not fix the CI failure — that's a resource problem, not a memory sharing problem.

Lesson for other projects: When operator tests fail under parallel Maven builds, check whether the failure is:

1. Resource exhaustion (pods timing out, Quarkus augmentation + Minikube competing for CPU/memory) → fix with -T1 or resource limits
2. Shared state (multiple test modules running concurrently, fighting over cluster resources) → fix with @TestInstance(PER_CLASS), namespace isolation, @ResourceLock

Misdiagnosing (2) when the actual cause is (1) leads to unnecessary refactoring that doesn't fix the problem.

Our evidence: 5 consecutive main branch failures (June 29–July 2). Fixed by PR #8448 (-T1 override). Root cause analysis in issue #8447. Code quality prototype in PR #8452.

Cross-Project Technique Adoption — Measure Before Copying

Symptom: A CI optimization that works great for Project A makes things worse (or adds pointless complexity) for Project B.

Root cause: Techniques are designed for specific constraints. Maven cache splitting was motivated by cache poisoning security at 100+ PRs and 10GB budget exhaustion — neither applies to a project with 20 PRs. forkCount=2 works when tests take seconds (Flink) but hurts when tests take milliseconds. GCS mirrors work when adopted from day one but cause cache thrashing mid-flight.

Rule: Before implementing: ask "What problem does this solve? Do we have that problem?" Check the original PR/issue — the motivation is there. Compare scale. If the benefit is marginal (<1 minute), the implementation must be trivially simple. Document negative results to prevent future re-attempts.

---

Verification: All claims verified against actual repository source code and CI evidence (July 2026). This is a living document — comments, corrections, and suggestions welcome.