CI Build Optimization — Timing Baselines and Measured Impact

[paoloantinori]

CI Build Optimization — Timing Baselines and Measured Impact

This document tracks the measured performance impact of Maven CI build optimizations applied to the Apicurio Registry project. It serves as a living reference for the team to evaluate whether each optimization is worth keeping.

Related: Discussion #8365 — Cross-Project Comparison (44 OSS Projects) | Issue #8348 — ttl.sh replacement

---

Baseline v1 — Pre-Optimization (June 22, 2026)

Three consecutive green runs on main before any optimizations:

Job	Run 1	Run 2	Run 3	Average
Build Java	11m36s	5m19s	5m38s	7m31s
Build UI	3m58s	3m41s	3m26s	3m41s
UT app-rest	6m49s	6m42s	7m24s	6m58s
UT app-sql	6m06s	6m14s	6m08s	6m09s
UT app-kafkasql	7m38s	7m23s	8m15s	7m45s
UT app-gitops	5m06s	5m40s	5m29s	5m25s
UT app-kubernetesops	4m01s	4m26s	5m03s	4m30s
UT app-other	13m58s	13m54s	13m36s	13m49s
UT non-app	7m59s	8m23s	7m02s	7m48s

---

Baseline v2 — Post First Wave (June 23-24, 2026)

After PRs #8341 (checkstyle skip), #8343 (git-commit-id skip), #8353 (source/javadoc skip + MAVEN_OPTS):

Job	Original	v2 Avg	Delta
Build Java	7m31s	4m29s	-40%
app-rest	6m58s	5m52s	-16%
app-sql	6m09s	5m49s	-5%
app-kafkasql	7m45s	7m10s	-7%
app-gitops	5m25s	4m56s	-9%
app-kubernetesops	4m30s	4m26s	-1%
app-other	13m49s	13m09s	-5%
non-app	7m48s	6m16s	-20%

Total shard time: 59m55s → 52m09s (-13%)

---

Baseline v3 — Post Second Wave (June 25, 2026)

After PRs #8347, #8375, #8377, #8369 + .mvn/maven.config -T 1C:

Job	v2 Avg	v3 Avg	Delta vs v2	Delta vs v1
Build Java	4m29s	3m38s	-19%	-52%
app-rest	5m52s	5m19s	-9%	-24%
app-sql	5m49s	4m30s	-23%	-27%
app-kafkasql	7m10s	6m02s	-16%	-22%
app-gitops	4m56s	3m37s	-27%	-33%
app-kubernetesops	4m26s	2m57s	-33%	-34%
app-other	13m09s	12m55s	-2%	-7%
non-app	6m16s	5m19s	-15%	-32%

Total shard time: 52m09s → 44m19s (-15% vs v2, -26% vs v1)

---

Baseline v4 — Post Third Wave (June 29, 2026)

After PRs #8378 (-Dlocal), #8367 (.mvn/maven.config), #8388 (surefire retry), #8390 (operator timeouts), #8349 (ttl.sh replacement):

Job	v3 Avg	Run 1 (Jun 25)	Run 2 (Jun 25)	Run 3 (Jun 29)	v4 Avg	Delta vs v3	Delta vs v1
Build Java	3m38s	3m23s	3m53s	3m54s	3m43s	+2%	-50%
app-rest	5m19s	5m05s	5m33s	5m30s	5m22s	+1%	-23%
app-sql	4m30s	4m30s	4m30s	5m49s	4m56s	+10%	-20%
app-kafkasql	6m02s	5m48s	6m17s	5m44s	5m56s	-2%	-24%
app-gitops	3m37s	3m38s	3m36s	3m35s	3m36s	0%	-34%
app-k8sops	2m57s	2m45s	3m09s	2m07s	2m40s	-10%	-41%
app-other	12m55s	12m43s	13m08s	12m43s	12m51s	-1%	-7%
non-app	5m19s	5m00s	5m38s	4m08s	4m55s	-8%	-37%

Total shard time: 44m19s → 43m59s (stable, -27% vs v1)

Cumulative improvement from v1: 59m55s → 43m59s = 15m56s saved per CI run (-27%)

---

Key Observations (v4)

• Build step stable at ~3m43s — 50% faster than original 7m31s
• app-kubernetesops hit a new low of 2m07s (single run) — 53% faster than v1's 4m30s
• non-app hit 4m08s — 47% faster than v1's 7m48s
• v3→v4 gains are modest because the big wins (plugin skips, parallel threads) were already applied
• Reliability improvements from this wave (surefire retry, operator timeouts, ttl.sh fix) don't show in timing but reduce manual retriggers

---

What Merged

PR	Date	Optimization	Impact
#8341	Jun 23	Skip checkstyle in unit test shards	~10-15s x 7 shards
#8343	Jun 23	Skip git-commit-id in unit test shards	~5s x 7 shards
#8353	Jun 23	Skip source/javadoc JARs + equalize MAVEN_OPTS	Build: -40%, non-app: -20%
#8347	Jun 24	Skip javadoc in CLI workflow	CLI-only
#8375	Jun 25	Fix Debezium MySQL CI flake (ClusterIP + timeouts)	Reliability
#8377	Jun 25	Add integration-tests to java path filter	CI coverage
#8369	Jun 25	Skip operator tests for CI-only changes	Reliability
#8378	Jun 29	Rename -Dfast → -Dlocal with plugin skip properties	Developer experience
#8367	Jun 29	.mvn/maven.config with -T 1C default parallel	Build speed
#8388	Jun 29	Surefire rerunFailingTestsCount=2 in CI	Reliability
#8390	Jun 29	Operator test timeouts increased + configurable	Reliability
#8349	Jun 29	Replace ttl.sh with workflow artifacts for operator tests	Reliability
#8439	Jul 2	Fix Maven lock contention (-Daether.syncContext.named.factory=noop)	Reliability
#8448	Jul 2	Fix operator test regression from -T 1C (force -T1 in Makefile)	Reliability

Open PRs

PR	Optimization	Status
#8435	Skip ZIP assembly in CI builds (-Dskip.zip.assembly=true)	Rebased after #8439 merge, CI re-running
#8442	Scalpel affected-module detection (draft)	CI green, stress-tested locally, awaiting team discussion

Note on #8439 — Maven 3.9+ Lock Contention

Maven 3.9+ introduced maven-resolver-named-locks for file-based resolver locking. Under -T parallel builds, multiple threads contend for these locks and the acquisition itself deadlocks — producing Could not acquire lock(s) failures. This is not a timing fluke: PR #8435 hit it in 2 of 3 CI runs, and main branch runs also fail intermittently.

Fix: -Daether.syncContext.named.factory=noop disables the file-based locking entirely. This is safe in CI because each GitHub Actions runner has its own .m2/repository — there is no cross-process contention. The locking was designed for multi-process scenarios (e.g., two Maven instances sharing a local repo), not multi-thread within a single process.

Scope: Applied to all 9 CI workflow files (verify, operator, release, reusable-docker-build) — only on Maven commands that use parallel threads. NOT added to .mvn/maven.config to avoid affecting local development. Previously validated in PRs #8417 and #8430 but was bundled with cache-splitting changes that were rejected — this PR applies the fix standalone.

Note on #8442 — Scalpel Affected-Module Detection

Scalpel (v0.3.7) is a Maven core extension that analyzes git diffs against the POM dependency graph to skip tests on modules unaffected by a PR's changes. No static mapping files — fully dynamic from POM structure. Used by Quarkus and Apache Camel.

Local stress testing (8 edge cases):

Scenario	Tested	Skipped	Correct?
.mvn/** change	41/41 (full build)	0	✅ fullBuildTriggers
serdes/kafka/avro-serde/ only	2/41	39	✅
common/ change	30/41	11	✅ java-sdk, config-index correctly skipped
app/src/ only	1/41	40	✅ app is a leaf module
pom.xml comment-only	0/41 (no-op)	41	✅ semantic POM analysis
pom.xml version property bump	22/41	19	✅ traces through dependencyManagement
-Dscalpel.enabled=false	41/41 (disabled)	0	✅ override works
-Dtest= filter + Scalpel	app tests ran	—	✅ but see below

Architectural finding: app test-depends on all serdes modules (<scope>test</scope>) for embedded @QuarkusTest integration tests (e.g., AvroSerdeTest, ProtobufSerdeTest). These 11 test classes start an embedded Quarkus registry and test the full serialize→register→deserialize roundtrip. This is a standard Quarkus testing pattern — fast (seconds, no Docker) and complementary to the container-based integration-tests/.

This means Scalpel correctly identifies app as affected by any serdes change, so all 6 app shards still run for serdes-only PRs. The savings are concentrated elsewhere:

PR scope	Savings
app/src/ only	Biggest — 1/41 tested, 40 skipped
java-sdk/ only	Large — ~2/41 tested
common/	Moderate — 30/41 tested, 11 skipped
serdes/ only	Modest — non-app shard benefits, app shards still run
pom.xml version bump	Variable — depends on which modules use the dependency

Recommendation: Keep the current architecture. Restructuring (moving serdes tests to integration-tests/ or a separate module) would sacrifice fast embedded testing for CI savings — inverted priorities. The Scalpel savings from app-only and java-sdk-only PRs are valuable enough on their own.

Planned

Task	Optimization	Expected Impact	Source
REG-43	Scalpel affected-module detection	See analysis below	Quarkus
REG-44	Develocity Build Scans	Automated performance monitoring	Quarkus, Micronaut, Spring
REG-50	Migrate integration tests from minikube to kind	Faster, more reliable	96% of K8s operators

---

Tried & Rejected (with evidence)

PR	Optimization	Result	Evidence
#8414	Surefire forkCount=2 for non-app shard	Slower (+16% to +33%)	Local benchmark: JVM fork startup overhead exceeds parallelism gain. Our tests average ~1ms each — fork startup (~300ms) dominates. Flink/ZooKeeper succeed because their tests take seconds. Threshold: forkCount > 1 helps when avg test duration × test count > fork startup × forkCount.
#8417	Split cache restore/save (actions/cache/restore)	Build failures	Maven lock contention with parallel threads. Root cause found: Maven 3.9+ resolver file locking (.locks/ directory). Fixed with -Daether.syncContext.named.factory=noop — build passes. BUT: not worth implementing. Research across real incidents (Angular cache poisoning $31K bounty, Ultralytics crypto miner, TanStack 84 malicious npm versions) shows cache splitting is motivated by: (1) security with pull_request_target triggers, (2) cache budget exhaustion at 100+ PRs, (3) ~30s save time per PR. At our scale (~20 PRs, standard pull_request), none of these justify the complexity. Closed as won't-do.
#8411	GCS Maven Central mirror	Regression	Changing repository ID from central to google-maven-central forces Maven to re-verify all cached artifacts. Build went from 3m43s to 13m53s (+274%). Works for projects that use the mirror from the start (gRPC Java) but causes cache thrashing when adopted mid-flight.
#8432	Maven Daemon (mvnd) for CI	Not viable	Local dev tool, not CI optimization. Co-creator: "little potential for CI." Cold start 20% slower. No OSS project uses mvnd in CI.
#8098	TracingResponseFilter for OTel	Redundant	Quarkus auto-instrumentation already sets http.response.status_code and marks 5xx as StatusCode.ERROR. Verified by Carles with test in #8415.

---

Visual Summary

Shard Timing Progression (v1 → v2 → v3 → v4)

---
config:
  xyChart:
    width: 800
    height: 400
---
xychart-beta
    title "Unit Test Shard Duration (seconds)"
    x-axis ["Build", "app-rest", "app-sql", "kafkasql", "app-gitops", "app-k8sops", "app-other", "non-app"]
    y-axis "Duration (seconds)" 0 --> 850
    bar [451, 418, 369, 465, 325, 270, 829, 468]
    bar [269, 352, 349, 430, 296, 266, 789, 376]
    bar [218, 319, 270, 362, 217, 177, 775, 319]
    bar [223, 322, 296, 356, 216, 160, 771, 295]

Loading

Bars: v1 (tallest) → v2 → v3 → v4 (shortest). Biggest cumulative gains in Build (-50%), app-k8sops (-41%), non-app (-37%).

Cumulative Time Saved

---
config:
  xyChart:
    width: 800
    height: 400
---
xychart-beta
    title "Total Shard Time (seconds) — Lower is Better"
    x-axis ["v1 Baseline", "#8341 checkstyle", "#8343 commitid", "#8353 src/javadoc", "#8347 cli-javadoc", "Jun 25 wave", "Jun 29 wave"]
    y-axis "Total shard time (seconds)" 2400 --> 3700
    line [3595, 3525, 3490, 3129, 3129, 2659, 2639]

Loading

Each point = total time across all 8 shards after that wave merged. 59m55s → 43m59s = 15m56s saved per CI run.

Where Time Is Still Spent (v4)

pie title Current CI Time Distribution (v4 — 43m59s total)
    "app-other (12m51s)" : 771
    "app-kafkasql (5m56s)" : 356
    "app-rest (5m22s)" : 322
    "app-sql (4m56s)" : 296
    "non-app (4m55s)" : 295
    "Build (3m43s)" : 223
    "app-gitops (3m36s)" : 216
    "app-k8sops (2m40s)" : 160

Loading

app-other dominates at 29% — mostly actual test execution time. Next optimization target: Scalpel (skip irrelevant modules). forkCount tuning was benchmarked and rejected (see below).

---

Last updated: 2026-07-02 (#8439 and #8448 merged, #8435 rebased). 14 PRs merged total. This is a living document — comments and suggestions welcome.