[CI] Consolidate ingestion integration test fixtures

[xiangfu0]

Summary

• Consolidate the pauseless controller-failure scenarios around one cluster and reference dataset.
• Share one fixture per pauseless server topology across segment-commit and consuming-transition failures, including dedup.
• Keep the full LLC Kafka 3 suite once and replace duplicate Kafka 4 coverage with a focused failure-and-recovery scenario.
• Restore the previously omitted Kinesis ingestion coverage, then group the shard-change matrix onto shared stream timelines while preserving independent tables, offsets, counts, and segment-state assertions.
• Share one hybrid Pinot/Kafka fixture across the canonical hybrid, DateTime-field, and ingestion-config scenarios.
• Keep exactly two GitHub integration jobs, but run two isolated Maven/Surefire lanes inside each job with disjoint build directories, temp/log paths, ZK/Pinot ports, JaCoCo files, and reports.
• Rebalance measured slow classes and focused suites across the four lanes.
• Harden the quota overload test to generate a concurrent burst rather than relying on a timing-sensitive sequential loop.
• Harden consolidated cleanup so assertion failures are aggregated, teardown stages are attempted, resources are released in dependency order, and failed shutdown ownership remains retryable.
• Raise serial integration-test Surefire maximum heap to 8 GiB while keeping its initial heap at 4 GiB. Concurrent CI lanes deliberately use 1–4 GiB per fork plus -XX:ActiveProcessorCount=2 to fit the 16 GiB / 4-vCPU runner.

Hosted result

Final workflow: Pinot Integration Tests run 31285625202, head 1e47ecdea6.

Two unchanged hosted attempts both passed below 50 minutes:

Attempt	Set 1	Set 2	Critical path
1	45m14s	46m48s	46m48s
2	46m01s	38m04s	46m01s

The worst observed totals were 46m01s for Set 1 and 46m48s for Set 2, leaving 3–4 minutes of headroom. Compared with the prior serial hosted run (70m08s / 72m00s), this is approximately 35% faster on both jobs.

Both attempts ran the same 1,715 physical TestNG invocations with 0 failures, 0 errors, and 3 intentional skips. All four Codecov uploads succeeded in both attempts.

Consolidation evidence

• Pauseless controller failures: 476.1s to 173.2s in the source PR, down 63.6%.
• Pauseless server failures: 305.6s to 163.5s, down 46.5%.
• LLC Kafka ingestion: 827.4s to 332.2s, down 59.9%.
• Grouped Kinesis workflow step: 15m46s to about 5m47s, down roughly 63%.
• Shared hybrid fixture: three standalone classes totaling 555.1s in the first lane experiment became one 28-test suite taking 271.5s and 260.6s in the final attempts, down 51–53%. The physical inventory changes from 60 to 28 by replacing DateTime's duplicate inherited generic matrix with two focused engine assertions; canonical hybrid coverage remains once.

Final lane counts:

• Set 1A: 139 invocations.
• Set 1B: 1,153 invocations.
• Set 2A: 113 invocations.
• Set 2B: 310 invocations.

Validation

• Final hosted attempts: 2/2 workflow successes; both integration jobs below 50 minutes.
• Shared hybrid: 28/28 in both attempts.
• Shared Kafka: 23/23 discovered in both attempts, with 2 intentional skips.
• Kinesis: 7/7 methods in both attempts, preserving 23 logical scenarios.
• QueryQuota: 13/13 in both attempts; the formerly failing quota-overload method passed in 2.529s and 2.415s.
• Hosted peak lane RSS remained about 3.1–5.0 GiB. No JVM/process OOM, exit 137, bind collision, cleanup assertion, or routing-table leak was found.
• Direct -Dtest operation is preserved; named suites are skipped for direct selections so a requested class runs exactly once.
• Effective-POM ownership audit found no duplicate or omitted active class identity; the only intentional inventory delta is the hybrid 60-to-28 focus.
• Dependency-aware JDK 25 compilation, Spotless, Checkstyle, license format/check, XML/shell/YAML/action validation, and git diff --check passed.

Supersedes #19192, #19193, #19194, and #19196.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.64%. Comparing base (e51b4e4) to head (1e47ecd).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19190      +/-   ##
============================================
- Coverage     66.65%   66.64%   -0.01%     
  Complexity     1423     1423              
============================================
  Files          3443     3443              
  Lines        218632   218663      +31     
  Branches      34793    34801       +8     
============================================
+ Hits         145726   145730       +4     
- Misses        61192    61216      +24     
- Partials      11714    11717       +3     

Flag	Coverage Δ
custom-integration1	?
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	66.64% <ø> (-0.01%)	⬇️
lane-a	100.00% <ø> (?)
lane-b	0.00% <ø> (?)
temurin	66.64% <ø> (-0.01%)	⬇️
unittests	66.64% <ø> (-0.01%)	⬇️
unittests1	57.31% <ø> (+0.08%)	⬆️
unittests2	38.90% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Jackie-Jiang]

Nice!