test/ci: enterprise-grade hardening — chaos e2e, mTLS, soak, perf trend, API gate, pipeline reliability

[lukekim]

Summary

Closes the P0/P1 gaps from the test-posture assessment: the SDK's resilience features now have end-to-end proof against real runtime failures, mTLS has real handshake coverage, performance is tracked as a trend with deterministic contracts, a nightly soak hunts slow leaks, an API-compatibility gate protects the public surface, and the CI pipeline itself is hardened against the flakes that kept turning PRs red.

P0 — CI pipeline reliability

• Spice CLI install: one composite action (.github/actions/setup-spice) with retries + binary verification + GITHUB_TOKEN (the JDK-matrix job lacked the token — the source of most installer flakes), replacing six inlined copies in two languages.
• Runtime startup: one composite action (.github/actions/start-spice-app) with a real /v1/ready poll, replacing four copies of the quickstart heredoc — and retiring the flaky sleep 10 in the pre-existing build/publish jobs as a side effect.
• OWASP split into two steps: a best-effort NVD data update (continue-on-error, step timeout) and a gating offline check — network flakiness can no longer mask real CVSS ≥ 7 findings, and findings can no longer be swallowed as flakes. nvdValidForHours=24 makes warm caches skip the refresh. Wired for an NVD_API_KEY secret (needs to be added in repo settings; free from NVD).

P0 — Chaos e2e (ChaosE2ETest + e2e-chaos job, every PR)

Manages its own spiced processes (dataset-free pod, no external dependencies) via a new SpicedProcess fixture and proves against the real runtime:

• crash + restart on the same address → same client reconnects and cached prepared statements transparently re-prepare
• query issued during downtime → recovers via retry backoff (LB-failover scenario)
• SIGKILL mid-stream → clean classifiable error, never a hang; client recovers
• SIGSTOP (frozen peer, undetectable by TCP alone) → keep-alive detects within its configured window (bound derived from SpiceClient's actual constants), recovery after SIGCONT

P1 — mTLS integration (MtlsTest, TestCerts, TLS mode in TestFlightSqlServer)

Runtime-generated EC P-256 CA/server/client certificates, real handshakes in-process: custom-CA trust, full mutual TLS including parameterized queries, and rejection paths that assert the failure is a TLS failure (this immediately caught the negative tests passing via IPv6 connection-refused before any handshake — both sides now pin 127.0.0.1).

P1 — Perf trend + soak + API gate

• PerfBenchmarkTest emits customSmallerIsBetter JSON (BENCH_JSON); nightly publishes the trend to gh-pages (branch created; action pinned to a full commit SHA; runs with -Djacoco.skip=true so latencies measure the real SDK, not coverage-instrumented classes). Benchmark ops assert full row counts so a result-dropping regression can't publish an "improved" latency.
• Nightly soak (SoakTest, 30 min): 4-worker mixed workload with periodic reset(); asserts zero errors, non-empty results throughout, leak-free close, bounded thread growth from a warm baseline, and p99 stability across the run. Local validation: ~8-9k ops/s, 500k+ ops per minute-long smoke.
• japicmp gate vs the last release in the quality job; verified the current surface is purely additive vs 0.6.0. japicmp.oldVersion is a pom property and the publish preflight fails on a vacuous self-comparison.

Security hardening (from adversarial review)

• publish.yaml never executes mutable remote code: the release pipeline downloads a pinned, sha256-verified CLI/runtime release, and the GPG signing key is imported only after all downloaded code has run.
• Third-party benchmark action pinned by full commit SHA (it receives a write token).

Review chain applied

• Codex adversarial review (17 findings): 15 fixed — including a soak heap-exhaustion hazard, chaos process-management races, silent no-op chaos gating, and the workflow security items above; 2 accepted with rationale (temp-dir/cert cleanup: throwaway 1-day fixtures in user-private OS temp on ephemeral runners).
• /simplify (4 parallel reviewers): composite-action consolidation, SpicedProcess extraction, shared countRows reuse, EC certs, dead-code and FQN cleanups; nightly chaos job dropped as a verbatim duplicate of the per-push job.

Validation

• 293 tests + SpotBugs green (mock-only and with a live runtime)
• Chaos e2e: 4/4 against real spiced, repeatedly
• Soak smoke: 60s @ 8.3k ops/s, zero errors, threads flat, 20.8M rows verified
• japicmp, jacoco-skip path, and all workflow/action YAML validated
• docs/testing.md documents the tiers and how to run each locally

Follow-ups (deferred, noted in review)

• Add NVD_API_KEY repo secret (speeds the OWASP data refresh)
• Enable GitHub Pages for gh-pages if the benchmark dashboard should be served (data accumulates in-branch regardless)
• Consider excluding ai.spice.example from the published jar (would obsolete the japicmp exclusion)

[Copilot]

Pull request overview

This PR hardens CI and test posture for the Spice Java SDK by adding runtime-backed resilience and TLS/mTLS integration tests, introducing long-running/perf signal coverage, and consolidating CI workflow logic into reusable composite actions.

Changes:

• Adds new test suites/fixtures: chaos E2E against a real spiced process, runtime-generated TLS/mTLS handshake coverage, nightly soak, and benchmark trend emission.
• Introduces CI hardening: composite actions for Spice install and runtime startup, nightly workflows, OWASP dependency-check split, and a japicmp API-compatibility gate.
• Exposes keep-alive tuning constants for resilience tests and adds build properties for japicmp/Jacoco-skip scenarios.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/test/java/ai/spice/TestFlightSqlServer.java	Adds TLS/mTLS capability to the in-process Flight SQL server fixture.
src/test/java/ai/spice/TestCerts.java	Introduces runtime-generated PEM cert fixtures for TLS/mTLS tests.
src/test/java/ai/spice/SpicedProcess.java	Adds a real spiced process manager for chaos/lifecycle tests.
src/test/java/ai/spice/SoakTest.java	Adds an availability-gated long-running soak test for leaks/latency/thread stability.
src/test/java/ai/spice/PerfBenchmarkTest.java	Adds deterministic row-count contracts and JSON benchmark emission for trend tracking.
src/test/java/ai/spice/MtlsTest.java	Adds in-process TLS and mutual-TLS handshake integration tests with negative-path assertions.
src/test/java/ai/spice/ChaosE2ETest.java	Adds gated chaos E2E tests validating retry/reconnect/keep-alive behaviors against real runtime failures.
src/main/java/ai/spice/SpiceClient.java	Exposes keep-alive tuning constants for tests and reuses them in channel configuration.
pom.xml	Adds japicmp “old version” property/gate configuration and ensures empty argLine when JaCoCo is skipped.
docs/testing.md	Documents the new tiered testing approach and how to run tiers locally.
.github/workflows/publish.yaml	Hardens release pipeline (pinned downloads, delayed GPG import) but still has a runtime startup sleep.
.github/workflows/nightly.yaml	Adds nightly soak and benchmark-trend publishing workflow.
.github/workflows/build.yaml	Consolidates setup/start logic, adds chaos job, adds japicmp gate, and hardens OWASP dependency-check execution.
.github/actions/start-spice-app/action.yml	New composite action to start a quickstart runtime and poll /v1/ready instead of sleeping.
.github/actions/setup-spice/action.yml	New composite action to install Spice CLI/runtime with retries and verification.

Suppressed comments (1)

.github/workflows/publish.yaml:78

• This workflow step still uses a fixed sleep for runtime startup. The PR description states build/publish jobs should use the shared /v1/ready poll (start-spice-app) to avoid flakiness; keeping sleep 10 here risks the same intermittent failures and contradicts the stated hardening.

      - name: Init and start spice app
        run: |
          spice init spice_qs
          cd spice_qs
          # Define the dataset inline rather than fetching spiceai/quickstart from

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (3)

.github/workflows/publish.yaml:80

• The PR description says the flaky fixed sleeps in build/publish jobs were retired in favor of the /v1/ready poll, but publish.yaml still starts the runtime with a fixed sleep (later in this step). This leaves the release pipeline exposed to the same readiness flake the composite action was introduced to remove.

      - name: Init and start spice app
        run: |
          spice init spice_qs
          cd spice_qs
          # Define the dataset inline rather than fetching spiceai/quickstart from

src/test/java/ai/spice/SpicedProcess.java:109

• ProcessBuilder is constructed with findBinary() directly; if no spiced binary is present this passes null as the executable and will throw a NullPointerException with little context. Even though ChaosE2ETest asserts binary presence, this helper should fail fast with a clear message (and avoid surprising NPEs if reused elsewhere).

    private void launch() throws Exception {
        ProcessBuilder builder = new ProcessBuilder(
                findBinary(),
                "--http", "127.0.0.1:" + httpPort,
                "--flight", "127.0.0.1:" + flightPort,

src/test/java/ai/spice/PerfBenchmarkTest.java:111

• recordBench() assumes that an existing BENCH_JSON file contains a valid JSON array. If the file exists but is empty/whitespace (or partially written from a prior run), JsonParser.parseString(...).getAsJsonArray() will throw and fail the benchmark run. Making the read tolerant of empty content avoids fragile failures in CI/local runs.

    private static synchronized void recordBench(String name, String unit, long value) throws Exception {
        String path = System.getenv("BENCH_JSON");
        if (path == null || path.isEmpty()) {
            return;
        }
        Path file = Path.of(path);
        JsonArray entries = Files.exists(file)
                ? JsonParser.parseString(Files.readString(file)).getAsJsonArray()
                : new JsonArray();

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.github/workflows/build.yaml:215

• The shell snippet builds an EXTRA string and expands it unquoted into the mvn command. If the API key ever contains whitespace or shell metacharacters, this can break argument parsing (and unquoted expansion is generally risky). Use an args array and pass it as "${EXTRA_ARGS[@]}" instead.

        run: |
          EXTRA=""
          if [ -n "$NVD_API_KEY" ]; then
            EXTRA="-DnvdApiKey=$NVD_API_KEY"
          fi
          mvn dependency-check:update-only -B $EXTRA

src/test/java/ai/spice/SpicedProcess.java:109

• SpicedProcess.launch() passes findBinary() directly into ProcessBuilder. If the binary is missing (or the method is reused outside the ChaosE2ETest gate), this will throw a NullPointerException from ProcessBuilder with little context. Fail fast with a clear IllegalStateException when no spiced binary is found.

    private void launch() throws Exception {
        ProcessBuilder builder = new ProcessBuilder(
                findBinary(),
                "--http", "127.0.0.1:" + httpPort,
                "--flight", "127.0.0.1:" + flightPort,
                "--metrics", "127.0.0.1:" + metricsPort);