[P1] Testcontainers E2E: cross-instance takeover + crash/restart + stale-owner

[qqeasonchen]

Tracking sub-issue of #5354. Part of the production-HA acceptance plan (Phase 2, P1).

What this PR adds

A Testcontainers-based E2E harness that exercises the cross-instance, multi-tenant, fencing-aware production code path. The test class is the answer to #5347 (D2b) for the production-HA flow specifically; the A2A-flow Testcontainers harness remains a separate issue.

Harness layout

A new eventmesh-testcontainers module (build.gradle + gradle.properties, Apache 2.0 license) provides:

• MetaStoreContainer: Nacos 2.3.x with persistent volume; expose 8848.
• BrokerContainer: RocketMQ 5.3.x nameserver + broker; expose 9876 (NS) and 10911 (broker).
• EventMeshContainer: built from docker/Dockerfile with eventmesh.delivery.topology=PARTITION_OWNED_PULL, eventmesh.meta.store=nacos, Nacos address injected; expose 8081 (admin) and 10000 (HTTP publish).

Test cases

1. CrashRestartTest: 2 Runtime instances, 1 topic with 4 partitions, 1000 events published; kill instance A, restart, assert all 1000 events delivered exactly once across the surviving + restarted instances.
2. TakeoverTest: 2 instances, instance A owns all 4 partitions, publish 100 events with no ACK from subscribers; kill A; instance B acquires the partitions, redelivers the 100 in-flight events (within the at-least-once bound), and continues.
3. StaleOwnerDispatchRejectedTest: 2 instances, instance A dispatches, manually bump the partition token in Nacos to simulate instance C taking over; A's next dispatch throws StaleOwnerException; the dispatcher removes the partition from its owned set on the next tick.
4. QuotaA2AStreamCancelTest: 1 instance, open 5 A2A streams (CONNECTIONS quota=3); 2 must be rejected with 429; cancel 1, the next submit must succeed.
5. MixedBackendTopologyTest: 1 instance with PARTITION_OWNED_PULL against RocketMQ 5 lite topic (partitionCountKnown=false) + no degradeWithoutPartitionCount flag; the boot fails with the documented error message; set the flag, the instance boots and falls back to the documented behavior.

Acceptance criteria

• The eventmesh-testcontainers module compiles and its gradle :eventmesh-testcontainers:test runs in CI within 15 minutes.
• The 5 test cases above are green.
• docs/production-readiness.md is updated to point to the harness and the supported backend × topology matrix.
• A row in docs/architecture-review/evidence.md links each test to its commit + CI run.

Verification

./gradlew :eventmesh-testcontainers:test

Depends on / blocks

• Depends on: [P0] propagate FencingToken epoch through dispatch / ACK / offset / DLQ #5360 (fencing), [P0] harden EventMeshFrame: FrameLimits, immutability, fuzz/boundary tests #5361 (immutable frame); the cross-instance tests exercise the production code paths introduced there.
• Blocks: [P1] chaos (MetaStore / broker / watch / RocksDB) + rolling upgrade + admin security #5364 (chaos tests use the same Testcontainers harness as a starting point).

References

• eventmesh-testcontainers/ (new module)
• eventmesh-runtime/.../cluster/PartitionOwnership.java
• eventmesh-runtime/.../delivery/ReliableDispatcher.java
• eventmesh-runtime/.../security/gate/SecurityGate.java
• eventmesh-runtime/.../a2a/A2AGatewayHttpHandler.java
• docs/production-readiness.md section 5

Part of the production-HA topology in #5354. See also #5352 and #5353. Closes #5347 for the production-HA flow (D2a #5346 was the A2A flow; D2b is split).

[qqeasonchen]

Deferring this sub-issue per plan-owner decision (2026-09-09): the cross-instance takeover coverage it demands needs Testcontainers (Nacos + RocketMQ5 containers), which cannot run in the current DLP-restricted dev environment (no Docker).

The compensating coverage already on develop:

• [P0] propagate FencingToken epoch through dispatch / ACK / offset / DLQ #5360 (PR fix(#5360): fence stale owners from durable side effects (offsets / broker ACK) #5371) — StaleOwnerFencingTest: fenced-owner ack() writes zero offsets and fires zero broker ACKs (the takeover no-side-effect contract, driven at dispatcher level)
• [P0] wire DeliveryTopology + MetaStore + FencingToken into UniRuntime boot (fail-fast) #5359 (PR fix(#5359): wire DeliveryTopology + MetaStore into runtime boot; rollback on failure #5370) — EventMeshApplicationStartupTest + UniRuntimeTopologyWiringTest: two-instance partition split, fail-fast wiring, boot rollback
• [P0] harden EventMeshFrame: FrameLimits, immutability, fuzz/boundary tests #5361 (PR fix(#5361): FrameLimits — enforce wire bounds at encode AND decode (no-alloc rejects) #5372) — 20k-iteration decode fuzz (boundary safety)

Re-open and execute when a Docker-capable CI/dev environment is available (CI platform outage permitting — Actions has been startup_failure since 2026-09-07).