Add a random churn/soak scenario mode (neutron chaos)

[berendt]

Closes #15

Summary

Adds a random churn/soak mode for Neutron — a new neutron chaos subcommand that, instead of building a topology once, runs for a configured duration and uses the scenario plan as the upper-bound envelope of what may exist at any moment. For the whole runtime it continuously creates and deletes resources at random, configurable intervals and parallelism, keeping the cloud under fluctuating churn. The schedule is fully seeded and reproducible: the same scenario + seed + duration + interval/parallelism produce the identical create/delete sequence.

The implementation reuses Phase 1 machinery (plan, neutron wrappers + run tagging, executor pool/backoff, metrics, run records, tag-based cleanup + quota pre-check) rather than duplicating it.

Change set (in commit order)

1. 05e1c14 Export executor.WithRetry and metrics.ComputeLatency for reuse — exports the existing per-operation retry/backoff policy and latency-percentile computation so the churn engine reuses them instead of re-implementing the transient/conflict/quota backoff and percentile math.

2. d8cba54 Add neutron per-interface router-interface removal wrapper — adds RemoveRouterInterface, mirroring CreateRouterInterface (exactly one of subnetID/portID), because the generic Delete cannot detach an interface and the existing DetachRouterInterfaces removes them all at once. The engine needs to delete one interface at a time; the call is timed under KindRouterInterface and stays IsNotFound-classifiable so a repeat detach reads as success.

3. 04bcaed Add chaos config block to the scenario format — an optional chaos: block (duration, random interval range, parallelism cap, churn-ratio/target-fill knobs). It is a pointer so an absent block stays nil and apply/generate ignore it. A custom Duration YAML type decodes Go duration strings (30m, 200ms) under strict YAML, which yaml.v2 does not do natively. Validation runs only when the block is present and does not require duration (a CLI flag may supply it).

4. d50dd75 Add internal/chaos plan dependency graph and node model — Build turns a plan into a churn graph: each plan resource becomes a node whose Parents are resolved exactly as executor.Apply resolves references, so the engine can later create only nodes whose parents exist and delete only nodes whose dependents are gone. Each node binds per-kind create/delete closures over the typed Neutron wrappers (delete uses Delete for every kind except router interfaces, which use RemoveRouterInterface). Floating IPs are omitted when no external network is available, mirroring apply. No scheduling yet.

5. aeb9351 Add deterministic chaos engine, scheduler, and clock seam — a single scheduler goroutine draws every churn decision from a seeded RNG (random inter-tick delay, random fan-out, create-vs-delete pick biased by the churn-ratio/target-fill controller and constrained to currently valid candidates). Operations run concurrently through a bounded semaphore with executor.WithRetry, each waiting on its dependency operations before acquiring a slot so the pool cannot deadlock and no operation violates the dependency graph. A clock seam keeps the determinism test off the wall clock. The engine reports the decision log, churn counters, population series, the final live set, and per-time-bucket latency/error stats.

6. efd20f6 Add chaos stats to run record and report rendering — the run record gains an optional Chaos field (create/delete and cycle counts, live-population summary, target fill, per-bucket latency/error stats). The schema lives in the run package so report/status render it without importing the chaos engine. WriteTable appends a churn summary and per-bucket table when set; WriteJSON nests it under a chaos key. An apply record (Chaos nil) renders byte-identically to before.

7. 26508bb Add neutron chaos subcommand wired to the chaos engine — neutron chaos merges chaos config from defaults, the scenario chaos: block, and flags (flags override), validates the merged config before any cloud call, pre-checks quota against the full plan, runs the seeded churn for the duration, and records the run with its churn stats. On a clean finish it tears the topology down by tag and reports a leak check; an interrupt or --no-cleanup leaves resources in place and prints the cleanup hint. buildPlanFromFlags now also returns the parsed scenario (apply/generate ignore the new return).

8. 412f985 Document the chaos churn mode in the README — documents the subcommand and flags, the chaos: scenario block, the churn stats and leak check, the deterministic scheduler / envelope ceiling / controller / teardown policy, the new package in the layout, and a Phase 1 roadmap entry.

Decisions taken against the issue's open questions

The issue left several decisions open; this branch resolves them as:

• Subcommand name: chaos (issue listed soak / churn / random / chaos). The scenario block is chaos: rather than the issue's example soak:.
• Config location: both a chaos: scenario block and CLI flags, with flags overriding.
• Churn control: both knobs — churn_ratio and a target_fill controller.
• Teardown default: auto-cleanup by tag on a clean finish (the issue leaned toward leaving resources for an explicit cleanup --run, but listed auto-cleanup as a valid alternative). --no-cleanup and interrupts leave resources in place with a cleanup hint, matching the safe path.
• Quota pre-check: retained — PrecheckQuota runs against the full plan up front, since the envelope is the worst case.

Reviewer note

The working tree has uncommitted changes to internal/chaos/engine.go and internal/chaos/engine_test.go (a missingParentID helper, a launch signature taking ctx/offset, and validation tweaks) that are not part of any commit and therefore not in this PR. This PR reflects only the committed state; those changes were left untouched by this publish step.

---

Implemented by planwerk-review 495ffd8-dirty with Claude:claude-opus-4-8