Repository navigation
Experiments
Documentation index · Source repository
Use this guide after Swarm deployment and Agent capacity setup. A saved scenario is reusable input; a run is one execution; a batch groups repetitions; a retry is a new attempt at an unfinished iteration.
- Open Run experiment in the Dashboard. Load a saved scenario or paste YAML; scenario reference defines the fields.
- Use Validate and inspect Agent availability. Validation checks the input, not runtime capacity or connectivity.
examples/monitoring.yamlis a small first experiment. - Set Runs, then choose Run. Follow Experiment progress and inspect the resulting records in Saved results.
You can submit Run experiment → Run repeatedly while another experiment runs. Each accepted submission joins the Controller's FIFO queue; all repetitions of one submission run before the next submission. Retry, continuation and Add runs also join this queue. The submitted YAML, seeds and inherited result group are captured at admission. Closing the browser does not cancel accepted work.
Experiment progress shows how many submissions are ahead of each waiting batch. Its stop control cancels only that submission, including its repetitions. Stopping or failing the active batch cancels its own remaining repetitions, then the Controller advances to the next submission after Peer cleanup and final telemetry collection. A successful single run's retained Peers are also removed before another submission starts. If cleanup cannot be confirmed, the next submission stays queued with Waiting for previous Peer cleanup and the Controller retries every three seconds after each failed attempt. Canceling that waiter does not bypass cleanup for later submissions.
Waiting does not advance a run's start time, phase timeout or schedule clock. Controller restarts retain the reserved results as interrupted; use Retry experiment / Retry from run to recover them manually. The Controller does not automatically replay the queue after restart.
The experiment network preparation API initializes a fresh Peer overlay before every run, including repetitions, retries, continuations and appended runs. It waits for Peer cleanup acknowledgments, and starts the experiment clock only after all Agents accept the new network. Preparing network is outside scenario phase and schedule time. Initialization failure prevents the first phase and follows the normal failure/queue rules. The supplied Swarm stack enables initialization. Existing deployments need the updated image and stack; see automatic network initialization.
In Run experiment, paste the YAML and set Runs beside Run to an integer from 1 to 100. The Controller queues all iterations and executes them sequentially, even if you close the browser. Each iteration gets a unique run ID, a separate result directory, and batchId, iteration, and repetitions in its metadata.
For more than one iteration, the Controller cancels or drains background jobs according to the scenario's exit policy, fences and removes that iteration's Peers, and refreshes Agent state before starting the next iteration. An execution or cleanup failure cancels all remaining iterations of that batch. Stop batch on any running or queued member cancels that submission's active iteration and remaining repetitions. Other submissions keep their place in the queue.
The scenario YAML is unchanged in every iteration. An explicit nonzero seed is reused; zero or an omitted seed creates a new recorded seed per iteration. Reusing a seed repeats sampling inputs, but Docker timing, eligible populations, and network execution can still differ. Queues are not automatically resumed after a Controller restart. Retained queued or running records are displayed as interrupted.
For automation, see run submission.
In Saved results, choose Add runs on a completed group, enter Additional runs, review the new total, then select Add and start. A group with 10 completed runs plus 10 additions keeps its batch ID and starts Run 11 of 20 through Run 20 of 20. The saved YAML and seed policy are reused; original results, individual notes and the group note remain available. A completed single run can also be expanded into a group. The total group limit remains 100 runs.
Every current iteration must be present and completed. Active execution/finalization, downloads, and run/batch analysis block admission; unfinished groups should use the existing retry/continuation controls first. The Controller cleans up previous Peers before the new sequence and applies the local free-space guard. Stop or failure cancels the new remainder; Retry from run retries unfinished additions without repeating completed iterations. Closing the browser does not stop accepted work.
After all additions finish, choose Analyze batch mean to recompute statistics over the entire group. Adding runs invalidates the previous membership cache: 10 + 10 yields 20 contributing runs for metrics defined in all 20 results, with equal weight per run. Previous attempts remain excluded. When adding from the Completed filter, the UI switches status to All so the extended group stays visible.
Initial submissions, retries, continuations and extensions all use a local result-groups/<batch-id>/batch.json record. Preparation interrupted before commit is rolled back; committed runs survive a restart as interrupted and can be retried. Refresh results before repeating a failed/timed-out request. Include result-groups/ in backups. Original experiment.json files and old analysis snapshots retain their historical totals; the current results list and fresh batch analysis use the committed group total.
The browser retains an initial submission key until a successful response. Retrying the same input after a lost response reuses the accepted group rather than creating another one. For automation, supply and reuse Idempotency-Key as described in the API guide.
Current-attempt ownership also survives individual result deletion. Deleting the newest retry leaves a missing iteration; an older failed attempt stays under Previous attempts and cannot silently become current again. Repair or restore the missing result before extending that group. A corrupt group journal makes that group unavailable and is reported in Controller logs, while other groups remain usable.
Use the active experiment’s stop control or Stop batch for a repetition group. Accepted cancellation displays Stopping… while jobs and Peers are cleaned up. Wait for the live state to settle; repeated clicks are not required. Stopping an active batch cancels its running and queued members. A Controller shutdown or crash has separate cleanup limits; see failures and shutdown.
When a Peer shuts down, its HTTP publication requests receive the same cancellation signal. Requests waiting for the publication gate or topic readiness stop promptly, before the final telemetry drain.
Each accepted sequence has a new executionId, including continuation that reuses run IDs. The UI sends that identity with a stop request and keeps Stopping… across dashboard refreshes. A delayed stop for an earlier execution cannot cancel a later retry or extension; API clients must also send X-KPL-Execution.
Execution state, Peer cleanup and data collection are separate. canceled does not prove cleanup succeeded. Saved results show cleanup errors and incomplete/unverified telemetry. A successfully retained single-run network has cleanupState: retained and dataState: collecting; its logs may continue to change.
After a restart, a saved run that still says running or queued is displayed as interrupted; its original metadata is preserved in the ZIP. This is a display status, not evidence that its Peers have stopped. Saved results do not automatically restore live experiment state, resume execution, or replay live counters. ZIP metrics are rebuilt from the retained log. An unreadable metadata file is displayed as unreadable; inspect the Controller logs and stored files before retrying.
If Run 4 of 10 failed after Runs 1–3 completed, choose the behavior deliberately:
| Action | What runs next | Previous attempts |
|---|---|---|
| Continue remaining | Only never-started Runs 5–10 | Run 4 stays failed; no replacement to reach ten successes |
| Retry from run 4 | Unfinished Runs 4–10, starting each at its first phase | Completed runs stay; retried iterations get new run IDs and retain earlier logs |
These are manual recovery operations, not phase checkpoints or automatic failover.
In Saved results, use Continue remaining (N) on a stopped series to resume its never-started runs. Previously completed, failed, and otherwise attempted runs are preserved and are not repeated. The original batch ID, run IDs, iteration numbers, total repetitions, saved YAML, and per-run seeds are retained; for example, a failure at Run 2 of 5 continues with Runs 3–5. This does not add replacement runs to reach five successes. Batch means continue to include only completed runs.
Continuation also works from saved results after a Controller restart. It requires a failed run, or an interrupted run that had started, and retained unstarted members. The Controller retries cleanup on the Agents recorded as owners before starting the remainder, even before they re-register after a Controller restart. An unreachable owner blocks new work. Legacy results without ownership records use currently registered Agents and cannot establish cleanup on unknown hosts. If cleanup fails, no remaining run starts; its error is shown and Continue remaining can be retried. A later experiment failure stops the remainder again. Concurrent continuation requests, active batches, downloads, and running analysis jobs block admission. Stop batch also cancels a continuation during cleanup. Closing the browser does not cancel accepted work.
Saved results → Retry from run N (count) restarts saved unfinished iterations, including the failed/interrupted iteration itself. Completed runs keep their results. For a failed single experiment, use Retry experiment. The Controller first removes old Peers, then starts each selected iteration from its first phase with the original scenario, iteration number and seed. If cleanup fails, the new runs remain canceled and can be retried after resolving the cause. Old logs remain under Previous attempts and are excluded from batch progress and mean analysis. Continue remaining retains its earlier behavior: skip attempted runs and execute only the never-started remainder.
Both modes wait for prior Peer cleanup before new work starts. Resolve unreachable Agents or cleanup errors before retrying. Active execution, finalization, downloads, and active run/batch analysis can block recovery. See the recovery API for admission rules. Record which attempts enter a comparison following the Reporting principles.
Experiment progress shows running runs first, queued runs in submission order next, and terminal runs last. Repetitions within each queued submission stay in numeric order. Saved results serves a different task: iterations within a batch stay in numeric Run n of M order, with previous attempts separate. See result browsing for filters, exports, and deletion.
Experiment progress shows Est. finish and approximate remaining time for each running or queued Run. A summary above the cards shows Group est. finish for each active repeated series. Times use the browser's local time zone; remaining time is calculated by the Controller, including earlier Runs in the same group's queue. Submissions behind another batch, or waiting for previous Peer cleanup, show Estimating… until they can start; no absolute finish is promised while that preceding work is unresolved.
The initial Scenario estimate combines waits, repeats and sampled intervals while accounting for overlapping background jobs, wait-jobs, stop-all, and onExit. Readiness barriers initially use their configured timeout allowance. Request counts use the configured maxima; actual eligible publishers/leavers may be fewer. Node startup, capacity waits, request processing and cleanup are not known in advance. As phases finish, their actual boundaries replace predictions. Completed successful Runs in the same active group supply average phase and total times, including cleanup, for later estimates; Based on N completed runs identifies this correction.
These are estimates, not deadlines. Estimating… means there is not enough timing information. If a Run exceeds its estimate, Taking longer than estimated appears and later queued Run/group estimates move forward. Stopping or finishing a Run removes its forecast. Estimates do not schedule an automatic stop and are not retained in saved results.
Korean translation: 한국어.
Home · Documentation guide · kpl-v3 source
Commands run from the kpl-v3 source checkout.