v0.2: crash and cancel drills

[svlandeg]

⚠️ Note that this runbook is now outdated and is only kept as reference.
Use the runbook for the current version instead. ⚠️

---

To test the robustness of the v0.2 infrastructure, both for an unexpected crash, as well as for a human-intended cancel.

cf. #23 for all setup and how to start a campaign. Make sure to run 0.2.2:

git checkout v0.2.2 -b v0.2.2

Cancel drill

Start a new campaign from a fresh branch. Once its first job is running:

curl -X POST http://127.0.0.1:8000/v0/campaigns/<campaign_id>/cancel \
  -H "Authorization: Bearer $FABERON_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"justification": "runbook cancel drill"}'

Expected: a campaign.cancel_requested event with actor human, the in-flight job runs to completion, then campaign.ended with stop_reason: "cancelled".

Note that an immediate scancel isn't implemented yet (which is why the in-flight job still is allowed to complete).

Crash drill

Start a new campaign from a fresh branch.
While an experiment's job is RUNNING, find and kill the server:

pkill -9 -f 'bin/faberon$'

(really brute force it like this, don't just hit Ctrl-C in the pane where you started faberon)

In the pane that originally started faberon, you'll see a short message "Killed".

Then restart it:

cd ~/git/autoresearch
faberon

Expected: the campaign resumes on its own. The running job is not resubmitted (submission is idempotent on the job name), the poll picks up where it left off, and the loop continues. Do it a second time right after a experiment.judged event to cover the between-experiments case.