spored: support email/webhook notifications when job completes (SPAWN_COMPLETE touched or TTL fires)

[scttfrdmn]

Summary

spored monitors instance lifecycle but has no way to notify users when a workload finishes. Users who close their laptops after launching a job have no way to know when it completes other than polling spawn status.

Use cases

1. Job completes — user touches /tmp/SPAWN_COMPLETE, spored executes --on-complete stop/terminate, then sends a notification
2. TTL fires — instance terminates due to deadline; notification includes job name, final status, and any --pre-stop output
3. Idle timeout — instance stops due to inactivity (possible crash/early completion); notification warns user to check logs

Proposed flags

spawn launch my-job \
    --notify email:user@example.com \
    --notify webhook:https://hooks.slack.com/... \
    --ttl 4h --on-complete stop

Or a global config:

spawn config set notify email:user@example.com

Implementation options

1. SNS — spored creates/reuses an SNS topic, subscribes the endpoint, publishes on state change. Simple, AWS-native, works with email/SMS/webhook.
2. Direct email via SES — spored sends email directly via SES when available.
3. Webhook — spored POSTs a JSON payload to the configured URL on completion.

The SNS approach is cleanest since it composes with EventBridge for more complex routing.

Message format

Should be human-readable, not raw JSON:

BuckAI job "my-job" completed on i-0abc123def456
Status: stopped (SPAWN_COMPLETE signal received)
Runtime: 2h 34m
Instance: r7i.4xlarge in us-east-2

To check outputs:
  spawn connect my-job

Context

Discovered while building Tutorial 01 (SageMaker jobs) for BuckAI Observatory. We implemented a workaround using EventBridge + SNS from outside spawn, but it only works for SageMaker-specific events. For EC2 workloads (Tutorials 02–04), users must poll manually. Native spored support would make this first-class for all workload types.

---

Migrated from spore-host/spore-host#346 — spawn issues belong on this repo.

[scttfrdmn]

Partial: the event hook already exists, the transports don't

Checked what spored has today against this ask.

Events — already covered. spored has a Notifier (pkg/agent/notifier.go) and already fires Notify(eventType, detail) at exactly the three points this issue lists:

#107 use case	already fires?	call site
Job completes (SPAWN_COMPLETE)	✅ "completion"	agent.go:1045
TTL fires	✅ "ttl_expired" (+ "ttl_warning" at T-5m)	agent.go:331/342
Idle timeout	✅ "idle_stopped" / "idle_hibernated" (+ "idle_warning")	agent.go:382/385/399

So the lifecycle plumbing — when to notify — is done and in the right place.

Transports — not covered. Notify has exactly one sink: the spore-bot Lambda (Slack), gated on NotifyURL + a Slack workspace ID + the spore-bot infra. There is no email, SNS, or generic webhook path (there's a literal // Future enhancement: Support webhooks, email, SNS marker at agent.go:1019). So the proposed --notify email:… / --notify webhook:… don't exist yet, and a user without spore-bot/Slack gets nothing.

So, to the submitter: does the existing Slack path solve your case? If you're already on spore-bot/Slack, completion/TTL/idle notifications work today — worth confirming. If you need email or a raw webhook (the BuckAI tutorial case sounds like it does, since you used external EventBridge+SNS), that's the real gap.

Scoping if we build it. The architecture is already the right shape — one internal Notify(eventType, detail) seam — so this is a transport-fanout change, not a rearchitecture:

1. Generalize Notifier to a list of sinks behind a small interface; keep the spore-bot sink as one implementation.
2. Add webhook (POST the existing notifyRequest JSON to an arbitrary URL — simplest, no AWS deps) and SNS (publish to a topic — composes with EventBridge, AWS-native; SES/email can ride SNS subscriptions rather than spored sending mail directly).
3. Parse --notify <scheme>:<target> (repeatable) + a spawn config set notify … default into config/tags spored reads — same tag-driven pattern as the other lifecycle settings.
4. Enrich the completion detail (currently empty at agent.go:1045) with the human-readable runtime/status/type payload this issue mockups — so non-Slack sinks carry it too.

Leaving open as an enhancement. The webhook sink is the cheapest first slice and unblocks the non-Slack case; SNS/email can follow.

[scttfrdmn]

Submitter answer: no, the Slack path doesn't cover us — we need email + generic transports

Confirming the requirement, since the triage comment asked.

The Slack/spore-bot path does not solve the motivating case. Prof. Moortgat (BuckAI Observatory, OSU) isn't on spore-bot or any Slack workspace — he's a researcher who launches a job, closes his laptop, and wants an email when it's done. That's the whole reason Tutorial 01 grew an external EventBridge→SNS→email workaround (common/notify.py): spored had no email sink, so we bolted one on outside spawn. It only catches SageMaker job-state events, so the EC2 tutorials (02–04) still leave the user polling spawn status by hand.

Priority order for us:

1. Email — the actual end-user need. Non-technical researcher, no chat tooling, just wants "your job finished."
2. Generic webhook — useful for us as integrators (wire completion into our own automation / dashboards without standing up spore-bot).
3. SNS — nice-to-have; valuable because it composes with EventBridge and because email/SMS can ride SNS subscriptions rather than spored sending mail directly.

So the webhook-first slice you proposed is the right cheapest start, but email is the one that actually unblocks the BuckAI case — whether that's delivered via SNS-subscription or SES directly doesn't matter to us, only that --notify email:prof@osu.edu works without any Slack/spore-bot dependency.

If native email lands, we delete common/notify.py entirely and Tutorial 01's notification story becomes a single --notify flag for all four tutorials instead of SageMaker-only EventBridge plumbing.