spawn sagemaker processing: MaxRuntimeInSeconds defaults to 86400 (24h); should support longer runtimes and document 5-day limit

[scttfrdmn]

Summary

spawn sagemaker processing hardcodes MaxRuntimeInSeconds = 86400 (24 hours) as both the default and effective cap. SageMaker Processing Jobs actually support up to 5 days (432,000 seconds). Long-running ML inference workloads (e.g. SAM ViT-H over 1655 Sentinel-2 tiles) regularly exceed 24 hours.

Current behavior

From internal/sagemaker/processing.go:

if cfg.MaxRuntimeSec == 0 {
    cfg.MaxRuntimeSec = 86400  // 24 hours default
}

From cmd/sagemaker.go:

processingCmd.Flags().IntVar(&procMaxRuntime, "max-runtime", 86400, ...)

Jobs that exceed 24 hours are stopped with Status: Stopped and no failure reason — easy to mistake for a completion.

Requested changes

1. Increase default to 172800 (48h) or document the 5-day max prominently
2. Add --max-runtime to the CLI help with the SageMaker limit documented (432000s = 5 days)
3. Consider job chaining: when a job hits the runtime limit, store progress checkpoint to S3 and restart from where it left off

Context

Discovered running Tutorial 01 (SAM fairy circle detection) for BuckAI Observatory. The job processes 1173 Sentinel-2 tile pairs on 2× ml.g5.4xlarge and takes ~30+ hours. Workaround: pass --stopping-condition MaxRuntimeInSeconds=172800 directly via the AWS CLI, but this should be a first-class spawn sagemaker option.

---

Migrated from spore-host/spore-host#351 — spawn issues belong on this repo.

[scttfrdmn]

Heads up before implementing: the referenced code doesn't exist in spore-host/spawn.

There is no spawn sagemaker command and no internal/sagemaker/ package in
this repo (spawn sagemaker --help → unknown command; no MaxRuntime/86400
SageMaker references anywhere in the Go source). So this isn't a one-line default
change — it would mean building the spawn sagemaker processing command itself.

Either:

• the SageMaker processing feature lives in a different tool/repo (and this issue
  should move there), or
• it was planned but never landed in spawn, in which case this is a net-new
  feature (spawn sagemaker processing with --max-runtime, the 5-day cap, etc.),
  not a quick fix.

Holding on this one pending clarification of where spawn sagemaker actually
lives. The other migrated quick-wins (#103, #104, #108, #110) reference real
spawn code and are being handled.