Proposal: feat(orchestrator): per-task execution timeout annotation

[mvanhorn]

Motivation

Cancel already works: POST /api/pipeline_runs/{id}/cancel terminates a run and its live execution nodes. There is no way to say "this training step may run at most 2 hours" so a stuck GPU job stops itself.

Kubeflow Pipelines (Timeout), Airflow (execution_timeout), Prefect, and Vertex all have task deadlines. Tangle is used at Shopify scale specifically to save compute. A missing deadline is the inverse of content-based cache: cache saves reruns; timeout saves the run that will never finish.

The orchestrator already (1) merges default_task_annotations + pipeline_run.annotations + task_spec.annotations before launch, and (2) polls running containers. Timeout is a new annotation consumed on that poll, not a new subsystem.

This is not #339 / #338 (GCS hashing timeouts) and not open PR #327 (retry disrupted Job pods).

Proposed solution

Honor an optional annotation:

cloud-pipelines.net/orchestration/timeout_seconds = positive integer seconds

Already merged at launch time in this order: orchestrator defaults, then pipeline-run annotations, then task annotations. A task annotation overrides a run-level default. Missing or empty → current behavior (no deadline).

On each internal_process_one_running_execution pass, after the existing cancel-vote block, if the container is still PENDING/RUNNING and now - started_at (or created_at if never started) exceeds the timeout: set desired_state = TERMINATED with terminated_reason = timeout, then reuse the same terminate-and-skip-downstream path already used for cancel, including shared-container voting.

Default: no timeout. Existing cancel tests remain byte-identical.

Out of scope: queue-wait SLA, per-attempt retry, preemption, k8s activeDeadlineSeconds as the only implementation (generic orchestrator must work for Docker/HF/SkyPilot too).

Happy to open a draft PR if this annotation + poll shape looks right. Looking for maintainer signal before writing code.