Idea - Add a [Montitoring] extension to environment and / or job templates

[leongdl]

Lets go back and follow the RFC process to discuss adding monitoring extensions to OpenJD - I started at the incorrect place here: #133

Description
Add a monitoring mechanism for Environment templates that lets an Environment periodically health-check processes running within its Session. An Environment currently defines onEnter and onExit Actions for setup and teardown, and RFC 0008 proposes onWrapTaskRun for intercepting task execution. However, once an Action is running, the runtime has no visibility into whether the process inside the Environment is still healthy — the only escape hatch is the Action timeout, which can waste hours of compute on stalled workloads.

This is especially relevant for container-based Environments (Docker, Apptainer) where the wrapped process can stall silently — the container process hangs, the GPU driver crashes, or the application deadlocks — while the outer docker exec remains waiting. But the problem is general: any long-running Environment Action (container or not) benefits from periodic health checks.

This RFC proposes a way for Environment templates to define a monitoring Action that the runtime invokes periodically alongside the running Action. Possible designs include an onMonitor Action on , a monitorCommand field on individual Actions, or a Task.Monitor template variable. The goal is to detect stalled workloads early and fail the task immediately rather than waiting for the timeout.

Depends on RFC 0008 (onWrapTaskRun).

Concrete example: detecting a crashed container
A render task is running inside a Docker container via onWrapTaskRun. The GPU driver crashes and the container exits silently. The docker exec process in the wrap script hangs waiting on a dead container. Without monitoring, the runtime waits for the Action timeout (potentially hours).

With onMonitor, the runtime periodically runs a health check:

$ docker ps --filter "id=$DOCKER_CONTAINER_ID" --format '{{.Status}}'
Up 12 minutes # healthy — container is running
When the container crashes:

$ docker ps --filter "id=$DOCKER_CONTAINER_ID" --format '{{.Status}}'
# empty — container is gone, exit non-zero
$ echo $?
1

[leongdl]

One other application of a monitor command:
aws-deadline/deadline-cloud-for-3ds-max#238

When VRay licensing is not working, 3dsmax jobs fail to detect the failure. One mechanism can be in the adaptor to poll the render progress. Another maybe to have a job / enviornment level watchdog.