Fail closed when session watchdog exits

.changes/unreleased/+controlled-session-cleanup-containment-loss.yaml

@@ -0,0 +1,2 @@
+kind: Changed
+body: Report premature controlled-session watchdog exit as cleanup-containment loss, fail the session closed without misreporting Docker observation, and pin watchdog cleanup to the session's verified Docker endpoint.

docs/CONTROLLED_SESSION_DESIGN.md

@@ -56,14 +56,16 @@ summary: Capability-scoped execution sessions that inherit Reploy's global conta
   private channel. A workload that starts before a later startup step fails is
   still terminated and its output is finalized through the same barrier.
   Before creating any session resource, the planned controller, workload, and
-  private-channel ownership plus the session, lease, and boot identities are
-  now durably recorded in the existing live-run state. Reploy monotonically
+  private-channel ownership plus the session, lease, boot, and exact verified
+  local Docker endpoint identities are now durably recorded in the existing
+  live-run state. Reploy monotonically
   fills each exact full container ID after Docker creates it, and both IDs are
   durable before either process starts. Verified cleanup removes that record;
   failed or unverifiable partial-preparation cleanup retains it. Reploy now
   derives the immutable watchdog cleanup manifest from the complete recorded
   ownership before startup. It names only the exact containers and private
-  channel, carries the host boot identity,
+  channel, carries the host boot identity and the same pinned local Docker
+  endpoint used to create both containers,
   represents the currently absent lease networks and volumes as empty arrays,
   and omits protocol authority. The watchdog is now launched from the same
   Reploy executable after that manifest is frozen and before either container
@@ -78,15 +80,20 @@ summary: Capability-scoped execution sessions that inherit Reploy's global conta
   verifies the same resources once more before it exits, and the parent waits
   boundedly for that exit. The child has no listener and accepts no later
   resource selection. Because parent and child are the same executable, this
-  internal manifest has no independent schema-version marker. Next-operation
+  internal manifest has no independent schema-version marker. A ready
+  watchdog's premature exit is observed by the attached supervisor. Before
+  activation it fails startup; after activation it latches the distinct
+  `cleanup-containment-lost` cause, emits only a safe structured diagnostic
+  while the controller transport remains usable, and starts ordinary bounded
+  teardown without misreporting Docker observation as lost. Next-operation
   restart reconciliation is also implemented: after a prior host boot or an
   abandoned current-boot owner lease, Reploy discovers any container whose
   full ID was not recorded by its frozen name, verifies every exact ownership
   label, removes and verifies both containers and the private channel under one
   bounded cleanup attempt, and retains incomplete ownership for a later retry.
-  A watchdog-owned retry loop while Docker remains unavailable is still a
-  later ownership phase, and controlled-session networking remains a later
-  phase.
+  A watchdog-owned retry loop while Docker remains unavailable and a bounded
+  durable post-crash diagnostic receipt are still later ownership phases, and
+  controlled-session networking remains a later phase.
 - Initial runtime: Linux containers under Docker
 - Motivating clients: OmegaFlow recording, sandboxed AI agents, security
   inspection, and untrusted-code execution
@@ -1095,13 +1102,15 @@ preparing -> active -> terminating -> terminated
 
 The first accepted termination cause is latched and never rewritten. Causes
 include controller-requested termination, workload exit, host cancellation,
-controller loss, Docker-observation loss, and startup failure. Later events
-remain diagnostic observations. Workload status, workload-output-finalization
-status, runtime-observation status, controller finalization status, and
-pre-delivery cleanup success are reported separately in the session result, so
-a late observation or cleanup failure can fail the operation without hiding
-its original cause. Controller exit and delivery-tail cleanup are reported
-separately by the invoking host operation after teardown.
+controller loss, Docker-observation loss, cleanup-containment loss, and startup
+failure. Cleanup-containment loss means the ready session watchdog exited
+before verified disarm; it does not claim Docker observation was lost. Later
+events remain diagnostic observations. Workload status,
+workload-output-finalization status, runtime-observation status, controller
+finalization status, and pre-delivery cleanup success are reported separately
+in the session result, so a late observation or cleanup failure can fail the
+operation without hiding its original cause. Controller exit and delivery-tail
+cleanup are reported separately by the invoking host operation after teardown.
 
 Channel closure is never successful completion. A controller granted the
 `complete` operation must explicitly send `complete` after receiving
@@ -1166,7 +1175,17 @@ Host Reploy starts one short-lived watchdog for each live controlled session.
 It first creates inert Docker resources and durably records their exact
 identities. Before starting either container, it passes the watchdog an
 immutable cleanup manifest containing the exact lease, container, network,
-volume, and host boot identities. The attached operation retains one
+volume, host boot, and verified local Docker endpoint identities. Controller
+and workload creation, attachment, start, observation, resize, stop, and
+cleanup use that one pinned endpoint; the watchdog and next-operation
+reconciliation reuse it instead of resolving the mutable active Docker context.
+Legacy `live-run-queue-v1` ownership written before endpoint pinning remains
+readable so abandoned resources do not become unrecoverable after upgrade.
+Because that state cannot identify the historical daemon, each legacy cleanup
+attempt resolves one currently selected verified local endpoint and pins every
+inspection and removal in that attempt to it; exact ownership labels still
+gate removal. Newly recorded controlled sessions always require the durable
+endpoint. The attached operation retains one
 end of a private parent pipe. A crash during inert resource creation leaves no
 untrusted code running and is handled by ordinary next-operation
 reconciliation.
@@ -1179,6 +1198,15 @@ later resource selection, and exits after verified cleanup. Although its
 underlying Docker connection has ordinary trusted-host authority, its code path
 is limited to the immutable resource set.
 
+The attached supervisor also observes the watchdog process itself. If a ready
+watchdog exits before workload activation, startup fails before workload code
+runs. If it exits while the workload is active, Host Reploy latches
+`cleanup-containment-lost`, sends a bounded structured diagnostic while the
+controller transport remains healthy, and begins normal teardown. The original
+termination cause still wins if termination had already begun. Verified host
+cleanup may complete the durable ownership record even though containment loss
+makes the session result unsuccessful.
+
 If Docker is unavailable, the watchdog retries until Docker returns or the host
 reboots. If both the attached operation and watchdog are killed, durable labels
 and deployment-scoped live-run state let the next locked Reploy operation
@@ -1247,6 +1275,16 @@ Diagnostics identify which operation failed, what Reploy attempted, whether
 the session channel or Docker lifecycle was observed, what cleanup ran, and the
 safe next action.
 
+The target crash-containment work adds a bounded durable incident receipt for
+failures that outlive the attached Host Reploy process. It records only
+allowlisted lifecycle, observation, exit-status, cleanup, and recovery facts.
+It does not duplicate PTY output, environment names or values, secrets,
+arbitrary container logs, or unrestricted Docker output. Host Reploy creates
+the exact private receipt target before startup and gives the watchdog only the
+narrow write authority needed for that target; the child does not select an
+arbitrary state path. A parent-liveness EOF proves parent loss but cannot by
+itself distinguish `SIGKILL`, an OOM kill, or another abrupt process death.
+
 ## Resource and Timeout Policy
 
 Controlled sessions have explicit limits for:

internal/controlledsession/lifecycle.go

@@ -31,6 +31,7 @@ const (
 	ObservationHostCancelV1                        ObservationKindV1 = "host-cancel"
 	ObservationControllerLostV1                    ObservationKindV1 = "controller-lost"
 	ObservationRuntimeObservationLostV1            ObservationKindV1 = "runtime-observation-lost"
+	ObservationCleanupContainmentLostV1            ObservationKindV1 = "cleanup-containment-lost"
 	ObservationStartupFailureV1                    ObservationKindV1 = "startup-failure"
 	ObservationWorkloadOutputsFinalizedV1          ObservationKindV1 = "workload-outputs-finalized"
 	ObservationWorkloadOutputsPublishedV1          ObservationKindV1 = "workload-outputs-published"
@@ -210,6 +211,11 @@ func (machine *MachineV1) Observe(observation ObservationV1) (TransitionV1, erro
 		// surfaces. Keep that barrier pending until the supervisor explicitly
 		// reports failed closure or its bounded finalization deadline expires.
 		machine.finalizePreActivationOutputsForRuntimeObservationLossLocked(observation.Reason)
+	case ObservationCleanupContainmentLostV1:
+		if err := validateCauseObservationV1(observation); err != nil {
+			return transition, err
+		}
+		machine.latchLocked(CauseCleanupContainmentLostV1, &transition)
 	case ObservationStartupFailureV1:
 		if observation.WorkloadStatus != nil || observation.Finish != nil ||
 			(observation.WorkloadOutputPending && observation.WorkloadOutputFinalizationStatus != nil) {

internal/controlledsession/lifecycle_test.go

@@ -537,6 +537,16 @@ func TestLifecycleOutputFinalizationExpiryAcceptsLateWorkloadExit(t *testing.T)
 				return err
 			},
 		},
+		{
+			name:  "cleanup containment lost",
+			cause: CauseCleanupContainmentLostV1,
+			start: func(machine *MachineV1) error {
+				_, err := machine.Observe(ObservationV1{
+					Kind: ObservationCleanupContainmentLostV1, Reason: "watchdog exited",
+				})
+				return err
+			},
+		},
 	}
 
 	for _, test := range tests {
@@ -706,6 +716,7 @@ func TestLifecycleFirstAcceptedCauseWinsConcurrentRace(t *testing.T) {
 		{Kind: ObservationHostCancelV1, Reason: "host interrupted"},
 		{Kind: ObservationControllerLostV1, Reason: "channel closed"},
 		{Kind: ObservationRuntimeObservationLostV1, Reason: "docker unavailable"},
+		{Kind: ObservationCleanupContainmentLostV1, Reason: "watchdog exited"},
 		{Kind: ObservationWorkloadExitV1, WorkloadStatus: &ProcessStatusV1{Kind: ProcessStatusExitedV1, Code: &code}},
 	}
 	var wait sync.WaitGroup

internal/controlledsession/model.go

@@ -10,6 +10,7 @@ const (
 	CauseHostCancelV1             TerminationCauseV1 = "host-cancel"
 	CauseControllerLostV1         TerminationCauseV1 = "controller-lost"
 	CauseRuntimeObservationLostV1 TerminationCauseV1 = "runtime-observation-lost"
+	CauseCleanupContainmentLostV1 TerminationCauseV1 = "cleanup-containment-lost"
 	CauseStartupFailureV1         TerminationCauseV1 = "startup-failure"
 )
 
@@ -242,7 +243,8 @@ func validateRequiredSafeTextV1(field string, value string) error {
 func validTerminationCauseV1(cause TerminationCauseV1) bool {
 	switch cause {
 	case CauseControllerTerminateV1, CauseWorkloadExitV1, CauseHostCancelV1,
-		CauseControllerLostV1, CauseRuntimeObservationLostV1, CauseStartupFailureV1:
+		CauseControllerLostV1, CauseRuntimeObservationLostV1,
+		CauseCleanupContainmentLostV1, CauseStartupFailureV1:
 		return true
 	default:
 		return false

internal/deploy/controlled_session_cleanup_manifest.go

@@ -20,6 +20,7 @@ import (
 type ControlledSessionCleanupManifest struct {
 	LiveRunID        string                                `json:"live_run_id"`
 	BootSession      string                                `json:"boot_session"`
+	DockerEndpoint   string                                `json:"docker_endpoint"`
 	ChannelDirectory string                                `json:"channel_directory"`
 	Controller       ControlledSessionContainerOwnershipV1 `json:"controller"`
 	Workload         ControlledSessionContainerOwnershipV1 `json:"workload"`
@@ -31,11 +32,12 @@ type ControlledSessionCleanupManifest struct {
 // from the exact durable ownership record. The session handle is deliberately
 // omitted because cleanup does not need protocol authority.
 func ControlledSessionCleanupManifestFromOwnership(ownership ControlledSessionOwnershipV1) (ControlledSessionCleanupManifest, error) {
-	if err := validateControlledSessionOwnershipV1(ownership); err != nil {
+	if err := validateCurrentControlledSessionOwnershipV1(ownership); err != nil {
 		return ControlledSessionCleanupManifest{}, fmt.Errorf("controlled-session cleanup manifest ownership: %w", err)
 	}
 	manifest := ControlledSessionCleanupManifest{
 		LiveRunID: ownership.LiveRunID, BootSession: ownership.BootSession,
+		DockerEndpoint:   ownership.DockerEndpoint,
 		ChannelDirectory: ownership.ChannelDirectory,
 		Controller:       ownership.Controller, Workload: ownership.Workload,
 		Networks: []string{}, Volumes: []string{},
@@ -53,6 +55,9 @@ func ValidateControlledSessionCleanupManifest(manifest ControlledSessionCleanupM
 	if err := validateBootSessionIDV1(manifest.BootSession); err != nil {
 		return fmt.Errorf("controlled-session cleanup manifest: %w", err)
 	}
+	if err := validateControlledSessionDockerEndpointV1(manifest.DockerEndpoint); err != nil {
+		return fmt.Errorf("controlled-session cleanup manifest: %w", err)
+	}
 	if !filepath.IsAbs(manifest.ChannelDirectory) || filepath.Clean(manifest.ChannelDirectory) != manifest.ChannelDirectory || !safeRecoveryIdentity(manifest.ChannelDirectory) {
 		return fmt.Errorf("controlled-session cleanup manifest channel directory must be a clean absolute path")
 	}

internal/deploy/controlled_session_cleanup_manifest_test.go

@@ -16,7 +16,7 @@ func TestControlledSessionCleanupManifestDerivesExactDurableOwnership(t *testing
 		t.Fatal(err)
 	}
 	if manifest.LiveRunID != ownership.LiveRunID || manifest.BootSession != ownership.BootSession ||
-		manifest.ChannelDirectory != ownership.ChannelDirectory || manifest.Controller != ownership.Controller ||
+		manifest.DockerEndpoint != ownership.DockerEndpoint || manifest.ChannelDirectory != ownership.ChannelDirectory || manifest.Controller != ownership.Controller ||
 		manifest.Workload != ownership.Workload || len(manifest.Networks) != 0 || len(manifest.Volumes) != 0 {
 		t.Fatalf("cleanup manifest = %#v", manifest)
 	}
@@ -33,6 +33,15 @@ func TestControlledSessionCleanupManifestDerivesExactDurableOwnership(t *testing
 	}
 }
 
+func TestControlledSessionCleanupManifestRejectsRemoteDockerEndpoint(t *testing.T) {
+	ownership := controlledSessionOwnershipFixtureV1(t.TempDir(), "run-0000000000000001", "reploy/env/workload:g-current")
+	ownership.BootSession = "boot-session"
+	ownership.DockerEndpoint = "tcp://builder.example:2376"
+	if _, err := ControlledSessionCleanupManifestFromOwnership(ownership); err == nil || !strings.Contains(err.Error(), "local unix or npipe") {
+		t.Fatalf("remote Docker endpoint error = %v", err)
+	}
+}
+
 func TestControlledSessionCleanupManifestRejectsInvalidDurableOwnership(t *testing.T) {
 	ownership := controlledSessionOwnershipFixtureV1(t.TempDir(), "run-0000000000000001", "reploy/env/workload:g-current")
 	ownership.BootSession = "boot-session"

internal/deploy/live_run_queue.go

@@ -9,6 +9,7 @@ import (
 	"io"
 	"path/filepath"
 	"regexp"
+	"strings"
 
 	"github.com/omry/reploy/internal/canonical"
 )
@@ -66,6 +67,7 @@ type ControlledSessionOwnershipV1 struct {
 	LiveRunID        string                                `json:"live_run_id"`
 	BootSession      string                                `json:"boot_session"`
 	SessionHandle    string                                `json:"session_handle"`
+	DockerEndpoint   string                                `json:"docker_endpoint,omitempty"`
 	ChannelDirectory string                                `json:"channel_directory"`
 	Controller       ControlledSessionContainerOwnershipV1 `json:"controller"`
 	Workload         ControlledSessionContainerOwnershipV1 `json:"workload"`
@@ -289,6 +291,11 @@ func validateControlledSessionOwnershipV1(ownership ControlledSessionOwnershipV1
 	if !controlledSessionHandlePatternV1.MatchString(ownership.SessionHandle) {
 		return fmt.Errorf("session handle must use session- followed by 64 lowercase hexadecimal characters")
 	}
+	if ownership.DockerEndpoint != "" {
+		if err := validateControlledSessionDockerEndpointV1(ownership.DockerEndpoint); err != nil {
+			return err
+		}
+	}
 	if !filepath.IsAbs(ownership.ChannelDirectory) || filepath.Clean(ownership.ChannelDirectory) != ownership.ChannelDirectory || !safeRecoveryIdentity(ownership.ChannelDirectory) {
 		return fmt.Errorf("channel directory must be a clean absolute path")
 	}
@@ -307,6 +314,24 @@ func validateControlledSessionOwnershipV1(ownership ControlledSessionOwnershipV1
 	return nil
 }
 
+func validateCurrentControlledSessionOwnershipV1(ownership ControlledSessionOwnershipV1) error {
+	if ownership.DockerEndpoint == "" {
+		return fmt.Errorf("Docker endpoint must be recorded for a new controlled session")
+	}
+	return validateControlledSessionOwnershipV1(ownership)
+}
+
+func validateControlledSessionDockerEndpointV1(endpoint string) error {
+	if !safeRecoveryIdentity(endpoint) {
+		return fmt.Errorf("Docker endpoint must be nonempty safe text")
+	}
+	scheme, _, found := strings.Cut(endpoint, ":")
+	if !found || (strings.ToLower(scheme) != "unix" && strings.ToLower(scheme) != "npipe") {
+		return fmt.Errorf("Docker endpoint must be a local unix or npipe endpoint")
+	}
+	return nil
+}
+
 func validateControlledSessionContainerOwnershipV1(ownership ControlledSessionContainerOwnershipV1, role string) error {
 	if err := validateControlledSessionContainerOwnershipStateV1(ownership, role); err != nil {
 		return err

internal/deploy/live_run_queue_file.go

@@ -169,7 +169,7 @@ func (lock *OperationLock) RecordControlledSessionOwnershipV1(ownership Controll
 		return ControlledSessionOwnershipV1{}, fmt.Errorf("controlled session workload generation does not match admitted live run %q", ownership.LiveRunID)
 	}
 	ownership.BootSession = admitted.BootSession
-	if err := validateControlledSessionOwnershipV1(ownership); err != nil {
+	if err := validateCurrentControlledSessionOwnershipV1(ownership); err != nil {
 		return ControlledSessionOwnershipV1{}, err
 	}
 	insert := sort.Search(len(queue.ControlledSessions), func(index int) bool {
@@ -230,7 +230,7 @@ func mergeControlledSessionOwnershipV1(
 	if err != nil {
 		return ControlledSessionOwnershipV1{}, err
 	}
-	if err := validateControlledSessionOwnershipV1(merged); err != nil {
+	if err := validateCurrentControlledSessionOwnershipV1(merged); err != nil {
 		return ControlledSessionOwnershipV1{}, err
 	}
 	return merged, nil