Skip to content

Supervise controlled-session lifecycle #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
omry wants to merge 1 commit into from
Open

Supervise controlled-session lifecycle #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 29 additions & 14 deletions docs/CONTROLLED_SESSION_DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,17 @@ summary: Capability-scoped execution sessions that inherit Reploy's global conta
bounded write admission path, and reports request, backpressure, and
disconnect failures without owning the channel or containers. A failed event
write makes the framed transport terminal so a later event cannot be appended
to a potentially partial frame. Full lifecycle orchestration and
controlled-session networking remain later slices.
to a potentially partial frame. The attached host lifecycle supervisor is
also implemented: it prepares inert resources, starts the controller before
the workload, activates only after workload setup succeeds, serializes
controller requests through the lifecycle machine, latches the first
termination cause, stops and independently observes the workload, finalizes
PTY output before the terminal result, waits boundedly for controller
completion and result acknowledgement, and removes both containers and the
private channel. A workload that starts before a later startup step fails is
still terminated and its output is finalized through the same barrier.
Crash watchdogs and restart reconciliation remain the next ownership phase;
controlled-session networking remains a later phase.
- Initial runtime: Linux containers under Docker
- Motivating clients: OmegaFlow recording, sandboxed AI agents, security
inspection, and untrusted-code execution
Expand Down Expand Up @@ -468,8 +477,8 @@ bytes are never parsed as protocol messages.
- `terminate`: request bounded graceful session termination.
- `complete`: after Host Reploy has emitted `workload_outputs_finalized`, declare
that the controller has finalized its client-owned results. It does not stop
an active workload and is rejected before workload output reaches a terminal
state.
an active workload and is rejected until successful publication of the
workload-output-finalization event.
- `acknowledge_terminated`: confirm receipt of the authoritative `terminated`
event. This payload-free protocol handshake is mandatory housekeeping, not a
granted capability, and is accepted only after Host Reploy has successfully
Expand Down Expand Up @@ -1071,27 +1080,33 @@ OmegaFlow these include the recording artifacts. Host Reploy does not open a
controller-finalization wait when `complete` was not granted and records that
controller as `not-completed` in the terminal result.
Repeated terminate or host cancel operations are idempotent. Input and resize
are rejected after `terminating` begins. A single `complete` remains valid
during termination while Host Reploy is waiting for controller finalization. A
`failed` workload-output result makes the session fail regardless of whether
the controller preserves and finalizes partial artifacts.
are rejected after `terminating` begins, and Host Reploy cancels any accepted
input or resize operation still blocked in the runtime before it begins
workload teardown. This cancellation does not stop request dispatch: a single
`complete` remains valid during termination while Host Reploy is waiting for
controller finalization. A `failed` workload-output result makes the session
fail regardless of whether the controller preserves and finalizes partial
artifacts.

Normal completion is:

1. Host Reploy observes workload exit, or a controller or host operation
requests termination.
2. Host Reploy atomically latches the cause and enters `terminating`.
3. Host Reploy performs bounded graceful termination followed by forced
termination when necessary.
3. Host Reploy cancels any in-flight workload request, then performs bounded
graceful termination followed by forced termination when necessary.
4. Host Reploy independently observes the exact workload container stopped.
5. Host Reploy drains and closes every declared workload-output surface under
the finite output-finalization deadline, then emits the one ordered
`workload_outputs_finalized` outcome.
6. When the live controller was granted `complete`, Host Reploy gives it a
bounded finalization period in which to close its client-owned output and
send `complete`. Without that grant, Host Reploy skips the wait and records
`not-completed`. A failed output outcome remains a session failure even when
partial client artifacts are finalized.
bounded finalization period only after that event is successfully published.
A response arriving while publication finishes is held until publication's
authoritative outcome is recorded; an earlier response is rejected. The
controller may then close its client-owned output and send `complete`.
Without that grant, Host Reploy skips the wait and records `not-completed`.
A failed output outcome remains a session failure even when partial client
artifacts are finalized.
7. Host Reploy removes the workload container, temporary mounts, networks, and
every other lease resource not required to deliver the final result. It
keeps the controller and private session channel alive.
Expand Down
52 changes: 44 additions & 8 deletions internal/controlledsession/lifecycle.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ const (
ObservationRuntimeObservationLostV1 ObservationKindV1 = "runtime-observation-lost"
ObservationStartupFailureV1 ObservationKindV1 = "startup-failure"
ObservationWorkloadOutputsFinalizedV1 ObservationKindV1 = "workload-outputs-finalized"
ObservationWorkloadOutputsPublishedV1 ObservationKindV1 = "workload-outputs-published"
ObservationWorkloadOutputFinalizationExpiredV1 ObservationKindV1 = "workload-output-finalization-expired"
ObservationControllerFinalizationExpiredV1 ObservationKindV1 = "controller-finalization-expired"
ObservationFinishedV1 ObservationKindV1 = "finished"
Expand All @@ -43,8 +44,13 @@ type ObservationV1 struct {
Kind ObservationKindV1
WorkloadStatus *ProcessStatusV1
WorkloadOutputFinalizationStatus *WorkloadOutputFinalizationStatusV1
Reason string
Finish *FinishV1
// WorkloadOutputPending records the narrow startup-failure case where
// the runtime started the workload before a later startup operation failed.
// The supervisor must then stop the workload and explicitly finalize its
// output before the lifecycle can finish.
WorkloadOutputPending bool
Reason string
Finish *FinishV1
}

type SnapshotV1 struct {
Expand All @@ -55,6 +61,7 @@ type SnapshotV1 struct {
RuntimeObservationStatus RuntimeObservationStatusV1
ControllerFinalizationStatus ControllerFinalizationStatusV1
AwaitingWorkloadOutputFinalization bool
AwaitingWorkloadOutputPublication bool
AwaitingControllerFinalization bool
AwaitingResultAcknowledgement bool
ResultAcknowledged bool
Expand All @@ -69,6 +76,7 @@ type TransitionV1 struct {
BeginTermination bool
WorkloadOutputFinalizationStatus WorkloadOutputFinalizationStatusV1
AwaitingWorkloadOutputFinalization bool
AwaitingWorkloadOutputPublication bool
AwaitingControllerFinalization bool
AwaitingResultAcknowledgement bool
RequestAccepted bool
Expand All @@ -93,6 +101,7 @@ type MachineV1 struct {
controller ControllerFinalizationStatusV1
runtimeObservation RuntimeObservationStatusV1
waitingOutputs bool
waitingPublication bool
waitingFinalize bool
resultDelivered bool
resultAcknowledged bool
Expand Down Expand Up @@ -123,6 +132,9 @@ func (machine *MachineV1) Observe(observation ObservationV1) (TransitionV1, erro
defer machine.mu.Unlock()
before := machine.state
transition := TransitionV1{Before: before, After: before, Cause: machine.cause}
if observation.WorkloadOutputPending && observation.Kind != ObservationStartupFailureV1 {
return transition, fmt.Errorf("%w: pending workload output is valid only for startup failure", ErrObservationRejected)
}
if observation.Kind == ObservationResultDeliveredV1 {
if observation.WorkloadStatus != nil || observation.WorkloadOutputFinalizationStatus != nil || observation.Reason != "" || observation.Finish != nil || machine.state != StateTerminatedV1 || machine.result == nil {
return transition, fmt.Errorf("%w: result delivery is valid only after termination and carries no payload", ErrObservationRejected)
Expand Down Expand Up @@ -184,6 +196,7 @@ func (machine *MachineV1) Observe(observation ObservationV1) (TransitionV1, erro
machine.controller = ControllerFinalizationStatusV1{Kind: ControllerFinalizationLostV1, Reason: observation.Reason}
}
machine.waitingFinalize = false
machine.waitingPublication = false
machine.latchLocked(CauseControllerLostV1, &transition)
case ObservationRuntimeObservationLostV1:
if err := validateCauseObservationV1(observation); err != nil {
Expand All @@ -198,17 +211,29 @@ func (machine *MachineV1) Observe(observation ObservationV1) (TransitionV1, erro
// reports failed closure or its bounded finalization deadline expires.
machine.finalizePreActivationOutputsForRuntimeObservationLossLocked(observation.Reason)
case ObservationStartupFailureV1:
if observation.WorkloadStatus != nil || observation.WorkloadOutputFinalizationStatus != nil || observation.Finish != nil {
return transition, fmt.Errorf("%w: startup failure carries only a reason", ErrObservationRejected)
if observation.WorkloadStatus != nil || observation.Finish != nil ||
(observation.WorkloadOutputPending && observation.WorkloadOutputFinalizationStatus != nil) {
return transition, fmt.Errorf("%w: startup failure carries a reason and at most one workload-output outcome", ErrObservationRejected)
}
if err := validateRequiredSafeTextV1("startup-failure reason", observation.Reason); err != nil {
return transition, fmt.Errorf("%w: %v", ErrObservationRejected, err)
}
if observation.WorkloadOutputFinalizationStatus != nil {
if err := validateWorkloadOutputFinalizationStatusV1(*observation.WorkloadOutputFinalizationStatus); err != nil {
return transition, fmt.Errorf("%w: %v", ErrObservationRejected, err)
}
}
if machine.controller.Kind != ControllerFinalizationUnknownV1 {
return transition, fmt.Errorf("%w: startup failure is invalid after controller activation", ErrObservationRejected)
}
machine.controller = ControllerFinalizationStatusV1{Kind: ControllerFinalizationStartupFailedV1, Reason: observation.Reason}
machine.latchLocked(CauseStartupFailureV1, &transition)
if observation.WorkloadOutputPending {
machine.workloadOutputs = WorkloadOutputFinalizationStatusV1{}
machine.waitingOutputs = true
} else if observation.WorkloadOutputFinalizationStatus != nil {
machine.completeOutputFinalizationLocked(*observation.WorkloadOutputFinalizationStatus)
}
case ObservationWorkloadOutputsFinalizedV1:
if observation.WorkloadStatus != nil || observation.WorkloadOutputFinalizationStatus == nil || observation.Reason != "" || observation.Finish != nil || !machine.waitingOutputs {
return transition, fmt.Errorf("%w: workload output finalization requires exactly one status while output finalization is pending", ErrObservationRejected)
Expand All @@ -225,6 +250,14 @@ func (machine *MachineV1) Observe(observation ObservationV1) (TransitionV1, erro
return transition, fmt.Errorf("%w: runtime observation loss requires failed workload output finalization", ErrObservationRejected)
}
machine.completeOutputFinalizationLocked(*observation.WorkloadOutputFinalizationStatus)
case ObservationWorkloadOutputsPublishedV1:
if observation.WorkloadStatus != nil || observation.WorkloadOutputFinalizationStatus != nil || observation.Reason != "" || observation.Finish != nil ||
machine.state != StateTerminatingV1 || machine.waitingOutputs || machine.workloadOutputs.Kind == "" || !machine.waitingPublication {
return transition, fmt.Errorf("%w: workload output publication is valid only after output finalization and carries no payload", ErrObservationRejected)
}
machine.waitingPublication = false
machine.waitingFinalize = machine.controller.Kind == ControllerFinalizationActiveV1 &&
containsOperationV1(machine.authorization.Operations, OperationCompleteV1)
case ObservationWorkloadOutputFinalizationExpiredV1:
if observation.WorkloadStatus != nil || observation.WorkloadOutputFinalizationStatus != nil || observation.Finish != nil || !machine.waitingOutputs {
return transition, fmt.Errorf("%w: output-finalization expiry carries only a required reason while output finalization is pending", ErrObservationRejected)
Expand All @@ -246,8 +279,8 @@ func (machine *MachineV1) Observe(observation ObservationV1) (TransitionV1, erro
if observation.WorkloadStatus != nil || observation.WorkloadOutputFinalizationStatus != nil || observation.Reason != "" || observation.Finish == nil {
return transition, fmt.Errorf("%w: finish requires exactly one terminal status set", ErrObservationRejected)
}
if machine.state != StateTerminatingV1 || machine.waitingOutputs || machine.workloadOutputs.Kind == "" || machine.waitingFinalize {
return transition, fmt.Errorf("%w: finish requires finalized workload output and no pending output or controller finalization", ErrObservationRejected)
if machine.state != StateTerminatingV1 || machine.waitingOutputs || machine.waitingPublication || machine.workloadOutputs.Kind == "" || machine.waitingFinalize {
return transition, fmt.Errorf("%w: finish requires published workload output finalization and no pending output or controller finalization", ErrObservationRejected)
}
if err := validateFinishV1(*observation.Finish); err != nil {
return transition, fmt.Errorf("%w: %v", ErrObservationRejected, err)
Expand Down Expand Up @@ -283,6 +316,7 @@ func (machine *MachineV1) Observe(observation ObservationV1) (TransitionV1, erro
transition.Cause = machine.cause
transition.WorkloadOutputFinalizationStatus = machine.workloadOutputs
transition.AwaitingWorkloadOutputFinalization = machine.waitingOutputs
transition.AwaitingWorkloadOutputPublication = machine.waitingPublication
transition.AwaitingControllerFinalization = machine.waitingFinalize
transition.AwaitingResultAcknowledgement = machine.resultDelivered && !machine.resultAcknowledged
transition.ResultAcknowledged = machine.resultAcknowledged
Expand Down Expand Up @@ -364,6 +398,7 @@ func (machine *MachineV1) ApplyRequest(request RequestV1) (TransitionV1, error)
transition.Cause = machine.cause
transition.WorkloadOutputFinalizationStatus = machine.workloadOutputs
transition.AwaitingWorkloadOutputFinalization = machine.waitingOutputs
transition.AwaitingWorkloadOutputPublication = machine.waitingPublication
transition.AwaitingControllerFinalization = machine.waitingFinalize
transition.AwaitingResultAcknowledgement = machine.resultDelivered && !machine.resultAcknowledged
transition.RequestAccepted = true
Expand Down Expand Up @@ -395,6 +430,7 @@ func (machine *MachineV1) snapshotLocked() SnapshotV1 {
RuntimeObservationStatus: machine.runtimeObservation,
ControllerFinalizationStatus: machine.controller, AwaitingControllerFinalization: machine.waitingFinalize,
AwaitingWorkloadOutputFinalization: machine.waitingOutputs,
AwaitingWorkloadOutputPublication: machine.waitingPublication,
AwaitingResultAcknowledgement: machine.resultDelivered && !machine.resultAcknowledged,
ResultAcknowledged: machine.resultAcknowledged, Result: cloneResultV1(machine.result),
}
Expand Down Expand Up @@ -441,8 +477,8 @@ func (machine *MachineV1) finalizePreActivationOutputsForRuntimeObservationLossL
func (machine *MachineV1) completeOutputFinalizationLocked(status WorkloadOutputFinalizationStatusV1) {
machine.workloadOutputs = status
machine.waitingOutputs = false
machine.waitingFinalize = machine.controller.Kind == ControllerFinalizationActiveV1 &&
containsOperationV1(machine.authorization.Operations, OperationCompleteV1)
machine.waitingPublication = machine.controller.Kind == ControllerFinalizationActiveV1
machine.waitingFinalize = false
}

func equalProcessStatusV1(left ProcessStatusV1, right ProcessStatusV1) bool {
Expand Down
Loading
Loading