[Feature]: Pause, update, and resume active threads as one operation

[kossoy]

Area

apps/server

Requested workflow

Please provide one operation to pause active threads → update the server → resume those threads. With several threads running, updating currently requires deciding when to interrupt them and then potentially reopening and prompting each thread individually.

The operation should capture the pre-update active-thread set, let work pause at a safe boundary or explicitly interrupt it, preserve the recovery information across the update, and resume only that captured set. Show which threads resumed and allow retrying the ones that did not. Threads already stopped by the user should stay stopped.

The existing Continue threads after restarts preference is useful, but does not expose this coordinated maintenance workflow. The current V1 → V2 upgrade also appears to have a migration-specific limitation described below.

Versions and environment

• Running server: 0.0.45-nightly.20261002.2561, orchestration protocol 1.
• Staged replacement: 0.0.46-nightly.20261003.2623, orchestration protocol 2.
• Linux remote server managed by a custom systemd user service; macOS desktop client.

Example scenario

1. Run the protocol-1 server with multiple active threads.
2. Enable Settings → General → Continue threads after restarts, including any relevant project overrides.
3. Update/restart into 0.0.46-nightly.20261003.2623 while those threads are still running.
4. Resume the pre-update active set together, without manually visiting each thread.

Expected behavior

An update should preserve the active-thread set and honor the continuation preference across the migration. If native provider sessions cannot be retained, offer a single action to continue those threads from their imported history and clearly explain that limitation.

Migration limitation motivating this request

This part is source-derived, not a completed runtime reproduction. I inspected both shipped nightly binaries before restarting the active server; I have not performed this upgrade with active threads.

The staged binary contains these source regions:

• apps/server/src/orchestration-v2/legacy/LegacyV1ThreadImporter.ts: importedThread() sets activeProviderThreadId: null and historyOrigin: "v1_import". Imported messages and turn items have runId: null and no provider thread/turn identity. Streaming legacy items become interrupted. The importer writes thread metadata and transcript events, but does not import runs or provider sessions.
• apps/server/src/orchestration-v2/RestartContinuation.ts: restartContinuationRun() requires a recoverable V2 run and a strong native provider-thread reference. continueRestartedRun() uses continueThreadsAfterServerUpdate to dispatch a continuation for a qualifying source run.
• The old server's serverRuntimeStartup.ts supports restart continuation through the V1 session binding and resume cursor. The new binary does not contain the old continueAfterServerUpdate marker.

This leaves no apparent path from an active V1 session to a qualifying V2 restart continuation. Please confirm whether another migration path covers this case; I am not claiming a reproduced runtime bug.

Impact and workaround

This blocks a convenient update while multiple threads are active. Waiting for all work to finish avoids interruption; manually continuing each thread after migration is the obvious fallback. Neither provides a bulk pause/update/resume workflow. No workaround was tested as part of this report.

The proposed workflow would also cover this boundary by carrying enough interrupted-run/resume state across migration, or keeping a recoverable pre-update active-thread list with a single Resume all action when native continuation is unavailable.

Related issues

• feat(threads): opt in to automatic resume after connection loss #13740 concerns connection loss.
• [Feature]: Add a native Start/Resume action for Codex threads without injecting a "continue" message #12825 concerns per-thread resume.

This request is for a coordinated bulk maintenance operation across active threads.

Investigated with OpenAI GPT-6-Astra in the Codex harness; findings are from shipped source inspection, not an executed migration test.

[juliusmarminge]

Note

Grok responding on behalf of Julius.

Triage

Thanks @kossoy for the careful source inspection and for being clear about what was and wasn't reproduced. I checked this on current main (31a9da179). Your reading of the staged nightly matches the source, and there's no second migration path that turns an in-flight V1 session into a V2 restart continuation. That gap comes from the documented protocol cutover rather than a regression in the restart setting. The bulk pause, update, and resume operation you describe isn't implemented today.

What an update already does

Settings → General → Continue threads after restarts (continueThreadsAfterServerUpdate, off by default, overridable per project) is the V2 mechanism. On a clean shutdown, prepareForShutdown in apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts records qualifying runs and enqueues provider-runtime.continue. After the process comes back, continueRestartedRun sends "Continue where you left off." for a cancelled source run that still has a strong native provider thread. Startup reconciliation can do the same for a run that's still running, or for a settled run whose background work was cancelled. Threads you already stopped stay stopped, because restartContinuationRun only accepts those cases.

The old per-update flag isn't part of this. ServerEnvironment advertises threadRestartContinuation but leaves serverUpdateThreadContinuation unset, noting that the per-update request flag isn't wired into the V2 update RPC. The client only sends continueRunningThreads when that capability is true, and withRunningThreadContinuation isn't in the live update layer. So the update button doesn't capture an active-thread set, wait for a safe boundary, or report which threads resumed.

A custom systemd unit also doesn't get the in-app updater unless it's the T3 background service (t3 service install). Restarting the process yourself still runs the shutdown finalizer, but only for V2 runs that qualify.

The protocol 1 to 2 cutover

Your notes are right, and nothing else fills the gap:

• importedThread() in LegacyV1ThreadImporter.ts sets activeProviderThreadId: null and historyOrigin: "v1_import". Transcript rows get runId: null and no provider thread, and a message that was still streaming becomes an interrupted turn item. The importer doesn't import runs or provider sessions.
• restartContinuationRun() then finds no run and no strong native provider thread, so continueRestartedRun() never runs for those threads.
• The supported continuation is the first later message: shouldPrepareLegacyImportHandoff builds a handoff from the imported text and starts a fresh provider session. That's manual and one thread at a time.

This is described in Threads from older T3 Code versions, Legacy orchestration migration, and Updating. Like you, this is from source and docs; the upgrade itself wasn't run.

What would be new

A single operation that captures the pre-update active set, pauses or interrupts it, resumes only that set, shows which threads came back, and lets you retry the rest (including a Resume all when native sessions can't be kept) would be a new workflow. The current setting resumes qualifying V2 threads quietly, with no status or retry surface, and doesn't drain work to a safe boundary first.

#13740 and #12825 are different, as you said. The closest existing proposal is Discussion #14813 (install a desktop update when agents are idle), which doesn't cover remote/systemd updates, a captured active set, resume status and retry, or the V1 handoff case.

If you'd like to push for the coordinated operation, Ideas is the best place. A maintainer will decide what to do with this issue. For this nightly jump, the supported path is to let running work finish (or accept the interruption), update, and send a new message on each thread you want to continue.

[AdEx-Partners-DE]

A data point from the other direction, and a building block that may help here.

I run a self-built server and redeploy it several times a day, so I hit exactly this: when is it safe to restart? Two things I learned:

• Waiting for idle works, but only if nothing new can start. My deploy script refuses while any run is in flight. With scheduled tasks and queued prompts around, the environment was never idle; this morning it stayed busy for hours.
• A forced restart did not bring the work back. I restarted once with three turns running and Continue threads after restarts on. None of the three continued by themselves. I have not found the cause yet, so take that as an observation, not a diagnosis.

What made the difference was a pause that is not a stop: in my fork one switch holds every queue and another holds every scheduled task, and neither touches a turn that is running. Turn both on, the running turns finish, nothing moves up, and the environment drains to idle on its own. Then update, then resume, and the queues start again by themselves. That is the "pause at a safe boundary" half of what you describe; the "resume the captured set after a hard interrupt" half I do not have.

Details and code are in #18179. Would a drain like that cover your case, or do you need to interrupt turns that would run for hours?

---

Posted by Claude (Opus 5.5, Claude Code via T3 Code) on behalf of @AdEx-Partners-DE.