Skip to content

[Bug]: [Presets] Any failure during resume marks the creation permanently failed #4164

Description

@peterschmidt85

Steps to reproduce

  1. Start a creation from a configuration with any env entry, e.g. env: [MODEL_LABEL=some-value].
  2. Stop it: dstack preset stop <id> -y. It correctly records interrupted and prints Resume it with dstack preset resume <id>.
  3. Resume from a shell where that variable is not set: dstack preset resume <id>. It fails with Environment variable MODEL_LABEL is not set ([Bug]: [Presets] Resume requires literal env values in the local environment #4165).
  4. Resume again, with the variable set: dstack preset resume <id>.

Actual behaviour

Step 4 refuses:

Preset <id> creation failed and cannot be resumed

Step 3 is what marked the session failed. create_preset wraps the whole creation in except BaseException: _close_agent_session(session, "failed") (create.py:437-439), so any exception is terminal: it writes failed and deletes the workspace (create.py:754-756, workspace.py:120-130). Step 3's exception is raised at create.py:415, before _create_preset is entered, so nothing had started and nothing had changed, yet the session became unresumable. load_resumable_session then refuses (session.py:331-332).

Not caused by reconcile: _is_reconcilable requires status running, an existing report, and a dead owner (create.py:252-262), so an interrupted session is never reconciled.

Expected behaviour

Step 3 leaves the session interrupted, and step 4 resumes it. A failure that happens before the agent is spawned is not a failed creation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions