Skip to content

[Bug]: Lifecycle finalizers can overwrite a concurrent worker-deletion crash #1443

Description

@EItanya

What happened?

When a worker disappears during ResumeActor or SuspendActor, DeleteWorker marks the actor
CRASHED and clears its worker assignment. The lifecycle workflow can subsequently re-read that
updated actor and unconditionally finalize it:

  • Resume changes CRASHED to RUNNING, leaving a running actor without a worker.
  • Suspend changes CRASHED to SUSPENDED, potentially without a valid snapshot.

Optimistic version checks do not prevent this because the finalizers read and update the new
CRASHED version.

Expected Behavior

finalization should require the actor to remain RESUMING or SUSPENDING. If worker deletion has already crashed it, the workflow should preserve CRASHED and return FailedPrecondition.

Steps to Reproduce

Regression tests can reproduce both cases by deleting the worker from the fake atelet
immediately after restore/checkpoint completes.

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

main

Kubernetes Version & Environment

No response

Host OS & Architecture

No response

Relevant Logs and Diagnostic Output

Additional Context

No response

Confirmation

  • I have searched existing issues and verified that this is not a duplicate.
  • I have verified that this issue occurs on the latest commit on main.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions