Skip to content

[Bug] [Master] Recovering a failed workflow changes a forced-success task to SUCCESS #18604

Description

@wcmolin

Search before asking

  • I searched the existing issues and found no issue reporting the same status transition.

What happened

After a failed task instance is manually set to forced success, its state is correctly changed to FORCED_SUCCESS.

If the workflow instance is then recovered using "Recovery Failed":

  • the forced-success task is skipped as expected;
  • downstream tasks continue to run;
  • but the state of the forced-success task instance is changed from FORCED_SUCCESS to SUCCESS.

The workflow can continue normally, but the information indicating that the task was completed through manual intervention is lost. The task then appears to have completed successfully through normal execution.

This behavior was reproduced on a deployment based on 3.4.2 and was also confirmed in the current dev branch source code.

What you expected to happen

After recovering the failed workflow:

  • the forced-success task should be treated as successful for dependency and workflow topology evaluation;
  • downstream tasks should continue to run;
  • the task instance state should remain FORCED_SUCCESS.

Recovering a workflow should not convert an existing FORCED_SUCCESS state to SUCCESS.

How to reproduce

  1. Create a workflow containing at least two sequential tasks: A -> B.
  2. Run the workflow and make task A fail.
  3. Perform the "Force Success" operation on task A.
  4. Confirm that the state of task A is FORCED_SUCCESS.
  5. Perform the "Recovery Failed" operation on the workflow instance.
  6. Wait for downstream task B to start.
  7. Check the state of task A again.

Actual result:

FORCED_SUCCESS -> SUCCESS

Expected result:

FORCED_SUCCESS -> FORCED_SUCCESS

Anything else

The issue is still present in the dev branch at commit:

eca5d5e3df52f812c7820b5da6de5feb5cc140e4

The relevant source-code path is:

  1. RecoverFailureTaskCommandHandler retains a FORCED_SUCCESS task instance when rebuilding the workflow execution graph because only FAILURE, KILL, and PAUSE tasks are recreated or recovered.
  2. The retained task receives a TaskStartLifecycleEvent when the recovered workflow starts processing the execution graph.
  3. TaskForceSuccessStateAction inherits the start-event behavior from TaskSuccessStateAction.
  4. The inherited success handling unconditionally persists the task state as TaskExecutionStatus.SUCCESS.

Relevant files:

  • dolphinscheduler-master/.../command/handler/RecoverFailureTaskCommandHandler.java
  • dolphinscheduler-master/.../task/statemachine/TaskForceSuccessStateAction.java
  • dolphinscheduler-master/.../task/statemachine/TaskSuccessStateAction.java
  • dolphinscheduler-master/.../task/statemachine/AbstractTaskStateAction.java

This does not prevent downstream execution, but it removes the audit distinction between a normally successful task and a manually forced-success task.

Version

dev

Are you willing to submit a PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

  • I agree to follow this project's Code of Conduct.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

backendbugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions