Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STORM-3655: Worker should die if its assignment has changed #3291

Merged
merged 1 commit into from
Jul 1, 2020

Conversation

kishorvpatil
Copy link
Contributor

What is the purpose of the change

The assignment change should lead to worker suicide. This is specially critical if Supervisor is down for some reason

How was the change tested

  1. Launcher WordCountTopology, wait for worker to start
  2. Stop the supervisor
  3. Re-balance topology so worker is assigned to next supervisor
  4. Worker should kill itself with worker.log entries similar to below.
2020-06-22 06:55:40.527 o.a.s.d.w.WorkerState refresh-connections-timer [INFO] Found conflicting assignments. We shouldn't be alive! Assigned: [[-1, -1]], Current: [[-1, -1], [14, 14], [12, 12], [10, 10], [8, 8], [6, 6], [4, 4], [2, 2], [7, 7], [5, 5], [3, 3], [1, 1], [13, 13], [11, 11], [9, 9]]
2020-06-22 06:55:40.529 o.a.s.u.Utils refresh-connections-timer [ERROR] Halting process: Worker died
java.lang.RuntimeException: Halting process: Worker died
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:518) ~[storm-client-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.daemon.worker.WorkerState.suicideIfLocalAssignmentsChanged(WorkerState.java:390) ~[storm-client-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.daemon.worker.WorkerState.refreshConnections(WorkerState.java:421) ~[storm-client-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:110) [storm-client-2.3.0.y.jar:2.3.0.y]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226) [storm-client-2.3.0.y.jar:2.3.0.y]
2020-06-22 06:55:40.552 o.a.s.d.w.Worker Thread-32 [INFO] Shutting down worker fwc2-1-1592662435 44c0bd8e-bd3c-4317-b09f-1f0deb3f2a7c 6700

Copy link
Contributor

@Ethanlm Ethanlm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Left a minor comment.

@@ -401,6 +418,7 @@ public void refreshConnections() {
}
}

suicideIfLocalAssignmentsChanged(assignment);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can place this as earlier as possible in this refreshConnections method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed

Copy link
Contributor

@Ethanlm Ethanlm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@Ethanlm Ethanlm merged commit 4fdfc0e into apache:master Jul 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants