[Bug]: task manager panic

[lxl66566]

After fixed #907 in #918, the repeat test on shutdown_rpc_should_shutdown_the_cluster() will continue to fail on task manager panic. This panic is not introduced by #918, because before #918, there's already a panic case.

This bug cannot be reproduced stably. The average panic possibility maybe 1/525 (tested failure round: 231, 134, 1211).

Log

1. failed3.log is the panic log before fix: shutdown cluster test timeout #918 change.
2. panic8.log, panic9.log, panic10.log is the three panic record tested on fix: shutdown cluster test timeout #918 (commit 1607307)

[github-actions]

👋 Thanks for opening this issue!

Reply with the following command on its own line to get help or engage:

• /contributing-agreement : to print Contributing Agreements.
• /assignme : to assign this issue to you.

[lxl66566]

When cluster is running, all tasks should exist, so we can use unwrap in getting tasks from task manager. But when cluster shutdown, the tasks will be removed from task manager top-to-down.
The remove behavior is performed by cmd_worker worker_as(), which runs on another parallel thread, so the removal timing is unspecified. So getting tasks may get an None, indicates the task has been removed by worker_as.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 14 days with no activity.