Handle leadership change and shutdown placement gracefully

[youngbupark]

Description

This PR is to handle leadership establishment and revoke scenario properly and shutdown placement gracefully to reduce the impact to runtime and the other placement nodes.

1. Add --cluster-bootstrap and --raft-logstore-path
2. Handle leadership establishment safely
3. Shutdown gRPC and Raft server gracefully
4. log out all Raft logs as debug level

Run three placement nodes:

placement --port 50005 --id node0 --initial-cluster node0=127.0.0.1:15050,node1=127.0.0.1:15051,node2=127.0.0.1:15052 --raft-logstore-path log-node0 

placement --port 50006 --id node1 --initial-cluster node0=127.0.0.1:15050,node1=127.0.0.1:15051,node2=127.0.0.1:15052 --raft-logstore-path log-node1 --cluster-bootstrap=false

placement --port 50007 --id node2 --initial-cluster node0=127.0.0.1:15050,node1=127.0.0.1:15051,node2=127.0.0.1:15052 --raft-logstore-path log-node2 --cluster-bootstrap=false

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #2377

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Unit tests passing
• End-to-end tests passing
• Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

[codecov]

Codecov Report

Merging #2390 into master will increase coverage by 1.04%.
The diff coverage is 55.97%.

@@            Coverage Diff             @@
##           master    #2390      +/-   ##
==========================================
+ Coverage   55.55%   56.59%   +1.04%     
==========================================
  Files          79       80       +1     
  Lines        6698     6836     +138     
==========================================
+ Hits         3721     3869     +148     
+ Misses       2721     2709      -12     
- Partials      256      258       +2     

Impacted Files	Coverage Δ
pkg/actors/actors.go	55.68% <0.00%> (+7.88%)	⬆️
pkg/actors/config.go	100.00% <ø> (ø)
pkg/placement/raft/logger.go	0.00% <0.00%> (ø)
pkg/placement/raft/server.go	0.00% <0.00%> (ø)
pkg/runtime/runtime.go	48.13% <0.00%> (-0.12%)	⬇️
pkg/placement/membership.go	43.58% <24.00%> (-31.85%)	⬇️
pkg/runtime/cli.go	3.84% <44.44%> (+3.84%)	⬆️
pkg/placement/placement.go	78.08% <65.00%> (-3.89%)	⬇️
pkg/actors/internal/placement.go	87.91% <87.91%> (ø)
pkg/runtime/config.go	100.00% <100.00%> (ø)
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49e3e6f...a1ac031. Read the comment docs.

[youngbupark]

/ok-to-test

[dapr-bot]

Found the available test cluster - dapr-aks-e2e-06 for linux. Please check the build status.

[dapr-bot]

Found the available test cluster - dapr-aks-e2e-08 for windows. Please check the build status.

[dapr-bot]

Congrats! All end-to-end tests have passed on linux. Thanks for your contribution!

[dapr-bot]

Congrats! All end-to-end tests have passed on windows. Thanks for your contribution!

[tcnghia]

Suggested change

	flag.BoolVar(&cfg.raftInMemEnabled, "inmem-store-enabled", cfg.raftInMemEnabled, "Enable in-memory log and snapshot store")
	flag.BoolVar(&cfg.raftInMemEnabled, "inmem-store-enabled", cfg.raftInMemEnabled, "Enable in-memory log and snapshot store. Can only work if --raftLogStorePath is unset.")

[tcnghia]

Will we anticipate more cases in this switch? Otherwise, change to if-else?

[youngbupark]

yes will do that.