[Bug]: TestHAKeeperCanBootstrapAndRepairShards failed

[w-zr]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):
- Hardware parameters:
- OS type:
- Others:

Actual Behavior

Expected Behavior

UT should pass.

Steps to Reproduce

No response

Additional information

No response

[volgariver6]

刚创建时的日志不全，跑ut时又发生了一次，上传日志
fail.log

[gouhongshen]

the error seems caused by timeout(2s) when invoke state, err := store1.getCheckerState().

and the log detail shows that some communication problems exist between the raft nodes (and the gossip nodes).

after 1000+ go test run -v TestHAKeeperCanBootstrapAndRepairShards and a dozen times make ut without error shows, I tried traffic control by executing shell command:

tc qdisc add dev lo root netem delay 50ms 10ms
tc qdisc add dev lo root netem loss 10% 

or add some random sleep in dragonbaot/internal/raft/raft.go::handleHeartbeatMessage().

And I got a similar timeout error, but not totally the same:

the same:

1. leader lost quorum
2. HAKeeper cannot be bootstrapping before the timeout
3. same term num elapsed

deference：

1. no gossip nodes have been marked as failed

so I am not sure it must be the network issue.

[gouhongshen]

the memberlist: errors may relate to these issues:

1. How to detect and react to TCP only failures hashicorp/memberlist#264
2. Document TCP-only operation hashicorp/memberlist#226

all mentioned k8s environment

[gouhongshen]

It's been a long time since it happened again, may close.

[YANGGMM]

又发现了
failed link:https://github.com/matrixorigin/matrixone/actions/runs/7027670848/job/19122446502?pr=13068

[gouhongshen]

TestHAKeeperCanBootstrapAndRepairShards.txt

[gouhongshen]

not working on it

[gouhongshen]

not working on it

[gouhongshen]

augment the hakeeperDefaultTimeout config may work.

[heni02]

最近CI UT没有失败，先关闭

https://github.com/matrixorigin/matrixone/actions/runs/7619951914/job/20753888415?pr=14324