Takes too long time for a cluster to recover from leader crash

[kikimo]

Please check the FAQ documentation before raising an issue

Please check the FAQ documentation and old issues before raising an issue in case someone has asked the same question that you are asking.

Describe the bug (must be provided)

Takes very long time for cluster to recover from leader crash.

Your Environments (must be provided)

• OS: uname -a
• Compliler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id (e.g. a3ffc7d8)
  https://github.com/liuyu85cn/nebula.git 730a39d

How To Reproduce(must be provided)

Steps to reproduce the behavior:

1. create a cluster with 3storage + 3graph + 1meta, a space with 3partition + 3replicas
2. keep inserting edges
3. send sigstop to one leader(you can check part leader by typiing show hosts in nebula-console)

Despite the fact that nebula storage elect a new leader very quickly, it takes nearly 5min for the cluster to get back to normal.

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Provide logs and configs, or any other context to trace the problem.

[liuyu85cn]

In one reproduce, raft take 15min do a rollback, and graph client looks stuck.

[Sophie-Xie]

In one reproduce, raft take 15min do a rollback, and graph client looks stuck.

@Aiee Go client, pls check it.

[Sophie-Xie]

raft #2903

[critical27]

This could a be quite complicated case, frankly speaking, I have no idea what will happen when sigstop is sent to thrift server, this may involved system futex. (raft take 15 min do a rollback, clearly impossible to blame rollback).

We could discover later.

[kikimo]

This could a be quite complicated case, frankly speaking, I have no idea what will happen when sigstop is sent to thrift server, this may involved system futex. (raft take 15 min do a rollback, clearly impossible to blame rollback).

We could discover later.

The effect of SIGSTOP is just stop scheduling this process from running on cpu temporarily, exactly same as when you run gdb and attach to the process, if the process's invoke of futex() cause block, I'm 100% sure that's not the problem of SIGSTOP. I'll add more detail later.

[kikimo]

closed by #3435