Region 265098266411008(61723, 0) is in ReadOnly state, expect: Writable

[killme2008]

What type of bug is this?

Unexpected error

What subsystems are affected?

Distributed Cluster

Minimal reproduce step

In some cases, the region may go into read-only state and can't be recovered at all

Internal Server Error: {\"error\":\"Region 265098266411008(61723, 0) is in ReadOnly state, expect: Writable\"}"

Please investigate and address this issue seriously, as it is impacting availability. @WenyXu @fengjiachun

What did you expect to see?

The region should recover normally even when node crash etc.

What did you see instead?

The region keeps in a read-only state.

What operating system did you use?

Not relevant

What version of GreptimeDB did you use?

0.8.2

Relevant log output and stack trace

No response

[killme2008]

@WenyXu Did we figure out the cause of this issue?

[WenyXu]

@WenyXu Did we figure out the cause of this issue?

Yes, the datanode ran out of CPU resources 🥲, and it can't send any heartbeats to metasrv at that moment.

[killme2008]

How and when to fix it? @WenyXu

[WenyXu]

How and when to fix it? @WenyXu

For now, #4128 may mitigate the impact of this problem. However, a better solution is to rely on the region HA mechanism.