Bug: SureralDB can't survive TiKV cluster restart

[pashinin]

Describe the bug

SurrealDB lost connection and can't reconnect normally. Logs full of:

2024-02-23T06:52:12.567638Z ERROR surreal::node: Error running node agent tick: There was a problem with a datastore transaction: Leader of region 5 is not found
2024-02-23T06:52:22.569187Z ERROR surreal::node: Error running node agent tick: There was a problem with a datastore transaction: Leader of region 5 is not found
2024-02-23T06:52:32.570532Z ERROR surreal::node: Error running node agent tick: There was a problem with a datastore transaction: Leader of region 5 is not found
2024-02-23T06:52:42.572181Z ERROR surreal::node: Error running node agent tick: There was a problem with a datastore transaction: Leader of region 5 is not found
2024-02-23T06:52:52.574179Z ERROR surreal::node: Error running node agent tick: There was a problem with a datastore transaction: Leader of region 5 is not found
2024-02-23T06:53:02.576163Z ERROR surreal::node: Error running node agent tick: There was a problem with a datastore transaction: Leader of region 5 is not found

Same error in a shell. The worst part is that /health endpoint returns OK 200.

If I manually restart SurrealDB - everything is ok.

Steps to reproduce

1. stop tikv cluster
2. start tikv cluster

Expected behaviour

SurrealDB can reconnect.

SurrealDB version

1.2.1 for linux on x86_64

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[ioannist]

We ran into the same error after an extended async operation that failed because of it #3408

I assume this happens when TiKV decides to change to another leader node for whatever reason.

[sgirones]

thanks for reporting, we are aware of it and will work on a fix

[manelmontilla]

After some time digging into it, it looks the issue is similar to what this PR in the TIKV Rust client tries solve: tikv/client-rust#445. But the the problem is that this PR seems to only try to fix the issue when the client is created during the server startup, but not for when a new the leader of a region is elected. The PR mentions that for these other cases, that is: after the client is created, it should be the application using the client the one retrying on the errors. So probably, we should modify the TIKV Datastore to gracefully handle those errors by recreating the client.