feat: retry connect to meta until meta online

[yezizp2012]

What's changed and what's your intention?

As title. This PR fixes #3065, but we still need to implement gRPC health check protocol mentioned in #41.

Please explain IN DETAIL what the changes are in this PR and why they are needed:

• Summarize your change (mandatory)
• How does this PR work? Need a brief introduction for the changed logic (optional)
• Describe clearly one logical change and avoid lazy messages (optional)
• Describe any limitations of the current code (optional)
• Add the 'user-facing changes' label if your PR contains changes that are visible to users (optional)

Checklist

• I have written necessary docs and comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

Resolve #3065 #295

[codecov]

Codecov Report

Merging #3165 (bc19551) into main (fb9174f) will decrease coverage by 0.00%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #3165      +/-   ##
==========================================
- Coverage   73.52%   73.51%   -0.01%     
==========================================
  Files         739      739              
  Lines      101695   101710      +15     
==========================================
+ Hits        74770    74771       +1     
- Misses      26925    26939      +14     

Flag	Coverage Δ
rust	73.51% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/rpc_client/src/meta_client.rs	0.00% <0.00%> (ø)
src/meta/src/barrier/mod.rs	69.45% <0.00%> (+0.32%)	⬆️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[BowenXiao1999]

I remembered that we have retry logic in Frontend/CN when connect to meta?

[yezizp2012]

I remembered that we have retry logic in Frontend/CN when connect to meta?

Nope, only the legacy Frontend does 😂. Currently, FE/CN will panic when meta is not online.

[BugenZhao]

LGTM.
BTW, I remember the purpose of moving these RPC interfaces into a separate crate is to implement a generic retry strategy for them. Any further idea on this? 🤣

[yezizp2012]

LGTM. BTW, I remember the purpose of moving these RPC interfaces into a separate crate is to implement a generic retry strategy for them. Any further idea on this? 🤣

Not really. 🤣 Well I'm not sure about this. NTFS.