feat: introduce the region role

[WenyXu]

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Introduce the region's role. Now, we only have two role:

The RegionRole::Follower: Readonly Region(mito2), Readonly Region(file).
The RegionRole::Leader: Writable Region(mito2).

Notes: We treat all File Region as RegionRole::Follower, even if it's the leader peer in the region routes.

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

#2639

[codecov]

Codecov Report

Merging #2640 (be4a115) into develop (88eb695) will decrease coverage by 0.41%.
The diff coverage is 41.50%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2640      +/-   ##
===========================================
- Coverage    85.01%   84.61%   -0.41%     
===========================================
  Files          739      740       +1     
  Lines       120141   120181      +40     
===========================================
- Hits        102137   101690     -447     
- Misses       18004    18491     +487     

[waynexia]

Notes: We treat all File Region as RegionRole::Follower, even if it's the leader peer in the region routes.

From this point, how about defining region's role in a group of permission? Something like POSIX permission (but they have many differences). E.g.:

• Can a region write to object store
• Can a region write to WAL
• Can a region read WAL
• Can a region read object store
• ...

By doing this, we don't need to add extra conditions for file regions. And it's more flexible and straightforward. Like, to reduce WAL subscribe workload, we can have a region that only fetches data from S3 but is not aware of the newest WAL change. As well as the compactor only needs read/write permission to object-store (and update manifest, but this might need WAL write permission in the future). And this is closer to the underlying engine's permission implementation.

[WenyXu]

@MichaelScofield @fengjiachun PTAL

[fengjiachun]

LGTM

[fengjiachun]

Notes: We treat all File Region as RegionRole::Follower, even if it's the leader peer in the region routes.

From this point, how about defining region's role in a group of permission? Something like POSIX permission (but they have many differences). E.g.:

• Can a region write to object store
• Can a region write to WAL
• Can a region read WAL
• Can a region read object store
• ...

By doing this, we don't need to add extra conditions for file regions. And it's more flexible and straightforward. Like, to reduce WAL subscribe workload, we can have a region that only fetches data from S3 but is not aware of the newest WAL change. As well as the compactor only needs read/write permission to object-store (and update manifest, but this might need WAL write permission in the future). And this is closer to the underlying engine's permission implementation.

This looks good, but we still need two completely separate roles to help metasrv easy to make better scheduling.