Skip to content

[chore](cloud) Implement idempotent injection framework for meta-service #51905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

gavinchou
Copy link
Contributor

This commit support to inject idempotent requests by replaying request with random delay

the following configs are added

CONF_mBool(enable_idempotent_request_injection, "false");
// idempotent_request_replay_delay_ms = idempotent_request_replay_delay_base_ms + random(-idempotent_request_replay_delay_range_ms, idempotent_request_replay_delay_range_ms)
CONF_mInt64(idempotent_request_replay_delay_base_ms, "10000");
CONF_mInt64(idempotent_request_replay_delay_range_ms, "5000");
// exclude some request that are meaningless to replay, comma separated list. e.g. GetTabletStatsRequest,GetVersionRequest
CONF_mString(idempotent_request_replay_exclusion, "GetTabletStatsRequest,GetVersionRequest");

to enable request replay

enable_idempotent_request_injection=true

to set random delay in a range
[idempotent_request_replay_delay_base_ms - idempotent_request_replay_delay_range_ms, idempotent_request_replay_delay_base_ms + idempotent_request_replay_delay_range_ms] the following example means the request will be replayed with a delay ranges from 5000ms to 15000ms

idempotent_request_replay_delay_base_ms=10000
idempotent_request_replay_delay_range_ms=5000

if the randomly chosen delay is negative the request will be not replayed (idempotent_request_replay_delay_range_ms can be set greater than idempotent_request_replay_delay_base_ms)

to exclude some requests (PB message names), the following examples means GetTabletStatsRequest and GetVersionRequest will not be replayed

idempotent_request_replay_exclusion=GetTabletStatsRequest,GetVersionRequest

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou gavinchou changed the title (cloud) Implement idempotent injection framework for meta-service [chore](cloud) Implement idempotent injection framework for meta-service Jun 18, 2025
This commit support to inject idempotent requests by replaying request with random delay

the following configs are added
```
CONF_mBool(enable_idempotent_request_injection, "false");
// idempotent_request_replay_delay_ms = idempotent_request_replay_delay_base_ms + random(-idempotent_request_replay_delay_range_ms, idempotent_request_replay_delay_range_ms)
CONF_mInt64(idempotent_request_replay_delay_base_ms, "10000");
CONF_mInt64(idempotent_request_replay_delay_range_ms, "5000");
// exclude some request that are meaningless to replay, comma separated list. e.g. GetTabletStatsRequest,GetVersionRequest
CONF_mString(idempotent_request_replay_exclusion, "GetTabletStatsRequest,GetVersionRequest");
```

to enable request replay
```
enable_idempotent_request_injection=true
```

to set random delay in a range
[idempotent_request_replay_delay_base_ms - idempotent_request_replay_delay_range_ms, idempotent_request_replay_delay_base_ms + idempotent_request_replay_delay_range_ms]
the following example means the request will be replayed with a delay ranges from 5000ms to 15000ms
```
idempotent_request_replay_delay_base_ms=10000
idempotent_request_replay_delay_range_ms=5000
```
if the randomly chosen delay is negative the request will be not replayed
(idempotent_request_replay_delay_range_ms can be set greater than idempotent_request_replay_delay_base_ms)

to exclude some requests (PB message names), the following examples
means GetTabletStatsRequest and GetVersionRequest will not be replayed
```
idempotent_request_replay_exclusion=GetTabletStatsRequest,GetVersionRequest
```
@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 92.31% (36/39) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.20% (1124/1351)
Line Coverage 66.81% (19373/28996)
Region Coverage 66.47% (9583/14418)
Branch Coverage 56.52% (5216/9228)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 19, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. cloud dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants