[Remote Compaction] Simulate e2e flow in Stress Test#13800
[Remote Compaction] Simulate e2e flow in Stress Test#13800jaykorean wants to merge 6 commits intofacebook:mainfrom
Conversation
jaykorean
changed the title
|
@jaykorean has imported this pull request. If you are a Meta employee, you can view this in D78862084. |
jaykorean
force-pushed
the
remote_compaction_stress_test_
branch
from
cda4e1f to
74062eb
Compare
|
@jaykorean has imported this pull request. If you are a Meta employee, you can view this in D78862084. |
| FLAGS_remote_compaction_wait_timeout) { | ||
| if (shared_->GetRemoteCompactionResult(scheduled_job_id, result).ok()) { | ||
| return CompactionServiceJobStatus::kSuccess; | ||
| } | ||
| Env::Default()->SleepForMicroseconds( | ||
| FLAGS_remote_compaction_wait_interval); | ||
| } |
There was a problem hiding this comment.
Did you find yourself having to tune FLAGS_remote_compaction_wait_interval and FLAGS_remote_compaction_wait_timeout? Otherwise I feel like they can be hard-coded. Similar to FLAGS_remote_compaction_worker_interval.
There was a problem hiding this comment.
FLAGS_remote_compaction_worker_interval -> We may tune this in the near future
FLAGS_remote_compaction_wait_interval and FLAGS_remote_compaction_wait_timeout -> I think these two can be hardcoded.
hx235
left a comment
hx235
left a comment
There was a problem hiding this comment.
LGTM - just make sure to fix the CI stress test failure and monitor future failures in the near future.
jaykorean
force-pushed
the
remote_compaction_stress_test_
branch
3 times, most recently
from
4a2f2a9 to
b8124af
Compare
jaykorean
force-pushed
the
remote_compaction_stress_test_
branch
3 times, most recently
from
af6e5bf to
c0b8275
Compare
jaykorean
force-pushed
the
remote_compaction_stress_test_
branch
from
c0b8275 to
844b1ea
Compare
|
@jaykorean has imported this pull request. If you are a Meta employee, you can view this in D78862084. |
|
@jaykorean merged this pull request in 217e075. |
3 participants
Summary
Simulate Remote Compaction in Stress Test by running a separate set of threads that runs remote compaction.
Queue and ResultMap for the remote compactions are stored in memory as part of the
SharedState. They are shared across main worker threads and remote compaction worker threads.enable_remote_compactionis replaced byremote_compaction_worker_threads.If
remote_compaction_worker_threadsis set to 0, remote compaction is not enabled in Stress Test.To Follow up
This PR covers happy path only. Failure injection in the remote worker thread will be added as a follow up.
Test Plan