Skip to content

Commit

Permalink
[ISSUE-173][FOLLOWUP] The size of single buffer flush should reach rs…
Browse files Browse the repository at this point in the history
…s.server.flush.cold.storage.threshold.size (#178)

###What changes were proposed in this pull request?
follow  #173, when single buffer flush is triggered, the buffer size should reach `rss.server.flush.cold.storage.threshold.size`,  we keep the same logic as MultiStorageManager#selectStorageManager

###Why are the changes needed?
we keep the same logic as MultiStorageManager#selectStorageManager

###Does this PR introduce any user-facing change?
No

###How was this patch tested?
Already added

Co-authored-by: leixianming <leixianming@didiglobal.com>
  • Loading branch information
leixm and leixianming committed Aug 22, 2022
1 parent 805a372 commit ab6be6e
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,9 @@ public ShuffleDataResult getShuffleData(

void flushSingleBufferIfNecessary(ShuffleBuffer buffer, String appId,
int shuffleId, int startPartition, int endPartition) {
if (this.bufferFlushEnabled && buffer.getSize() >= this.bufferFlushThreshold) {
// When we use multistorage and trigger single buffer flush, the buffer size should be bigger
// than rss.server.flush.cold.storage.threshold.size, otherwise cold storage will be useless.
if (this.bufferFlushEnabled && buffer.getSize() > this.bufferFlushThreshold) {
flushBuffer(buffer, appId, shuffleId, startPartition, endPartition);
}
}
Expand Down

0 comments on commit ab6be6e

Please sign in to comment.