Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

passing log candidates as batch for and copy and delete #7375

Merged
merged 1 commit into from
Aug 3, 2023

Conversation

achouhan09
Copy link
Member

@achouhan09 achouhan09 commented Jul 3, 2023

Explain the changes

  1. Currently one log object at a time is passed for copy and delete. In this PR we are trying to pass a batch of objects for both copy and delete individually.

@achouhan09 achouhan09 requested review from dannyzaken, tangledbytes, a team and nbecker-cibot and removed request for a team July 3, 2023 15:26
@achouhan09 achouhan09 marked this pull request as draft July 3, 2023 15:26
@achouhan09 achouhan09 changed the title passing log candidates as batch of 100 for and copy and delete passing log candidates as batch for and copy and delete Jul 3, 2023
@achouhan09 achouhan09 force-pushed the batch-repl branch 3 times, most recently from 83d50d5 to 787eb23 Compare July 6, 2023 05:48
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jul 6, 2023
@achouhan09 achouhan09 marked this pull request as ready for review July 7, 2023 10:25
@achouhan09 achouhan09 force-pushed the batch-repl branch 3 times, most recently from 7bfa145 to e569d0e Compare July 10, 2023 10:58
@achouhan09 achouhan09 marked this pull request as draft July 11, 2023 11:45
@pull-request-size pull-request-size bot added size/M and removed size/L labels Jul 12, 2023
@achouhan09 achouhan09 marked this pull request as ready for review July 12, 2023 12:16
@achouhan09 achouhan09 force-pushed the batch-repl branch 2 times, most recently from d4fb06e to f041762 Compare July 12, 2023 13:25
@achouhan09 achouhan09 force-pushed the batch-repl branch 3 times, most recently from 1f06e01 to a4b5719 Compare July 14, 2023 08:09
@achouhan09 achouhan09 force-pushed the batch-repl branch 3 times, most recently from 7c08428 to 3d8f875 Compare July 24, 2023 14:48
@achouhan09 achouhan09 marked this pull request as ready for review July 24, 2023 14:49
// if there is already a candidate for the same key, then we have a conflict
if (logs_for_key_time_map.has(log.time)) {
// If there is already a candidate for the same key, then we have a conflict
if (logs_for_key_time_map.has(log_time)) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Utkarsh-pro
Just for information while writing the unit tests I checked that this condition is not working as expected because of strict comparison, we are comparing two different time objects here which put this condition to always false state. Updated the code with string comparison now working fine.
Pls let me know if you have any more thoughts. Thanks

@achouhan09 achouhan09 force-pushed the batch-repl branch 3 times, most recently from 11534bd to 7921081 Compare July 26, 2023 05:23
@achouhan09 achouhan09 force-pushed the batch-repl branch 3 times, most recently from cf4191f to 7b96749 Compare July 27, 2023 11:55
@achouhan09 achouhan09 force-pushed the batch-repl branch 6 times, most recently from ab96dd8 to 66c4423 Compare July 31, 2023 11:24
for (const candidate of src_dst_objects_list) {
await this.process_candidate(src_bucket, dst_bucket, candidate);
const action = await this.process_candidate(candidate);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unless I missed anything, process_candidate doesn't need to be async and there is no reason to wait for it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes right, no need to async updated the code. Thanks

Signed-off-by: Aayush Chouhan <aayush.chouhan97@gmail.com>
@achouhan09 achouhan09 merged commit 69a0233 into noobaa:master Aug 3, 2023
7 checks passed
@achouhan09 achouhan09 deleted the batch-repl branch August 3, 2023 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants