Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(cleanup): cleanup par + max batch size #1967

Merged
merged 9 commits into from
Dec 19, 2023
Merged

fix(cleanup): cleanup par + max batch size #1967

merged 9 commits into from
Dec 19, 2023

Conversation

gurinderu
Copy link
Contributor

@gurinderu gurinderu commented Dec 18, 2023

Description

In this PR we fix the count of cleanup batch and parallelize deletion.

Motivation

If cleanup is slow it dramatically decreases the whole system performance.

Proposed Changes

  • Make a cleanup batch smaller
  • Parallelize deletion

Additional Notes

  • Now par delete isn't limited by parallel factor, it means if the batch size is 1024 it will produce 1024 tasks and try to run them in parallel. As a result, it will spawn a lot of tasks on the blocking poll which utilize them fully, but it is ok.
  • Also introduced an ActorKey which should be used as a datastore key in NET-632

Checklist

  • The code follows the project's coding conventions and style guidelines.
  • All tests related to the changes have passed successfully.
  • Documentation has been updated to reflect the changes (if applicable).
  • All new and existing unit tests have passed.
  • I have self-reviewed my code and ensured its quality.
  • I have added/updated necessary comments to aid understanding.

Reviewer Checklist

  • Code has been reviewed for quality and adherence to guidelines.
  • Tests have been reviewed and are sufficient to validate the changes.
  • Documentation has been reviewed and is up to date.
  • Any questions or concerns have been addressed.

@gurinderu gurinderu changed the title Cleanup max size fix(cleanup): https://github.com/fluencelabs/nox/pull/1967/files Dec 18, 2023
@gurinderu gurinderu changed the title fix(cleanup): https://github.com/fluencelabs/nox/pull/1967/files fix(cleanup): cleanup max batch size Dec 18, 2023
@gurinderu gurinderu added the e2e Run e2e workflow label Dec 18, 2023
@gurinderu gurinderu marked this pull request as ready for review December 19, 2023 09:40
@gurinderu gurinderu requested review from justprosh, folex and kmd-fl and removed request for justprosh December 19, 2023 09:40
@gurinderu gurinderu changed the title fix(cleanup): cleanup max batch size fix(cleanup): cleanup par + max batch size Dec 19, 2023
aquamarine/src/plumber.rs Outdated Show resolved Hide resolved
aquamarine/src/plumber.rs Show resolved Hide resolved
aquamarine/src/plumber.rs Outdated Show resolved Hide resolved
@@ -106,24 +107,29 @@ impl ParticleDataStore {
}

pub async fn batch_cleanup_data(&self, data: Vec<(String, PeerId)>) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is data: Vec<(String, PeerId)>?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pairs of the particle id and worker id

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's rename data argument to something sensible, otherwise it doesn't make any sense. Reads like you're going to clean up Vec<(String, PeerId)>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@gurinderu gurinderu enabled auto-merge (squash) December 19, 2023 18:04
@folex folex disabled auto-merge December 19, 2023 19:25
@folex folex merged commit 1063eae into master Dec 19, 2023
3 of 7 checks passed
@folex folex deleted the cleanup-max-size branch December 19, 2023 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
e2e Run e2e workflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants