Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto-tuning concurrency and buffer size related parameters depending on remote storage type #658

Closed
Slach opened this issue May 21, 2023 · 0 comments · Fixed by #660
Closed

Auto-tuning concurrency and buffer size related parameters depending on remote storage type #658

Slach opened this issue May 21, 2023 · 0 comments · Fixed by #660
Assignees
@Slach
Milestone
2.2.7

Comments

@Slach
Copy link
Collaborator

Slach commented May 21, 2023

  • calculate upload_concurrency as round(sqrt(MAX_CPU/2) only for remote storage type which supports SDK level concurrency for one upload stream and calculate it as MAX_CPU/2 for storage which not support SDK level concurrency
  • always calculate download_concurrency as MAX_CPU/2
  • calculate SDK level concurrency as upload_concurrency + 1
  • decrease max_parts_count for storage types which available this option from 10000 to 5000 by default
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Slach Slach

Labels
None yet
Projects
None yet
Milestone
2.2.7
Development

Successfully merging a pull request may close this issue.

Auto-tuning concurrency and buffer size related parameters depending on remote storage type
1 participant
@Slach