Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add traffic shaper for download tasks #1654

Merged
merged 6 commits into from
Sep 28, 2022

Conversation

greenhandatsjtu
Copy link
Contributor

@greenhandatsjtu greenhandatsjtu commented Sep 6, 2022

Signed-off-by: greenhandatsjtu sunhengke@sjtu.edu.cn

Description

For now, when a peer download from seed peers, the speed is limited by perPeerRateLimit and totalRateLimit.
However, when there's only one task running with perPeerRateLimit set to 20M and totalRateLimit set to 100M, then 80% of the bandwidth is wasted. That's why we introduce traffic shaper.
With the help of sampling traffic shaper, the rate limit of every task can be dynamically adjusted based on the context, the download speed of all tasks in the past second, which significantly improves the download performance.

Related Issue

Motivation and Context

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation Update (if none of the other choices apply)

Checklist

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
@codecov-commenter
Copy link

codecov-commenter commented Sep 6, 2022

Codecov Report

Merging #1654 (2fef1de) into main (5d5ea63) will increase coverage by 0.19%.
The diff coverage is 84.37%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1654      +/-   ##
==========================================
+ Coverage   42.03%   42.22%   +0.19%     
==========================================
  Files         125      126       +1     
  Lines       15436    15594     +158     
==========================================
+ Hits         6488     6585      +97     
- Misses       8432     8491      +59     
- Partials      516      518       +2     
Flag Coverage Δ
Object-compatibility-e2etests ?
e2etests ?
unittests 42.22% <84.37%> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
client/config/peerhost.go 35.73% <ø> (ø)
client/daemon/peer/peertask_manager.go 26.08% <50.00%> (+0.41%) ⬆️
client/daemon/peer/traffic_shaper.go 86.01% <86.01%> (ø)
client/daemon/peer/peertask_conductor.go 57.61% <100.00%> (-0.23%) ⬇️
client/daemon/peer/peertask_piecetask_poller.go 32.78% <0.00%> (-14.21%) ⬇️
client/daemon/rpcserver/subscriber.go 48.85% <0.00%> (-1.73%) ⬇️
client/daemon/peer/piece_manager.go 52.83% <0.00%> (-0.14%) ⬇️

client/daemon/peer/peertask_manager.go Outdated Show resolved Hide resolved
client/daemon/peer/peertask_conductor.go Outdated Show resolved Hide resolved
client/daemon/peer/peertask_conductor.go Outdated Show resolved Hide resolved
client/daemon/peer/traffic_shaper.go Show resolved Hide resolved
client/daemon/peer/traffic_shaper.go Outdated Show resolved Hide resolved
client/daemon/peer/traffic_shaper.go Outdated Show resolved Hide resolved
client/daemon/peer/traffic_shaper.go Outdated Show resolved Hide resolved
client/daemon/peer/traffic_shaper.go Outdated Show resolved Hide resolved
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
Copy link
Member

@gaius-qi gaius-qi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gaius-qi gaius-qi merged commit e33ad5a into dragonflyoss:main Sep 28, 2022
@greenhandatsjtu greenhandatsjtu deleted the feat/traffic-shaper branch September 28, 2022 13:18
gaius-qi pushed a commit that referenced this pull request Jun 28, 2023
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants