Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-Write of Conveyor-Throttler and the throtteling system #4056

Closed
bari12 opened this issue Oct 2, 2020 · 0 comments · Fixed by #4079
Closed

Re-Write of Conveyor-Throttler and the throtteling system #4056

bari12 opened this issue Oct 2, 2020 · 0 comments · Fixed by #4079
Assignees
@bziemons
Labels
feature Overview Used for overview issues, linking to other issues Transfers
Projects
Rucio 1.24 "Aqua Donkey"
Milestone
1.24.0

Comments

@bari12
Copy link
Member

bari12 commented Oct 2, 2020

Motivation

Due to scalability issue this entire workflow needs to be re-written.
@bari12 bari12 added feature Transfers Overview Used for overview issues, linking to other issues labels Oct 2, 2020
@bari12 bari12 added this to the 1.24.0 "Aquadonkey" milestone Oct 2, 2020
@bari12 bari12 added this to In progress in Rucio 1.24 "Aqua Donkey" Oct 2, 2020
bziemons added a commit to bziemons/rucio that referenced this issue Oct 16, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
676731d
bziemons added a commit to bziemons/rucio that referenced this issue Oct 20, 2020
@bziemons
Transfers: Prepare database, add simple test; rucio#4056
a0881d1
bziemons added a commit to bziemons/rucio that referenced this issue Oct 22, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
e9da9ff 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Oct 27, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
e03a89a
bziemons added a commit to bziemons/rucio that referenced this issue Oct 27, 2020
@bziemons
Transfers: Prepare database, add simple test; rucio#4056
20ab03e
bziemons added a commit to bziemons/rucio that referenced this issue Oct 27, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
8888587 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Oct 28, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
50a4b52 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Oct 28, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
179a57d
bziemons added a commit to bziemons/rucio that referenced this issue Oct 28, 2020
@bziemons
Transfers: Prepare database, add simple test; rucio#4056
f1083e0
bziemons added a commit to bziemons/rucio that referenced this issue Oct 28, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
e32d510 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Oct 28, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
f88dbd7 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Oct 29, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
aac888a
bziemons added a commit to bziemons/rucio that referenced this issue Oct 29, 2020
@bziemons
Transfers: Prepare database, add simple test; rucio#4056
72d37e3
bziemons added a commit to bziemons/rucio that referenced this issue Oct 29, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
591476f 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 5, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
9b5752f 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 5, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
a2ee1fb
bziemons added a commit to bziemons/rucio that referenced this issue Nov 5, 2020
@bziemons
Transfers: Prepare database, add simple test; rucio#4056
a80bbfe
bziemons added a commit to bziemons/rucio that referenced this issue Nov 5, 2020
@bziemons
Transfers: Add tests for preparer; rucio#4056
c2118ea 
Implement throttling in preparer and disable/remove throttler tests for now.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 9, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
e61046f 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 9, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
7ee9d6c 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 9, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
a959f61 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 9, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
485fbf5 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 10, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
dddffcf 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
@bziemons bziemons linked a pull request Nov 10, 2020 that will close this issue
Transfers: Add conveyor-preparer daemon; #4056 #4079
Merged
bziemons added a commit to bziemons/rucio that referenced this issue Nov 10, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
566107e 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 10, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
09897a3 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 12, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
96baf5a 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 12, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
85fe2e2 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 12, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
6127745 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 12, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
7296fff 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 12, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
69f6156 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 16, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
6847ac6 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 16, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
21f331d 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Nov 30, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
45041e0 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Dec 1, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
6afac9d 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Dec 1, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
7ffb32c 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
bziemons added a commit to bziemons/rucio that referenced this issue Dec 1, 2020
@bziemons
fixup! Transfers: Add preparer daemon; Fix rucio#4056
6c338e0
bziemons added a commit to bziemons/rucio that referenced this issue Dec 1, 2020
@bziemons
fixup! Transfers: Add preparer daemon; Fix rucio#4056
5418348
bziemons added a commit to bziemons/rucio that referenced this issue Dec 1, 2020
@bziemons
fixup! Transfers: Add preparer daemon; Fix rucio#4056
d0786be
bziemons added a commit to bziemons/rucio that referenced this issue Dec 1, 2020
@bziemons
fixup! Transfers: Add preparer daemon; Fix rucio#4056
0808ee6
bziemons added a commit to bziemons/rucio that referenced this issue Dec 1, 2020
@bziemons
Transfers: Add preparer daemon; Fix rucio#4056
9f8af47 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
Rucio 1.24 "Aqua Donkey" automation moved this from In progress to Done Dec 1, 2020
bari12 added a commit that referenced this issue Dec 1, 2020
@bari12
Merge pull request #4079 from bziemons/feature-4056-add_conveyor_prep…
f61390c 
…arer

Transfers: Add conveyor-preparer daemon; #4056
maany pushed a commit to maany/rucio that referenced this issue Jan 21, 2021
@bziemons @maany
Transfers: Add preparer daemon; Fix rucio#4056
d999c74 
Searches for minimum distance source and sets QUEUED or WAITING state
based on throttler mode and limits.
Add PREPARING request state.
Add conveyor.use_preparer config option.
Add tests for preparer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bziemons bziemons

Labels
feature Overview Used for overview issues, linking to other issues Transfers
Projects
No open projects
Rucio 1.24 "Aqua Donkey"
Done
Milestone
1.24.0
Development

Successfully merging a pull request may close this issue.

Transfers: Add conveyor-preparer daemon; #4056 bziemons/rucio
2 participants
@bari12 @bziemons