Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collection schedule by user #2369

Merged
merged 23 commits into from May 1, 2023
Merged

Collection schedule by user #2369

merged 23 commits into from May 1, 2023

Conversation

IsaacMilarky
Copy link
Contributor

@IsaacMilarky IsaacMilarky commented Apr 28, 2023

Description
Add a mechanism to enforce a standard of fairness between collection instances with multiple users. This is done in two parts:

Part A (Handle repos that haven't been collected):

  1. Split up all users that have new repos into 4 randomly assigned groups
  2. For each group:
    • query a set of new repos that belong to this group of users sorted by lowest weight first
    • Start repos up to the limit, break if the max_repo limit is reached

Part B (Handle repos that have been collected before):

  1. Split up all users into 4 randomly assigned groups
  2. For each group:
    • query a set of old repos that belong to this group of users sorted by lowest weight first
    • Start repos up to the limit, break if the max_repo limit is reached

This mechanism is used for all collection hooks: core, secondary, and facade.

@ABrain7710 and I developed this general algorithm over a meeting this week

Signed commits

  • Yes, I signed my commits.

… startup

Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
augur/tasks/util/collection_util.py Show resolved Hide resolved
augur/tasks/util/collection_util.py Outdated Show resolved Hide resolved
augur/tasks/util/collection_util.py Outdated Show resolved Hide resolved
augur/tasks/util/collection_util.py Outdated Show resolved Hide resolved
augur/tasks/util/collection_util.py Show resolved Hide resolved
augur/tasks/util/collection_util.py Outdated Show resolved Hide resolved
augur/tasks/util/collection_util.py Outdated Show resolved Hide resolved
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
ABrain7710
ABrain7710 previously approved these changes May 1, 2023
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
@IsaacMilarky IsaacMilarky merged commit 508be1f into dev May 1, 2023
1 check passed
@IsaacMilarky IsaacMilarky deleted the collection-schedule-by-user branch May 1, 2023 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants