Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The Dark Reaper should run continuously instead of exiting early when there’s nothing to do #4377

Closed
dchristidis opened this issue Mar 4, 2021 · 2 comments · Fixed by #4379
Closed

The Dark Reaper should run continuously instead of exiting early when there’s nothing to do #4377

dchristidis opened this issue Mar 4, 2021 · 2 comments · Fixed by #4379
Assignees
@dchristidis
Labels
Consistency checks enhancement
Milestone
1.25.1

Comments

@dchristidis
Copy link
Contributor

Motivation

The Dark Reaper has its own implementation of list_rses() which only returns RSEs that have at least one quarantined replica. This is not optimal for two reasons:

  1. If there are no quarantined replicas, then the Reaper exists early. Daemons running on Kubernetes are expected to run continuously.
  2. If quarantined replicas are added for an RSE that didn’t have any when the daemon started, then they won’t be processed until the next restart.

Modification

Replace the custom list_rses() implementation with the one from the core.
@dchristidis dchristidis self-assigned this Mar 4, 2021
dchristidis added a commit to dchristidis/rucio that referenced this issue Mar 4, 2021
@dchristidis
Consistency: Allow the Dark Reaper to run coninuously rucio#4377
12d9bf6 
This is a quick and dirty solution to running the Dark Reaper on
Kubernetes. We can improve it with some refactoring in the future.
@dchristidis dchristidis mentioned this issue Mar 4, 2021
Merged
dchristidis added a commit to dchristidis/rucio that referenced this issue Mar 4, 2021
@dchristidis
Consistency: Allow the Dark Reaper to run continuously rucio#4377
718ee58 
This is a quick and dirty solution to running the Dark Reaper on
Kubernetes. We can improve it with some refactoring in the future.
bari12 added a commit that referenced this issue Mar 16, 2021
@bari12
Merge pull request #4379 from dchristidis/patch-4377-Consistency__All…
dda450a 
…ow_the_Dark_Reaper_to_run_continuously

Consistency: Allow the Dark Reaper to run continuously #4377
bari12 pushed a commit that referenced this issue Mar 16, 2021
@dchristidis @bari12
Consistency: Allow the Dark Reaper to run continuously #4377
e65be09 
This is a quick and dirty solution to running the Dark Reaper on
Kubernetes. We can improve it with some refactoring in the future.
@bari12 bari12 added this to the 1.25.1 milestone Mar 16, 2021
dchristidis added a commit to dchristidis/rucio that referenced this issue Apr 7, 2021
@dchristidis
Consistency: Do not overwrite the list of RSEs rucio#4377
f65d77a 
Otherwise, the original list is lost and the Dark Reaper will not pick
up newly-added quarantined replicas.
dchristidis added a commit to dchristidis/rucio that referenced this issue Apr 7, 2021
@dchristidis
Consistency: Correct function definition and documentation rucio#4377
86725c9
bari12 pushed a commit that referenced this issue Apr 12, 2021
@dchristidis
Consistency: Allow the Dark Reaper to run continuously #4377 (#4524)
fd32a44 
* Consistency: Do not overwrite the list of RSEs #4377

Otherwise, the original list is lost and the Dark Reaper will not pick
up newly-added quarantined replicas.

* Consistency: Correct function definition and documentation #4377
bari12 pushed a commit that referenced this issue Apr 12, 2021
@dchristidis @bari12
Consistency: Allow the Dark Reaper to run continuously #4377 (#4524)
bf1385d 
* Consistency: Do not overwrite the list of RSEs #4377

Otherwise, the original list is lost and the Dark Reaper will not pick
up newly-added quarantined replicas.

* Consistency: Correct function definition and documentation #4377
@ericvaandering
Copy link
Contributor

@dchristidis We still see issues with this in 1.25.2. The DR runs continuously, it just refuses to pick up new work from the database.

More specifically, we add a file to quarantined_replicas for Purdue and start DR. It deletes the file. We add another file to QR and DR says it has nothing to do. Then we restart DR and it picks up the work.

Thoughts?

@dchristidis
Copy link
Contributor Author

Rucio 1.25.1 introduced a partial fix (#4379): the Dark Reaper could now start even if the quarantined replicas table were empty. However, due to an oversight, it still wouldn’t process newly-added replicas.
Rucio 1.25.3 contains an additional fix (#4524) which should correctly address the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dchristidis dchristidis

Labels
Consistency checks enhancement
Projects
None yet
Milestone
1.25.1
Development

Successfully merging a pull request may close this issue.

Consistency: Allow the Dark Reaper to run continuously #4377 dchristidis/rucio
3 participants
@bari12 @ericvaandering @dchristidis