Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

list_and_mark_unlocked_replicas() should also order by updated_at #6009

Closed
dchristidis opened this issue Dec 1, 2022 · 0 comments · Fixed by #6010
Closed

list_and_mark_unlocked_replicas() should also order by updated_at #6009

dchristidis opened this issue Dec 1, 2022 · 0 comments · Fixed by #6010
Assignees
@dchristidis
Labels
Deletion enhancement
Milestone
1.30.1

Comments

@dchristidis
Copy link
Contributor

Description

The sorting is currently done using the tombstone column:

rucio/lib/rucio/core/replica.py

Lines 2681 to 2683 in c2b415b

).order_by(
models.RSEFileAssociation.tombstone
).with_for_update(

When deleting rules with purge_replicas=True, the tombstone is set to the Unix Epoch. The order of replicas with identical tombstone is undefined.

Motivation

In operations, when an RSE is getting full, it might be possible to improve the situation by forcing the deletion of the largest unlocked replicas. However, if there are already a lot of small replicas with Epoch tombstone, the procedure might be ineffective.

Ordering by updated_at would allow some form of control over this.

Change

Order by tombstone, then by updated_at.
@dchristidis dchristidis self-assigned this Dec 1, 2022
@dchristidis dchristidis linked a pull request Dec 1, 2022 that will close this issue
Deletion: Order unlocked replicas also by updated_at #6009 #6010
Merged
bari12 pushed a commit that referenced this issue Dec 14, 2022
@dchristidis
Deletion: Order unlocked replicas also by updated_at #6009
9eb72fb 
The main motivation behind this comes from operations. When conducting
deletion campaigns involving many small-sized files (e.g. logs), in case
the rate of deletion is lower than the rate of transfer, operators may
opt to force the deletion of larger replicas in order to alleviate the
situation. However, if both small and large replicas have a Unix Epoch
tombstone, there is no way to influence the order by which they will be
deleted. Modifying the query to order by tombstone, then by updated_at,
removes this limitation.
bari12 added a commit that referenced this issue Dec 14, 2022
@bari12
Merge pull request #6010 from dchristidis/patch-6009-Deletion__Order_…
428c067 
…unlocked_replicas_also_by_updated_at

Deletion: Order unlocked replicas also by updated_at #6009
@bari12 bari12 added this to the 1.30.1 milestone Dec 14, 2022
bari12 pushed a commit that referenced this issue Dec 14, 2022
@dchristidis @bari12
Deletion: Order unlocked replicas also by updated_at #6009
d9868ed 
The main motivation behind this comes from operations. When conducting
deletion campaigns involving many small-sized files (e.g. logs), in case
the rate of deletion is lower than the rate of transfer, operators may
opt to force the deletion of larger replicas in order to alleviate the
situation. However, if both small and large replicas have a Unix Epoch
tombstone, there is no way to influence the order by which they will be
deleted. Modifying the query to order by tombstone, then by updated_at,
removes this limitation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dchristidis dchristidis

Labels
Deletion enhancement
Projects
None yet
Milestone
1.30.1
Development

Successfully merging a pull request may close this issue.

Deletion: Order unlocked replicas also by updated_at #6009 dchristidis/rucio
2 participants
@bari12 @dchristidis