Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve disk vs tape source prioritization #4906

Closed
rcarpa opened this issue Oct 13, 2021 · 0 comments
Closed

improve disk vs tape source prioritization #4906

rcarpa opened this issue Oct 13, 2021 · 0 comments
Assignees
@rcarpa
Labels
enhancement Transfers
Milestone
1.26.6

Comments

@rcarpa
Copy link
Contributor

rcarpa commented Oct 13, 2021

Motivation

a source which fails a transfer is de-prioritized rapidly. The goal is to avoid trying idefinetely a source which has issues, and to try other sources. Assuming 2 sources : a tape; and a disk one. The disk source will be tried first, if it fails, at next iteration we'll try the second (tape) source. This behavior is not perfect. A transitory issue on the disk source will result in mounting the tape.

Modification

As a first step, just ensure that the disk source is retried more than once before going to tape.
A second improvement would be to detect that type of disk source error. If it's a recoverable error, retry disk; If it's a non-recoverable error, useless to retry that source.
@rcarpa rcarpa self-assigned this Oct 13, 2021
rcarpa added a commit to rcarpa/rucio that referenced this issue Oct 13, 2021
@rcarpa
Transfers: reduce priority of tape sources. rucio#4906
e7eb6e7 
Avoid going to tape sources immediately after failing a transfer from
disk. This is to avoid mounting the tape just because a transfer fails
because of a transitory issue with a disk source.
bari12 added a commit that referenced this issue Oct 19, 2021
@bari12
Merge pull request #4907 from rcarpa/patch-4906-improve_disk_vs_tape_…
d36d346 
…priority

Transfers: reduce priority of tape sources. #4906
bari12 pushed a commit that referenced this issue Oct 19, 2021
@rcarpa @bari12
Transfers: reduce priority of tape sources. #4906
3e237e1 
Avoid going to tape sources immediately after failing a transfer from
disk. This is to avoid mounting the tape just because a transfer fails
because of a transitory issue with a disk source.
@bari12 bari12 added this to the 1.26.6 milestone Oct 19, 2021
@bari12 bari12 changed the title Transfers: improve disk vs tape source prioritization improve disk vs tape source prioritization Oct 19, 2021
@bari12 bari12 closed this as completed Oct 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcarpa rcarpa

Labels
enhancement Transfers
Projects
None yet
Milestone
1.26.6
Development

No branches or pull requests
2 participants
@bari12 @rcarpa