Ensure MSTransferor does not request more copies than RSEs available #11844
Conversation
|
Jenkins results:
|
| msg = f"Found only {len(rses)} RSEs listed, hence we need to lower " | ||
| msg += f"the number of copies from {copies} to {len(rses)}" | ||
| self.logger.warning(msg) | ||
| copies = len(rses) |
There was a problem hiding this comment.
Does sticking with rses is best approach in terms of distributing the load? I fear that such static behavior will only place wf to RSEs which are used by upstream providers rather to random set of copies. I rather suggest to use randomization of copies with size of rses list here. But of course it is not required per-se (just an observation).
There was a problem hiding this comment.
The current policy is that all input data placement is carried out with a single copy, with one exception to NANO workflows, which define 2 copies for each container. In practice, if this code gets executed, the number of copies will be > 0 and < 2 (hence, 1).
In other words, adding randomization will not have any effect unless we decide to change the policy (and having a random definition of copies might not be the best behavior as well).
There was a problem hiding this comment.
thanks for explanation. Feel free to proceed.
Fixes #11843
Status
ready
Description
Some input data placement policies request more than 1 replica copies (e.g. NANOAOD workflows), hence we need to ensure that a rule does not request more copies than the actual number of RSEs listed in the RSE expression, otherwise Rucio will fail the rule creation request.
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
None
External dependencies / deployment changes
None