Replicas registration: removing adler32 requirement in add_replica methods #3494 #3538
Conversation
| @@ -167,7 +168,7 @@ def list_replicas(self, dids, schemes=None, unavailable=False, | |||
| exc_cls, exc_msg = self._get_exception(headers=r.headers, status_code=r.status_code, data=r.content) | |||
| raise exc_cls(exc_msg) | |||
|
|
|||
| def add_replica(self, rse, scope, name, bytes, adler32, pfn=None, md5=None, meta={}): | |||
| def add_replica(self, rse, scope, name, bytes, adler32=None, pfn=None, md5=None, meta={}): | |||
There was a problem hiding this comment.
I think in principle this is fine now, however, it would allow adding replicas without any checksum at all.
Currently the schema validation would allow that.
rucio/lib/rucio/api/replica.py
Line 202 in 6df1b03
It does call the schema validation there, but since it compares with
dids where adler32 is not mandatory (due to datasets/containers) we can end up also with files without checksums.@davidgcameron @mlassnig @cserf @dchristidis would be good to check this too. I am not sure if we want to allow that, I think there needs to be at least an option to require one, or the other, checksum.
There was a problem hiding this comment.
Can the requirement be specified via a community-specific policy or configuration? For ATLAS checksums should be enforced but I can imagine other cases where it should be optional, like registering files residing on a storage which doesn't support checksums, mutable files, or pseudo-files like say a dynamic web page. I think we should be flexible and not restrict the way rucio can be (ab)used by any community :)
There was a problem hiding this comment.
I think we should do that. There is a few possibilities, one of them would be to place it in the schema (Which can already be community specific). The difficulty with that is that we would have to change this quite a bit though, since for now you could not just make the adler32 field mandatory. (That part of the schema is validated for datasets/containers too, and would fail them, since they do not have checksums)
Another option is to put this into rucio.cfg or config table. Something like required_checksum.
I am a little bit more leaning to the schema direction, since this makes this work natively also for multi-vo mode, where different VOs might have different requirements.
There was a problem hiding this comment.
I would like to add few cents.
I'm not sure, but can you confirm making one checksum mandatory might limit the functionalities of per-RSE checksum compatibility?
In that case enforcing adler32 will throw away compatibility with, e.g., legacy gridftp sites (the triggering reason for the per-RSE compatibility).
For this reason I'd tend to the solution which would check all the provided values and verify that there is at least one, more than having a given algorithm set up statically as VO o deployment setting.
If this is not the case, then the per VO setting seems to me the best one, while having to interact with the schema is in mi mind not (read never) ideal.
There was a problem hiding this comment.
I was not following this closely, but just to throw my few cents into the discussion.
There should be a way to express the following things:
- "every file must have a checksum, doesn't matter which one"
- "every file must have a checksum, and it must be expliclity this one"
- "no checksums are needed"
- "no checksums are allowed on special RSEs"
- "no checksums are allowed on special files"
I guess a combination of both approaches makes sense here. A schema validation for the global "decision", and then more specific attributes to express the more fine-grained selections.
gabrielefronze
force-pushed
the
feature-3494
branch
from
dca34f1
to
12ebcea
Compare
Auto stash before rebase of "refs/heads/feature-3494"
gabrielefronze
force-pushed
the
feature-3494
branch
from
12ebcea
to
7cf42a9
Compare
|
Closing; Currently re-discussing a different approach |
Replicas registration: removing adler32 requirement in add_replica methods #3494