[Question] How to force checksum to be recomputed?

[eduardnegru]

I want to force the checksum computation server side. Is this possible in Xrootd? I know that Xrood version < 4, had a feature called recomputeMD5. How to simulate recomputeMD5 in XrootD version >= 4?

[abh3]

Currently, there is no way to automatically do a checksum calculation unless a client ask for the checksum. That happens automatically for files copied in using xrdcp with the checksum option. Other clients would need to ask for the checksum opon closing the file. Could you provide one or two scenarios for "forcing a checksum"?

…

On Fri, 20 Mar 2020, eduardnegru wrote: I want to force the checksum computation server side. Is this possible in Xrootd? I know that Xrood version < 4, had a feature called recomputeMD5. How to simulate recomputeMD5 in XrootD version >= 4? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #1160

[eduardnegru]

I have the following scenario: A client creates a file and stores it on a server. That file has a checksum associated and my intuition is that it is stored in a cache. A client wants to check if that file is corrupt by recomputing its checksum. Since checksum computation is intensive the client does not want to download the file and recompute it locally. The client would like the checksum to be recomputed server side and then sent back.

Is the checksum returned from the command below returned from a cache, or the checksum is recomputed every time the client issues that command?

xrdfs root://host:port query checksum path

If it is returned from a cache, what happens if the file gets corrupted after its original checksum is inserted in the cache?

xrdcp's cksum option allows for a validation of checksum, but it throws an error if the file is already present in the destiantion and therefore the checksum computation step is not reached. In my use case, the file is already present in destination, and I just want the server to recompute the file checksum and compare it with the one given by the client.

[xrootd-dev]

On Sat, 21 Mar 2020, eduardnegru wrote: I have the following scenario: A client creates a file and stores it on a server. That file has a checksum associated and my intuition is that it is stored in a cache. A client wants to check if that file is corrupt by recomputing its checksum. Since checksum computation is intensive the client does not want to download the file and recompute it locally. The client would like the checksum to be recomputed server side and then sent back. Is the checksum returned from the command below returned from a cache, or the checksum is recomputed every time the client issues that command? ``` xrdfs root://host:port query checksum path

If the server supports checksuming (which practically all HEP servers do), then if the checksum has not yet been computed it is computed at the time of query, recorded in the file's extended attributes, and returned to the requesting client. Subsequent queries will return what is stored in the file's attributes (i.e. it will not be recomputed). ```

If it is returned from a cache, what happens if the file gets corrupted after its original checksum is inserted in the cache?

Well, actually nothing in most cases. However, if the file is stored in a filesystem that provides data integrity for files at rest (e.g. ZFS, BtrFS, HDFS, etc) then the filesystem will detet the corruption and return an error.

xrdcp's cksum option allows for a validation of checksum, but it throws an error if the file is already present in the destiantion and therefore the checksum computation step is not reached. In my use case, the file is already present in destination, and I just want the server to recompute the file checksum and compare it with the one given by the client.

There is no way to force a server to recompute the checksum once it has been computed (well, not from any visible API). At this point you will need to read the file back and compute the checksum locally. Please feel free to add an enhancement request ticket in github if a forcible recomputation is something that is needed. Andy

[abh3]

This has moved to issue 1161 so I am closing this issue.