Delete file lists in slices. #10792
Delete file lists in slices. #10792
Conversation
|
Jenkins results:
|
| delResult = ctx.unlink(pfnList) | ||
|
|
||
| for pfnSlice in list(grouper(pfnList, self.msConfig["filesToDeleteSliceSize"])): | ||
| delResult.append(ctx.unlink(pfnSlice)) |
There was a problem hiding this comment.
Todor, please educate me on the behaviour of this ctx.unlink() call. I understand it will return the value None if the file was successfully deleted, right?
How about the response in case of a failure?
And does the response list maintain the same order of input files provided in the list?
I'm asking these questions to see if there is a better way to account for successful deletions (from L315).
There was a problem hiding this comment.
Hi @amaltaro , It does return None for any successful deletion, but I do not remember what was the value if otherwise. And I for sure cannot guarantee the order of deletions. So one thing is for sure, from the last time I was checking - L315 is the one and only condition to assure that all of the deletions from the bulk list of pfns have been executed successfully.
Sum deletedSuccess in the slices.
todor-ivanov
force-pushed
the
bugfix_MSUnmerged_BuffOverflow_fix-10713
branch
from
803c09c
to
041bea8
Compare
|
Jenkins results:
|
|
Jenkins results:
|
Fixes #10713
Status
not-tested
Description
One of the possible reasons for having the
Cherripythread restarting in theMSUnmergedservice is a buffer overflow of gfal2 for directories with long lists of files to be deleted. In order to prevent this to happen, with the current change we suggest to slice the list of files in chunks of 100, before we send it togfal2.Is it backward compatible (if not, which system it affects?)
YES
Related PRs
The
msconfigparameter is about to be set with a separate config and linked here.External dependencies / deployment changes
None