Investigate feasibility of creating Cleanup jobs from a central microservice

[amaltaro]

Impact of the new feature
Future microservice to cleanup unneeded unmerged files (MSUnmerged?)

Is your feature request related to a problem? Please describe.
No. We are considering to implement a new microservice to be hosted in CMSWEB, which will be responsible for fetching a dump of all the unmerged files in a given RSE (from the Consistency Checking tool), figuring out which files are no longer needed in the system, and providing those unneeded files as input to cleanup jobs submitted to one of our production schedds. Those cleanup jobs should be very similar (if not equal) to our standard production Cleanup jobs, thus running locally on the site CPU resources and deleting files "locally".

Describe the solution you'd like
This is meant to be a somehow high level investigation on how feasible it would be to reuse the WMCore/WMSpec/Steps package to create the Cleanup jobs.
We also have to explore how we could use the WMCore/Storage/Backends package to use the correct tool for the file deletions.
A third point of investigation will be on the job post processing, such that we can identify what succeeded or failed, and whether files managed to be deleted or not. One could argue that, as a first alpha product, we trust that part of the files will be deleted and there is no need to know how many jobs are succeeding, and how many files are successfully deleted.

The solution will likely be in the form of a document explaining how to plug all these pieces, what is required, etc. No need to make any implementation at this stage.

Describe alternatives you've considered
Anything else that I might have missed here?

Additional context
none

[khurtado]

EDIT:
Conclusion: This is feasible (both remote deletion or local deletion from the sites themselves). More details below:

https://github.com/dmwm/WMCore/wiki/Investigate-feasibility-of-creating-Cleanup-jobs-from-a-central-microservice

Closing this issue