Refactor LocationUpdateTask GQ CherryPy thread to perform pileup data location based on blocks

[amaltaro]

Impact of the new feature
Global WorkQueue CherryPy thread

Is your feature request related to a problem? Please describe.
This feature needs to be deployed together with: #11620

As we move from full to partial pileup container support within WM, we need to ensure that the Global WorkQueue LocationUpdateTask thread starts executing pileup location look-up based on the pileup blocks (rucio datasets), such that their location can be properly reflected in the workqueue elements.

Describe the solution you'd like
Granularity of the pileup data location needs to be refactored from Rucio container to dataset level. In addition, we need to adopt a concurrent library to ensure that this task can properly scale with the very large pileup containers (e.g. 30k rucio datasets).

Data availability follows this logic:

• if the workqueue element is set to NoPileupUpdate=true, then we do not need to perform and data location update; otherwise
• each RSE that hosts at least 1 rucio dataset should be marked as an actual location for that pileup dataset.

Note that if http requests fail, there is nothing to be done and we retry it in the next cycle.

Describe alternatives you've considered
For the http library, we can either use pycurl_manager or asyncio.

Additional context
This is part of the meta issue: #11537

[d-ylee]

Code would be here: https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/GlobalWorkQueue/CherryPyThreads/LocationUpdateTask.py

[amaltaro]

Based on this comment: #11619 (comment)
I feel like we should rely on MSPileup to fetch an up-to-date pileup data location.

The MSPileup REST API:
https://cmsweb.cern.ch/ms-pileup/data/pileup

data provides currentRSEs for each pileup data. This is where the pileup is currently available.

In addition, if a pileup name is not defined in MSPileup database, its location should be reported as an empty list [].

[d-ylee]

@amaltaro Would I use the MSPileup API in the GQ CherryPy thread to update the location?

[amaltaro]

@d-ylee Hi Dennis, yes. I am in favor of adopting MSPileup data everywhere in the WM system that pileup location needs to be resolved.
For this CherryPy thread, it will likely be one HTTP call to fetch the pileup information; the rest is only local processing of that data and the usual pileup location update within the workflows (workqueue elements).

If you prefer, I am happy to have a chat to go over the details as well. Just ping me on Mattermost.