Perform R&D with pileupconf data service

[vkuznet]

Impact of the new feature
WMAgent and management of jobs

Is your feature request related to a problem? Please describe.
Working on #11884 issue we faced large complexity to adjust pileup configuration files located in job's tarball. In order to make any change each individual pilupe configuration file should be extracted, changed and written back to tarball. This procedure can be significantly simplified if we'll have configuration file service.

Describe the solution you'd like
It seems to me it would be worth to explore pileup configuration service which will store pileup configuration files and provides API to change them. Here is I envision it:

• </create>/path POST HTTP api to store pileup conf file, where path represents dataset (pileup) path
• </delete>/path DELETE HTTP api to delete configuraiton file
• </search>/path GET HTTP api to search for configuration file
  NOTE: here I used optional </action> end-point (such as /<search>) for demonstration purposes but they are not strictly required if we'll follow HTTP method since each method define the action.

There are different ways to organize backend, e.g.

• use key-value store, e.g. Redis. It will only store dataset-pileup-path vs pileup json file pairs
• NoSQL database, e.g. MongoDB, if we want to have more comprehensive search over content of pileupconf.hson content
• traditional RDBMS, e.g. ORACLE, if we want additional relationship stored in DB backend

But all of these solution can be wrapped into simple HTTP based service which can be installed on CMSWEB and used by different parts, from WMAgent to data-ops queries/requests, to query, update, create, and delete pileup configuration files.

The benefits of such service is many-fold;

1. Provide common place to store configuration files
2. Provide common place to query and modify configuration files
3. Independent service from WMAgent machinery
4. Easy integration in different worklfows machineries, e.g. data-ops can change configuration on a fly, i.e. before or after workflow is placed into WMA system

Describe alternatives you've considered
Do nothing and keep using sandbox tar ball which currently holds configuration files

Additional context
This is a nice example of pure micro-service architecture where a single piece, i.e. handling configuration files, is delegated to the service. Such service has no connection with other WMCore Micro-services or WMAgent components but will be used by later comprehensively everywhere.

[vkuznet]

FYI: @amaltaro , @klannon , @todor-ivanov , @khurtado