WMAgent: Implement Schema validation functionalities

[todor-ivanov]

Impact of the new feature
WMAgent

Is your feature request related to a problem? Please describe.
As part of the process of migration to Docker containers for WMAgent, we had to refactor also the whole deployment and initialization process. Previously as part of this process was a mandatory step for manually cleaning the whole backend relational database for the agent. This was happening upon a long draining process and during the full WMAgent redeployment.

The migration to docker gives us the benefit of logical separation of the WMAgent services in terms of deployment procedures. We currently have 3 independent containers:

• WMAgent
• MariaDB
• CouchDB

This opens a good opportunity to reuse previous instances of the same database in the process of WMAgent container redeployment of which we should take advantage. This is basically preserving both::

• All previously run workflows' run and state files archived inside the agent (at the dockerMount point on the host)
• All previously run workflows' records in the backend relational database.

The above are the two needed conditions to avoid the long draining process (almost a month) before upgrading our agents. But there is one mandatory check that needs to happen before we let all this fly. We must crosscheck at startup time on every agent, if the relational database schema of the WMAgent version of the ramping agent is compatible (basically the same) with the one that is currently present at the database Server where the agent is about to connect.

A rudimentary check based on mysqldump and a simple diff: ( manage-common.sh#L66 and manage-common.sh#L83 respectively), was implemented for the MariaDB agents with satisfying success. But there are few downsides of this implementation:

• The respective tools for Oracle doing the same job exp and expdb create binary files which are, first impossible to compare with a simple diff, second they require the installation of Oracle instant client, and third they preserve all those binary files at the server, where we have no access to them. Alternative approach was suggested by our Oracle contact and described here: WMAgent: Implement all Oracle functionalities needed for the WMAgent initialization  #11720 (comment)
• The initial schema file is created during the first initialization of the agent and compared on every subsequent restart. This gives us the ability to basically check only if there have been any schema change between the first initialization of a container and any subsequent restart, but does not give us the possibility to do schema validation across versions. The mechanism is fully implemented in the set of initialization scripts we have created for the new containerized model, and the initial schema file is created here , but the schema file for every version should be created inside the WMCore code and updated with the version, so that it could always be used as a starting point.

Describe the solution you'd like
A solution which:

• Would be equally applicable to both Oracle and MariadB agents
• Which would provide schema validation mechanism across WMAgent versions

In order for this to happen, we must search for a method of creating a pattern schema associated with the WMAgent version inside the WMCore code, and update it with the version. This may require a change of the current mechanism of database initialization.

Describe alternatives you've considered
Do nothing and continue the procedure of wiping out the the whole database during every agent initialization. This:

• Is a process prone to human errors (even though with the containerized approach we have some additional check implemented)
• This holds us back to the extremely long process of draining agents between version upgrades and keeps the operational load on the Team quite high.

Additional context

Part of the following meta issue:

• Deploy and run WMAgent with Docker container #11314