Schema Builder

[maany]

Is your feature request related to a problem? Please describe.
We'd like to create smaller simpler schemas that can be combined into larger and more complex schemas.

Describe the solution you'd like
Consider the following schema files (must be present at config_validation_engine/schemas):

Filename: lightweight_component.schema

lightweight_component:
  name: {type: str(), required: true, use_default: false}
  config: {type: from_config_schema(), required: true, use_default: false}

Filename: voms_config.schema

voms_config: 
  voms_fqan: { type: str(), required: true, use_default: false }
  pool_accounts: {**type: list(include('pool_account'))**, requried: true, use_default: false}
  vo: {**type: virtual_organization()**, required: true, use_default: false}
  comment: { type: str(), required: true, use_default: false }

Filename: pool_account.schema

pool_account:  
  base_name: { type:str(), required: true, use_default: true}
  initial_uid: { type: int(min=10000), required: true, use_default: false }
  users_num: { type: int(), required: true, use_default: true }
  primary_group: { type: include('linux_group'), required: true, use_default: true }

and so on....

We'd like to generate a schema file as follows:

lightweight_component:
  name: {type: str(), required: true, use_default: false}
  config: {type: from_config_schema(), required: true, use_default: true}
---
voms_config:
  voms_fqan: { type:str(), 
  pool_accounts: list(include('pool_account'))
  vo: include('virtual_organization')
  comment: str()
---
  base_name: str()
  initial_uid: int(min=10000)
  users_num: int()
  primary_group: linux_group()
---
voms_config:
  voms_fqan: str()
  pool_accounts: list(include('pool_account'))
  vo: include('virtual_organization')
  comment: str()
---
.... and so on..

To achieve this funcionality, we can

• create a new python package called schema in the root dir
• create a new python class called SchemaElement
• create a new python class called Schema

The SchemaElement class can be used as a wrapper for all the yaml files that represent smaller schemas for various components of a site level configuration file.

This action must be performed by a new class or function called SchemaBuilder or schema_builder.

The schema builder must account for dependencies created due to the include() function present in yamale. It should make sure that if a schema file contains a line that includes 'some_other_schema', then in the final output file some_other_schema must be present after the current schema file.

var: include('some_other_schema')
---
some_other_schema: ...

Ideally, there must not be any cyclic dependencies present in the schemas. We shall discover it along the way.

A mechanism to resolve dependencies could be (up for debate ;) ):
We can parse all the schema files and segregate them in lists.
If a schema file has 0 include() statements, then we add it to a list called statum0 (i.e it has 0 modalities).
If a schema file has 1 include() statement and the included schema has 0 include statements, then we add it to a list called stratum 1 (i.e. it has modality=1).
If a schema file has 1 include() statement and the included schema also has 1 include statement, then we add the parent schema file to a list called stratum2 (i.e. it has modality = 2).
If a schema file has 2 include() statements and the included schemas are stratum0's, then we add it to the list called stratum2 (modality = 2).
If a schema file has 2 include() statements and one of the included schemas is a stratum1 while the other is a stratum0, then we add it to stratum(2 + 1) i.e. stratum3 list.

If a file has n include statements and the included schema file has modalities x1, x2, x3,..., xn respectively, then the file must belong to the stratum(n + x1 + x2 + ... + xn) list.

We can then create an ordered sequence such that elements in stratum0 appear before elements in stratum1 and so on..

For the schemas in the same stratum (>0), if a schema has any include() statements that refer to another schema in the same stratum, the current schema should be present before the other schema.

Once we have the ordered list, the final output essentially should be a rendered yaml file in the same sequence.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
At present, we are writing the output of schema builder by hand as described in #4