refactor: Rename, reorganize schema module

[ksbrar]

Redo of #1936 (which was reverted by 6d8474a) . This ends up being a significant reorganization:

Changes

• (1) Moves ludwig.utils.schema into ludwig.marshmallow, and (2) Renames ludwig.marshmallow module to ludwig.schema and ludwig/marshmallow/marshmallow_schema_utils.py to ludwig/schema/utils.py. Also, for convenience, all of the functionality inside of the current schema.py (namely, get_schema() which returns the full Ludwig schema and validate_config()) lives inside ludwig/schema/__init__.py to make imports elsewhere more readable.
• Moves TrainerConfig and get_trainer_jsonschema() into ludwig/schema/trainer.py.
• Moves all optimizer configs out of ludwig/modules/optimization_modules.py and into ludwig/schema/optimizers.py'
• Moves sequence_encoder_registry out of ludwig/combiners/combiners.py, converts it into a full-fledged Registry(), and places it in ludwig/encoders/registry.py.
• Moves all combiner configs out of ludwig/combiners/combiners.py and into a new ludwig/schema/combiners/ directory (module) with a file per individual combiner. Again, to make things more readable elsewhere, __init__.py is set so that a user can import directly from this submodule (e.g. a user can write from ludwig.schema.combiners import ConcatCombinerConfig).

Misc.

• Small change to train_with_backend integration test.
• Some docstring/comment changes.

Overview:

• The structure of new schema module is the following:

ludwig.schema               <-- Meant to contain all schemas, utilities, helpers related to describing and validating Ludwig configs.
├── __init__.py             <-- Contains fully assembled Ludwig schema (`get_schema()`), `validate_config()` for YAML validation, and all "top-level" schema functions.
├── utils.py                <-- An extensive set of marshmallow-related fields, methods, and schemas that are used elsewhere in Ludwig.
├── trainer.py              <-- Contains `TrainerConfig()` and `get_trainer_jsonschema`
├── optimizers.py           <--- Contains every optimizer config (e.g. `SGDOptimizerConfig`, `AdamOptimizerConfig`, etc.) and related marshmallow fields/methods.
└── combiners/
    ├── __init__.py         <-- Imports for each combiner config file (making imports elsewhere more convenient).
    ├── utils.py            <-- New location of `combiner_registry`, `get_combiner_jsonschema()`, `get_combiner_conds()`
    ├── base.py             <-- New location of `BaseCombinerConfig`
    |
    ├──  comparator.py      <-- New location of `ComparatorCombinerConfig`
    ... <file for each combiner> ...
    └──  transformer.py     <-- New location of `TransformerCombinerConfig`

Do not merge until after #1961 goes through! Ready for review.

[github-actions]

Unit Test Results

       6 files  ±0         6 suites  ±0   1h 34m 7s ⏱️ + 3m 29s
2 774 tests ±0  2 741 ✔️ ±0    33 💤 ±0  0 ❌ ±0 
8 322 runs  ±0  8 219 ✔️ ±0  103 💤 ±0  0 ❌ ±0 

Results for commit de7769e. ± Comparison against base commit a01336d.

♻️ This comment has been updated with latest results.

[justinxzhao]

Could you add your directory structure description as header """ comments in the relevant files (perhaps replacing the copyright header, which we shouldn't add for new files)?

ludwig.schema               <-- Meant to contain all schemas, utilities, helpers related to describing and validating Ludwig configs.
├── utils.py                <-- An extensive set of marshmallow-related fields, methods, and schemas that are used elsewhere in Ludwig.
├── schema.py               <-- Contains the fully assembled Ludwig schema and validate() function that is used for user-input YAML validation. Users should generally only need to look at this.
└── __init__.py

[ksbrar]

Could you add your directory structure description as header """ comments in the relevant files (perhaps replacing the copyright header, which we shouldn't add for new files)?

ludwig.schema               <-- Meant to contain all schemas, utilities, helpers related to describing and validating Ludwig configs.
├── utils.py                <-- An extensive set of marshmallow-related fields, methods, and schemas that are used elsewhere in Ludwig.
├── schema.py               <-- Contains the fully assembled Ludwig schema and validate() function that is used for user-input YAML validation. Users should generally only need to look at this.
└── __init__.py

Added

[tgaddair]

ludwig/schema/schema.py is weird. can you move this function into ludwig/schema/__init__.py instead?

[ksbrar]

moved

[ksbrar]

this actually causes a circular import error

I don't know enough about how python does submodule initialization but it seems as though to import schema.utils (which is used in e.g. the combiner and trainer files) it will also pull in that init file (which itself pulls in schemas from those trainer/combiner files)

something I talked about with @justinxzhao in the past was moving the trainer, combiner, optimizer schemas into this new ludwig.schema module as well. I was going to do that in a follow up PR, but should I do this now?

[ksbrar]

Going to move the other schemas over into this module, per conversation with @tgaddair

[tgaddair]

Looks great, just two things to be fixed.