Adding initial code for the MLOps with Databricks

[riserrad]

Purpose

• It adds the working code that was produced so far by myself, @dtzar, @odaibert, @sudivate

Does this introduce a breaking change?

[ ] Yes
[X] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

It adds the initial code to the repo

How to Test

• Please follow instructions on README to setup the infrastructure. More instructions on how to setup the train pipeline to come.

Initial usage:

• Create your databricks cluster and take note of
  • The Cluster ID
  • A Personal Access Token you can create following this guidance

Then:

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

• Test the code

# First, manage to have your cluster up and running for the process through cluster_manager.py
python3 ./cluster_config/cluster_manager.py --help

# Then, invoke the Train pipeline:
python3 ./aml_service/pipelines/train_pipeline.py --help

What to Check

Make sure that the AML pipeline is triggered and successfully executed:

• Experiment steps ran
• Model registered on AML Service