SinaraML Step metrics tutorial

SinaraML Step metrics provides way to track experiments flow. Every step creates empty metrics which contains runid only. Metrics are saved to the storage in job run mode only. Metrics are not saved to the storage when running substep notebooks in the interactive mode.

Metrics in fact represent dictionary with no predefined structure. The only predefined field exsist - it's run_id.

Every value can be addeed with add_metric(, ) method. Example

substep.add_metric('experiment_name', 'evaluation_metrics_example')

Also any json structure as dictionary or array can be a value:

substep.add_metric("version", 
                   {'bento_service_version': bento_service_version})

Metrics are suitable to compare different runs as experiments because metrics of all step runs can be loaded and analyzed as single dataset.

# all_metrics_path should represent wildcard path to stored metrics files for example:
# /data/home/jovyan/example_median_house_price/test/model_eval/*/reports_model_eval_ipynb/metrics.json
metrics = spark.read.json(all_metrics_path)

# display metrics schema
metrics.printSchema()

# select fields and values and display table data
df = metrics.select('experiment_name', 'eval_result.r2', 'eval_result.rmse', 'run_id', 'version.bento_service_version')
df.show()

You can see example of using metrics in pipeline step in https://github.com/4-DS/house_price-model_eval repository. There are two notebooks:

1. model_eval.ipynb - eval substeb
2. visualize.ipynb - visualize series of collected metrics

Step should be executed to collect metrics. For instance:

python step.dev.py

After running step open and run visualize.ipynb notebook to visually analyze metrics.