Utilize Elyra.ai to orchestrate model building

[dcmartin]

Background

Existing capabilities from Home Assistant Jupyter notebook addon are insufficient for general-purpose AI model building; the Elyra.ai project is attempting to provide a more complete solution.

Proposal

Utilization of Elyra.ai as a mechanism to execute a series of Jupyter notebooks that will process ground truth, and potentially other aspects of the model building, and generate a result which may be inspected for quality, e.g. confusion matrix.

Expected workflow would be based upon availability of content organized into a hierarchical structure, e.g. a standard Web directory listing (n.b. httpd.apache.org/docs/2.0/mod/mod_dir.html). For example see nVidia DIGITS (https://docs.nvidia.com/deeplearning/digits/digits-user-guide/index.html)

1. Create asset with human-name with collected metadata in a defined repository (e.g. Github)
2. Define extrinsic asset components; content, code, etc.. from one or more repositories (e.g. Github, directory, S3, ..)
3. Execute asset; either batch or dynamic; provide container composition of Elyra functionality
4. Monitor asset execution until completion and provide state
5. Update asset post-execution (e.g. pull-request to Github repository)

Results from execution will be utilized by other services to enable CI/CD and other processes.

Examples

Confusion matrix

nVidia DIGITS model build

Watson classifier heatmap