KAML-D can be deployed on any cloud platform as well as on-premises, given you can run Kubernetes. Existing open source components KAML-D uses:
- Kubernetes for workload management and to ensure portability
- TensorFlow for machine learning execution
- JupyterLab/JupyterHub for data scientists
- Storage layer: To hold the datasets, Minio, Ceph, as well as cloud-provider specific offerings such as EBS, with built-in dotmesh support for snapshots
New open source components KAML-D introduces:
- KAML-D Workbench: a graphical UI for data scientists, data engineers, developers, and SREs to manage datasets as well as to test and deploy ML algorithms. Builds on the metadata layer to find and visualize datasets. Builds on the storage layer to store and load datasets.
- KAML-D Metadata Hub: a data and metadata layer using PrestoDB and Elasticsearch for indexing and querying datasets.
- KAML-D Observation Hub: a comprehensive observability suite for Site Reliability Engineers (SRE) and cluster admins as well as developers on the app level to understand the health of the KAML-D platform and troubleshoot issues on the platform and application level:
- Prometheus and Grafana for end-to-end metrics and monitoring/alerting
- EFK stack for (aggregrated) logging
- Jaeger for (distributed) tracing
Note that user management and access control is outside of the scope of KAML-D. We will, however provide for an RBAC-enabled setup and offer support for standard Kubernetes integration points such authentication via OpenID Connect with LDAP or GitHub.
If you want to learn more about the design decisions and KAML-D's origin, check out this ca. 14 min long YouTube video:
The UX is central to KAML-D. Users can have different roles, for example data scientists, data engineers, developers, SREs, or admins and for each role the UX should be pleasing and rich. UX over performance, strive for simplicity and cleanliness.
For now, the plan is to implement the KAML-D UI as a JupyterLab plug-in.