This repository contains the components and examples used to run Extract-Transform-Load (ETL) operations on an AIStore cluster.
For more information on how ETL works in AIStore, refer to the following documentation:
- AIS ETL Docs
- AIS ETL CLI Docs
- AIS Python SDK Docs (includes ETL usage)
- transformers — ready-to-use ETL transformers that can be deployed on an AIStore cluster
- runtime — ETL runtime definitions for
init_classfunction - deploy — utility tools and deployment configurations
- examples — example scripts for using and testing ETL features
- docs — documentation-related content
To begin using ETLs in AIStore, you must first deploy an AIStore cluster on Kubernetes. Two reference deployments are provided, each serving a different purpose:
- Folder:
deploy/dev/k8s/kustomize - Intended for: Local AIStore development and functional testing of ETL transformers.
- How to use: Start a local Kubernetes cluster (for example, with kind or minikube) and follow the step-by-step instructions in the referenced folder to deploy a minimal AIS cluster configured for ETL development.
- Documentation: README
- Folder:
deploy/prod/k8s - Intended for: Production-grade deployments of AIStore at scale.
- How to use: Use the provided Dockerfiles to build AIS images and follow the tooling in the companion repository to install, upgrade, and monitor the cluster.
- Documentation: AIS/K8s Operator and Deployment Playbooks
After deploying the AIS cluster, you can confirm that it is reachable by running:
$ ais etl show
A blank list ([]) and the absence of error messages indicate that the cluster is healthy and ready for you to register ETL transformers.
Note For the ETL functionality you must first initialize the transformation logic so that the cluster can execute it. The following folders and examples in this repository demonstrate how to build and register your own transformers.