Cast Iron

The platform-agnostic data storage and ETL system leveraging MinIO and other tools to enable mission endstates.

Basics

This project provides an example environment using MinIO to create a basic, cloud-agnostic ETL system.

Main Parts of the system include:

• Object Storage (AWS S3, MinIO, etc.)
• Message Queue (Kafka, Nats, etc.)
• Worker (Celery, Faust, etc.)
• ETL (Bash Scripts, Python)
• Database (MS SQL, MySQL, PostgreSQL, SQLite, etc.)

Getting Started

• Install Docker
• Run the docker compose

$ docker-compose up

Once started, the following areas are accessible:

• MinIO at localhost:9000
• PostgreSQL
  • User: castiron
  • Password: castiron
  • Host: localhost
  • Port: 5432
  • Database: castiron