Skip to content

xenoISA/isA_Cloud

Repository files navigation

isA Cloud - Cloud-Native Infrastructure Platform

Kubernetes Infrastructure + Service Mesh + API Gateway

Python Kubernetes Apache APISIX Consul ArgoCD


Overview

isA Cloud is the isA platform's cloud-native infrastructure center, providing:

This Repository Provides

isa_common Python Library (native async clients)

  • Direct async connections to 8 infrastructure backends
  • No intermediate gRPC layer β€” clients connect to native ports
  • Source in isA_common/isa_common/

Infrastructure Deployment (Kubernetes)

  • PostgreSQL, Redis, Neo4j, MinIO, NATS, Mosquitto, Loki, Grafana, Qdrant
  • Consul (service discovery), APISIX (API gateway)

GitOps Configuration

  • Kubernetes deployment configs (Kustomize)
  • ArgoCD application definitions
  • Multi-environment management (dev/staging/production)

CI/CD Pipeline

  • GitHub Actions automated builds
  • Image push to ECR
  • Automated deployment config updates

External Business Services (other repositories)

These services have source code in their own repos; isA_Cloud contains only their Kubernetes deployment configs:

  • isA_user β€” 27 user microservices (account, auth, session, organization...)
  • isA_Agent β€” AI agent service
  • isA_MCP β€” Model Control Protocol service
  • isA_Model β€” AI model service
  • isa-data β€” Data service
  • web-service β€” Web service

Architecture

Overall Architecture

                         External Traffic
                              β”‚
                              β–Ό
                      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                      β”‚ Apache APISIX β”‚  API Gateway (Port: 9080)
                      β”‚   (Gateway)   β”‚  - Dynamic routing (auto-sync Consul)
                      β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜  - Auth/rate-limit/CORS
                              β”‚
                              β–Ό
                      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                      β”‚    Consul     β”‚  Service Discovery
                      β”‚ (42 services) β”‚  - Health checks
                      β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜  - KV storage
                              β”‚
       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”‚                      β”‚                      β”‚
       β–Ό                      β–Ό                      β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Infrastructureβ”‚     β”‚  isa_common   β”‚      β”‚ Business     β”‚
β”‚ (this repo   β”‚     β”‚  Python SDK   β”‚      β”‚ Services     β”‚
β”‚  deploys)    β”‚     β”‚ (this repo)   β”‚      β”‚ (other repos)β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€     β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€      β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ PostgreSQL   │◄────│ AsyncPostgres β”‚      β”‚ isA_user     β”‚
β”‚ Redis        │◄────│ AsyncRedis    β”‚      β”‚  β”œβ”€ auth     β”‚
β”‚ Neo4j        │◄────│ AsyncNeo4j    β”‚      β”‚  β”œβ”€ account  β”‚
β”‚ MinIO        │◄────│ AsyncMinIO    β”‚      β”‚  └─ ... (27) β”‚
β”‚ NATS         │◄────│ AsyncNATS     β”‚      β”‚              β”‚
β”‚ Mosquitto    │◄────│ AsyncMQTT     β”‚      β”‚ isA_Agent    β”‚
β”‚ Loki         β”‚     β”‚ AsyncDuckDB   β”‚      β”‚ isA_MCP      β”‚
β”‚ Grafana      β”‚     β”‚ AsyncQdrant   β”‚      β”‚ isA_Model    β”‚
β”‚ Qdrant       β”‚     β”‚ ConsulRegistryβ”‚      β”‚ isa-data     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜      β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
      Native Ports         Direct
   (5432,6379,7687,        Connection
    4222,9000,1883,        (no gRPC
    6333,3100)              layer)

GitOps Workflow

1. Developer commits code
   β”œβ”€ isA_Cloud: modify isa_common library or deployment configs
   └─ External repo: modify business service code
                     β”‚
                     β–Ό
2. CI Pipeline (GitHub Actions)
   β”œβ”€ Lint & Test (pytest for Python, security scans)
   β”œβ”€ Build Docker images
   β”œβ”€ Push to ECR
   └─ External repos trigger repository_dispatch β†’ isA_Cloud
                     β”‚
                     β–Ό
3. Update GitOps config (automatic)
   β”œβ”€ CD workflow updates deployment.yaml with new image tag
   └─ Git commit & push
                     β”‚
                     β–Ό
4. ArgoCD auto-sync (within 30 seconds)
   β”œβ”€ Detects Git changes
   └─ Applies to Kubernetes
                     β”‚
                     β–Ό
5. Kubernetes rolling update
   β”œβ”€ Creates new Pods β†’ health check β†’ Consul registration
   └─ APISIX route sync (CronJob every 5 minutes)

Repository Structure

isA_Cloud/
β”œβ”€β”€ isA_common/                   # Python infrastructure library
β”‚   β”œβ”€β”€ isa_common/
β”‚   β”‚   β”œβ”€β”€ __init__.py           # Exports (v0.3.1)
β”‚   β”‚   β”œβ”€β”€ async_base_client.py  # Abstract base for all clients
β”‚   β”‚   β”œβ”€β”€ async_client_config.py
β”‚   β”‚   β”œβ”€β”€ async_redis_client.py
β”‚   β”‚   β”œβ”€β”€ async_postgres_client.py
β”‚   β”‚   β”œβ”€β”€ async_nats_client.py
β”‚   β”‚   β”œβ”€β”€ async_neo4j_client.py
β”‚   β”‚   β”œβ”€β”€ async_minio_client.py
β”‚   β”‚   β”œβ”€β”€ async_qdrant_client.py
β”‚   β”‚   β”œβ”€β”€ async_duckdb_client.py
β”‚   β”‚   β”œβ”€β”€ async_mqtt_client.py
β”‚   β”‚   β”œβ”€β”€ consul_client.py      # Service discovery
β”‚   β”‚   └── events/               # Event-driven billing architecture
β”‚   β”œβ”€β”€ tests/                    # pytest test suite
β”‚   └── pyproject.toml
β”‚
β”œβ”€β”€ deployments/
β”‚   β”œβ”€β”€ kubernetes/               # Kustomize configs
β”‚   β”‚   β”œβ”€β”€ local/                # KIND cluster
β”‚   β”‚   β”œβ”€β”€ staging/              # Staging K8s
β”‚   β”‚   └── production/           # Production K8s (HA)
β”‚   β”œβ”€β”€ argocd/                   # ArgoCD app-of-apps
β”‚   β”œβ”€β”€ terraform/                # AWS IaC (staging)
β”‚   └── charts/isa-service/       # Generic Helm chart
β”‚
β”œβ”€β”€ .github/workflows/            # CI/CD pipelines
β”œβ”€β”€ tests/                        # Integration test scripts
β”‚   β”œβ”€β”€ contracts/                # Logic contracts (8 services)
β”‚   β”œβ”€β”€ test_auth_via_apisix.sh
β”‚   └── ...
β”‚
└── docs/                         # Documentation

Core Components

1. isa_common β€” Python Infrastructure SDK

Native async clients connecting directly to backend services on their native ports:

Client Backend Port Methods Status
AsyncRedisClient Redis 6379 53 Complete
AsyncPostgresClient PostgreSQL 5432 19 Complete
AsyncNATSClient NATS 4222 33 Complete
AsyncNeo4jClient Neo4j 7687 37 Partial
AsyncMinIOClient MinIO 9000 35 Complete
AsyncQdrantClient Qdrant 6333 25 Complete
AsyncDuckDBClient DuckDB embedded 27 Complete
AsyncMQTTClient Mosquitto 1883 29 Complete

Additional local-mode clients: AsyncSQLiteClient, AsyncLocalStorageClient, AsyncChromaClient, AsyncMemoryClient

Usage:

from isa_common import AsyncRedisClient, AsyncPostgresClient, AsyncNATSClient

# Direct connection to Redis on native port
async with AsyncRedisClient(host="localhost", port=6379) as redis:
    await redis.set("session:user_123", session_data, ttl=3600)
    session = await redis.get("session:user_123")

# Direct connection to PostgreSQL
async with AsyncPostgresClient(host="localhost", port=5432, database="mydb") as pg:
    rows = await pg.query("SELECT * FROM users WHERE org_id = $1", "org_123")

# Direct connection to NATS with JetStream
async with AsyncNATSClient(host="localhost", port=4222) as nats:
    await nats.publish("orders.created", {"order_id": "123"})
    messages = await nats.pull_messages("USAGE_EVENTS", "billing-consumer")

2. Apache APISIX (API Gateway)

  • Unified traffic entry (Port: 9080)
  • Dynamic routing (auto-sync from Consul)
  • Auth (JWT/Key Auth), rate limiting, CORS
  • Admin API: http://localhost:9180

3. Consul (Service Discovery)

  • Service registration/discovery, health checks, KV config
  • 42 registered services (33 business + 9 infrastructure)
  • UI: http://localhost:8500

4. ArgoCD (GitOps)

  • Git β†’ Kubernetes auto-sync, declarative deployments
  • Multi-environment: dev (auto-sync), staging (auto-sync), production (manual sync)

Quick Start

Prerequisites

Tool Version Install
Docker 20.10+ Docker Desktop
kubectl 1.28+ brew install kubectl
kind 0.20+ brew install kind
Python 3.12+ brew install python

Local Infrastructure (Docker Compose)

docker compose up -d             # Start all 11 infrastructure services
docker compose ps                # Check status
make health                      # Quick health check

Install isa_common

make install                     # Or: cd isA_common && pip install -e ".[dev]"

Run Tests

make test                                    # All tests
make test-unit                               # Unit tests only (no infra needed)
make test-service s=redis                    # Single service tests
make test-smoke                              # Billing pipeline smoke tests

Local Kubernetes Deployment (KIND)

cd deployments/kubernetes/scripts
./kind-setup.sh          # Create KIND cluster
./kind-deploy.sh         # Deploy all services
./check-services.sh      # Check status

# Access services
open http://localhost:9080    # APISIX Gateway
open http://localhost:8500    # Consul UI
open http://localhost:3000    # Grafana

Infrastructure Services

Service Port Purpose
APISIX 9080, 9180 API Gateway
Consul 8500, 8600 Service Discovery
PostgreSQL 5432 Relational Database
Redis 6379 Cache/Sessions
Neo4j 7474, 7687 Graph Database
MinIO 9000, 9001 Object Storage
NATS 4222, 8222 Message Queue
Mosquitto 1883 MQTT Broker
Loki 3100 Log Aggregation
Grafana 3000 Visualization
Qdrant 6333 Vector Database

Deployment

Environments

Environment Cluster Namespace Branch
dev KIND local isa-cloud-dev develop
staging KIND/EKS isa-cloud-staging main
production EKS/GKE isa-cloud-production production

See Production Deployment Guide for EKS/GKE setup.


Documentation

Core Docs

Runbooks

Test Scripts


Contributing

Commit Conventions

feat: New feature
fix: Bug fix
docs: Documentation
refactor: Refactoring
test: Tests
chore: Build/tooling

License

MIT License


Made with care by the isA Team

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Packages

 
 
 

Contributors