exec-queue-tor

A code execution platform with a gRPC ingress, outbox-based eventing to Kafka, and a container-isolated executor. The system stores submitted code, publishes execution jobs to Kafka, consumes and executes them inside Docker for isolation, and exposes execution results via a simple API. Observability is powered by Prometheus and Grafana. The project provides Kubernetes manifests for local deployment and k6 scripts for load testing.

Repository layout

• code-saver/: gRPC service to receive code and persist it (producer/outbox writer).
• worker/: outbox worker publishing events to Kafka.
• code-executor/: Kafka consumer that executes code in Docker and exposes results.
• k8s/: Kubernetes manifests for local cluster deployment (services, deployments, Kafka).
• load-testing/: k6 script and proto for load testing end-to-end.
• docs/: Detailed documentation (architecture, deployment, observability, security, ops, etc.).

Quickstart

1. Prerequisites

• Docker and Docker Compose
• Kubernetes (local cluster such as kind, k3d, or Minikube)
• kubectl
• Helm (for alternative Kafka/Grafana/Prometheus installs)
• k6 (for load testing)

2. Run locally with Docker Compose (developer workflows)

Refer to service-level Dockerfile and any docker-compose.yml in submodules as needed during development. For full end-to-end workflows, prefer the Kubernetes setup below.

3. Deploy to a local Kubernetes cluster

See docs/deployment-k8s.md for step-by-step instructions, including Kafka setup and service NodePorts. After deploying, you should have:

• gRPC ingress for code-saver reachable via NodePort (see manifests)
• REST endpoint from code-executor for retrieving results
• Kafka broker reachable within the cluster
• Prometheus and Grafana running for metrics and dashboards

4. Load test the system

See docs/load-testing.md. The provided load-testing/load-testing.js script submits code via gRPC and polls the result API. Adjust target host/ports to match your cluster.

5. Observability

See docs/observability.md for Prometheus metrics, key queries, and Grafana dashboards.

Documentation

• Architecture: docs/architecture.md
• API & Proto workflow: docs/api.md
• Kubernetes deployment: docs/deployment-k8s.md
• Observability (Prometheus/Grafana): docs/observability.md
• Load testing (k6): docs/load-testing.md