devops-sandbox

A self-service platform for spinning up isolated, short-lived environments on a single Linux VM. Each environment gets its own Docker network, Nginx route, log stream, and health monitor. When the TTL expires — or you say so — everything is torn down and archived automatically.

---

What This Is

Think of it as a stripped-down internal Heroku with a chaos engineering toggle.

You create an environment. A container starts, Nginx opens a route to it, a log shipper attaches, and a health poller begins watching it every 30 seconds. You can crash it, pause it, or cut its network — then recover it. When the TTL runs out, the cleanup daemon destroys it without any input from you.

The demo app inside each environment is a simple Flask server. The platform is the project, not what runs inside it.

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         EC2 Linux VM                            │
│                                                                 │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │                    Control Plane                        │   │
│   │                                                         │   │
│   │   ┌──────────────┐          ┌──────────────────────┐   │   │
│   │   │  FastAPI      │          │   Cleanup Daemon     │   │   │
│   │   │  :8000        │          │   (60s loop)         │   │   │
│   │   │               │          │                      │   │   │
│   │   │  POST /envs   │          │  reads envs/*.json   │   │   │
│   │   │  GET  /envs   │          │  checks TTL expiry   │   │   │
│   │   │  DELETE /envs │          │  calls destroy_env   │   │   │
│   │   │  GET  /logs   │          └──────────────────────┘   │   │
│   │   │  GET  /health │                                      │   │
│   │   │  POST /outage │          ┌──────────────────────┐   │   │
│   │   └──────┬────────┘          │   Health Poller      │   │   │
│   │          │                   │   (30s loop)         │   │   │
│   │          │ wraps             │                      │   │   │
│   │          ▼                   │  GET /health per env │   │   │
│   │   ┌──────────────┐          │  3 failures → DEGRADED│  │   │
│   │   │  Bash Scripts │          │  writes health.log   │   │   │
│   │   │               │          └──────────────────────┘   │   │
│   │   │  create_env   │                                      │   │
│   │   │  destroy_env  │                                      │   │
│   │   │  simulate     │                                      │   │
│   │   └──────┬────────┘                                      │   │
│   └──────────┼────────────────────────────────────────────── ┘   │
│              │                                                    │
│   ┌──────────▼──────────────────────────────────────────────┐   │
│   │                     Data Plane                          │   │
│   │                                                         │   │
│   │   ┌─────────────────────────────────────────────────┐  │   │
│   │   │              Nginx Container  :80                │  │   │
│   │   │                                                  │  │   │
│   │   │   nginx/conf.d/env-abc123.conf  ← auto-written  │  │   │
│   │   │   nginx/conf.d/env-def456.conf  ← auto-written  │  │   │
│   │   │                                                  │  │   │
│   │   │   Routes by Host header: env-abc123.sandbox.local│  │   │
│   │   └──────┬───────────────────┬───────────────────────┘  │   │
│   │          │                   │                           │   │
│   │   ┌──────▼──────┐   ┌───────▼──────┐                   │   │
│   │   │sandbox-net- │   │sandbox-net-  │   (one Docker     │   │
│   │   │env-abc123   │   │env-def456    │    network        │   │
│   │   │             │   │              │    per env)       │   │
│   │   │ ┌─────────┐ │   │ ┌──────────┐│                   │   │
│   │   │ │env-abc123│ │   │ │env-def456││                   │   │
│   │   │ │:5000    │ │   │ │:5000     ││                   │   │
│   │   │ │Flask app│ │   │ │Flask app ││                   │   │
│   │   │ └─────────┘ │   │ └──────────┘│                   │   │
│   │   └─────────────┘   └─────────────┘                   │   │
│   │                                                         │   │
│   │   State:  envs/env-abc123.json   (id, ttl, port, status)   │
│   │   Logs:   logs/env-abc123/app.log                      │   │
│   │           logs/env-abc123/health.log                   │   │
│   └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Request flow:

Browser / curl
     │
     ▼
  Nginx :80  ──► routes by Host header ──► env container :5000
     │
     ▼ (also)
  FastAPI :8000  ──► wraps bash scripts ──► Docker + filesystem

---

Prerequisites

Requirement	Version
Ubuntu	22.04 LTS
Docker	24+
Python	3.10+
jq	any
curl	any
make	any

Install everything on a fresh Ubuntu 22.04 VM:

# Docker
sudo apt update
sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io
sudo usermod -aG docker $USER && newgrp docker

# Everything else
sudo apt install -y python3 python3-pip jq curl make git
pip3 install fastapi uvicorn --break-system-packages

---

Quick Start

From zero to a running environment in 5 commands:

git clone https://github.com/YOUR_USERNAME/devops-sandbox.git
cd devops-sandbox
docker build -t sandbox-demo-app:latest ./demo-app
make up
make create

That's it. The output will give you a URL and TTL. Open the URL in your browser.

---

Repo Structure

devops-sandbox/
├── platform/
│   ├── create_env.sh        # spins up an environment
│   ├── destroy_env.sh       # tears one down
│   ├── cleanup_daemon.sh    # TTL watcher, runs every 60s
│   ├── simulate_outage.sh   # crash / pause / network / recover
│   └── api.py               # FastAPI control plane
├── nginx/
│   ├── nginx.conf           # main config, includes conf.d/
│   └── conf.d/              # per-env configs, auto-written on create
├── monitor/
│   └── poller.sh            # health checker, runs every 30s
├── demo-app/
│   ├── app.py               # Flask app with GET / and GET /health
│   ├── requirements.txt
│   └── Dockerfile
├── logs/                    # gitignored — runtime logs per env
├── envs/                    # gitignored — JSON state per env
├── Makefile
└── README.md

---

Makefile Targets

make up                        # start Nginx + cleanup daemon + health poller + API
make down                      # stop everything, destroy all envs
make create                    # interactive: prompts for name and TTL
make destroy ENV=env-abc123    # destroy a specific environment
make logs    ENV=env-abc123    # tail that environment's app.log
make health                    # print health status of all active envs
make simulate ENV=env-abc123 MODE=crash   # run outage simulation
make clean                     # wipe all logs, state, and Nginx configs

---

API Reference

The API runs on port 8000. Interactive docs at http://YOUR_IP:8000/docs.

Method	Endpoint	What it does
POST	/envs	Create a new environment
GET	/envs	List all active environments + TTL remaining
DELETE	/envs/:id	Destroy an environment
GET	/envs/:id/logs	Last 100 lines of app.log
GET	/envs/:id/health	Last 10 health check results
POST	/envs/:id/outage	Trigger an outage simulation

Create an environment:

curl -X POST http://localhost:8000/envs \
  -H "Content-Type: application/json" \
  -d '{"name": "my-app", "ttl": 600}'

Trigger an outage:

curl -X POST http://localhost:8000/envs/env-abc123/outage \
  -H "Content-Type: application/json" \
  -d '{"mode": "pause"}'

Valid outage modes: crash pause network recover stress

---

Full Demo Walkthrough

1 — Start the platform

make up

This starts Nginx (Docker), the cleanup daemon (background), the health poller (background), and the FastAPI server on port 8000.

---

2 — Create an environment

make create
# Enter name: demo-app
# Enter TTL:  300

Output:

✅ Environment ready!
   ID:      env-f9c029f5
   URL:     http://<YOUR_IP>:8112
   TTL:     300s (expires at 2026-05-10 05:00:00)
   Health:  http://<YOUR_IP>:8112/health

---

3 — Hit the app

curl http://localhost:8112/
curl http://localhost:8112/health

---

4 — Check health status

make health

After 30 seconds the poller will have written its first entry to logs/env-f9c029f5/health.log.

---

5 — Simulate an outage

# Pause the container (app becomes unreachable)
make simulate ENV=env-f9c029f5 MODE=pause

# Watch the health poller detect it (within 90s)
make health

After 3 consecutive failed health checks, the environment status flips to degraded.

---

6 — Recover

make simulate ENV=env-f9c029f5 MODE=recover

The container unpauses, the status resets to running, and health checks resume passing.

---

7 — View logs

make logs ENV=env-f9c029f5

Or via the API:

curl http://localhost:8000/envs/env-f9c029f5/logs

---

8 — Destroy manually (or wait for TTL)

make destroy ENV=env-f9c029f5

This stops the container, removes the Docker network, deletes the Nginx config, reloads Nginx, archives the logs to logs/archived/, and removes the state file.

If you don't destroy it manually, the cleanup daemon will do it when the TTL expires.

---

How Each Piece Works

Environment Lifecycle

create_env.sh does the following in order:

1. Generates a unique ID (env- + 8 hex chars from /proc/sys/kernel/random/uuid)
2. Picks a random port in the range 8080–8179
3. Creates a dedicated Docker network (sandbox-net-$ENV_ID)
4. Connects Nginx to that network so it can proxy to the container
5. Starts the app container with sandbox.env=$ENV_ID label
6. Writes state to envs/$ENV_ID.json atomically (write to .tmp, then mv)
7. Writes an Nginx upstream + server block to nginx/conf.d/$ENV_ID.conf
8. Reloads Nginx with nginx -s reload
9. Starts docker logs -f piped to logs/$ENV_ID/app.log in the background, saves PID

destroy_env.sh reverses all of that:

1. Kills the log shipper process by PID
2. Stops and removes all containers with label=sandbox.env=$ENV_ID
3. Disconnects and removes the Docker network
4. Archives logs/$ENV_ID/ to logs/archived/$ENV_ID/
5. Deletes the Nginx config and reloads Nginx
6. Deletes the state file

Cleanup Daemon

Runs as a background process. Every 60 seconds it reads every file in envs/, compares created_at + ttl against the current epoch time, and calls destroy_env.sh on anything that has expired. All actions are timestamped and written to logs/cleanup.log.

Health Poller

Every 30 seconds it reads all state files, hits GET /health on each environment's port, and records the HTTP status and latency to logs/$ENV_ID/health.log. After 3 consecutive non-200 responses it sets the environment's status to degraded in the state file and prints a warning to stdout.

Nginx Routing

Nginx runs as a Docker container. Its main config includes conf.d/*.conf. Every create writes a new file there and every destroy removes it. After each change, the script runs docker exec sandbox-nginx nginx -s reload. The environment is reachable by direct port (e.g. :8112) or by setting Host: env-f9c029f5.sandbox.local in the request header.

Outage Simulation

simulate_outage.sh has a guard at the top that refuses to run against sandbox-nginx, sandbox-api, or sandbox-cleanup. The modes work as follows:

Mode	What happens	How to recover
crash	docker kill — container stops immediately	docker start via recover
pause	docker pause — process frozen, port still open	docker unpause via recover
network	docker network disconnect — container isolated	docker network connect via recover
recover	undoes whichever mode was active	—
stress	runs yes > /dev/null inside the container to spike CPU	kill the process manually

---

Known Limitations

• Port range is 8080–8179, so a maximum of ~100 concurrent environments per VM
• Log shipping uses Approach A (simple docker logs -f pipe) — no log aggregator
• Nginx reload causes ~1 second of disruption per config change
• The stress mode uses a basic shell trick, not stress-ng, so CPU impact varies
• No HTTPS — this is an HTTP-only sandbox platform
• Public IP detection uses ifconfig.me which may be slow or unavailable in some networks; the EC2 metadata endpoint (169.254.169.254) is used as a fallback
• State files are stored on disk with no locking — concurrent creates on the same port could theoretically collide (rare given the random port selection)

---

Security Notes

This platform is designed for internal sandbox use on a trusted network. Do not expose port 8000 (API) or 8080–8179 (env ports) to the public internet without adding authentication. The API has no auth layer by design — add an API key middleware or put it behind a VPN for any real deployment.