LLM in Serverless Container

Performance comparison of TinyLlama-1.1B inference on GCP Cloud Run vs AWS Lambda.

Quick Start

# Build the Docker image (downloads model during build)
make build

# Run locally
make run-local
# → http://localhost:8080/health

# Deploy to cloud
make deploy-gcp
make deploy-aws

# Run load tests (requires k6: https://k6.io/docs/get-started/installation/)
make test-gcp GCP_URL=https://your-service.run.app
make test-aws AWS_URL=https://your-lambda-url.lambda-url.us-east-1.on.aws

# Generate analysis
make analyze

Project Structure

app/main.py          - FastAPI application (model loading + inference)
Dockerfile           - Container image (model baked in, works on both clouds)
k6/load_test.js      - k6 load testing script
analysis/analyze.py  - Parse results → plots + text analysis
scripts/             - Cloud deployment & cleanup scripts
docs/report.tex      - LaTeX report (Overleaf-ready)

Architecture

Client → [Cloud Load Balancer] → Container (FastAPI + TinyLlama) → Response
         GCP Cloud Run                                               
         AWS Lambda (via Web Adapter)                               

Both platforms run the exact same Docker image. The app auto-detects which platform it's on via environment variables (K_SERVICE for GCP, AWS_LAMBDA_FUNCTION_NAME for AWS).

API

Endpoint	Method	Description
/generate	POST	Generate text from prompt
/health	GET	Health check
/	GET	Service info

POST /generate

{"prompt": "What is cloud computing?", "max_tokens": 50}

Load Testing

Uses k6 with constant-arrival-rate executor for precise request rate control. Tests run at: 1, 5, 10, 25, 50 req/s × 30s each.

Requirements

• Docker
• k6
• Python 3.10+ with matplotlib, pandas (for analysis)
• gcloud CLI (for GCP deployment)
• AWS CLI (for AWS deployment)