ML gRPC Service with Canary Deployment

Production-ready ML inference service with gRPC interface and automated canary deployment strategy using Envoy proxy.

📋 Table of Contents

• Overview
• Architecture
• Canary Deployment Strategy
• Prerequisites
• Quick Start
• Deployment
• Testing
• Monitoring
• Rollback

🎯 Overview

This project implements a machine learning inference service with:

• gRPC API for high-performance predictions
• Multi-version support (v1.0.0 and v2.0.0)
• Canary deployment with gradual traffic shifting
• Envoy proxy for intelligent load balancing
• Health checks for automatic failover
• CI/CD pipeline with GitHub Actions

🏗 Architecture

┌─────────────┐
│   Client    │
└──────┬──────┘
       │
       ↓
┌─────────────────────────────────┐
│      Envoy Proxy (50051)        │
│  ┌───────────────────────────┐  │
│  │ Weighted Load Balancer    │  │
│  │  - v1: 90% → 50% → 0%    │  │
│  │  - v2: 10% → 50% → 100%  │  │
│  └───────────────────────────┘  │
└────────┬───────────────┬────────┘
         │               │
    ┌────↓────┐    ┌────↓────┐
    │ v1.0.0  │    │ v2.0.0  │
    │ Model   │    │ Model   │
    └─────────┘    └─────────┘

🚀 Canary Deployment Strategy

What is Canary Deployment?

Canary deployment is a progressive rollout strategy that reduces risk by gradually shifting traffic from an old version to a new version. Named after the "canary in a coal mine" practice, it allows you to detect issues early with minimal user impact.

Our Implementation

We use a three-stage canary rollout:

Stage 1: Initial Canary (10%)

• v1.0.0: 90% traffic (stable baseline)
• v2.0.0: 10% traffic (canary)
• Duration: Monitor for errors, latency, accuracy
• Decision point: If metrics are good → proceed to Stage 2

Stage 2: Expanded Canary (50%)

• v1.0.0: 50% traffic
• v2.0.0: 50% traffic
• Duration: Collect more data with balanced traffic
• Decision point: If metrics remain stable → proceed to Stage 3

Stage 3: Full Rollout (100%)

• v1.0.0: 0% traffic (deprecated)
• v2.0.0: 100% traffic (new baseline)
• Result: Complete migration to new version

Rollback Option

• v1.0.0: 100% traffic (back to stable)
• v2.0.0: 0% traffic (disabled)
• Trigger: Any critical issues detected

Why This Strategy?

✅ Risk Mitigation: Problems affect only a small percentage of users initially
✅ Data-Driven: Make decisions based on real production metrics
✅ Fast Rollback: Instant revert to previous version if needed
✅ Zero Downtime: Both versions run simultaneously
✅ A/B Testing: Compare model performance in production

📦 Prerequisites

• Docker and Docker Compose
• Python 3.11+
• Git

🚀 Quick Start

1. Initial Setup

# Clone the repository
git clone <your-repo-url>
cd ml_grpc_service

# Run setup script
chmod +x setup.sh
./setup.sh

This will:

• Install Python dependencies
• Generate gRPC proto files
• Create ML models (v1 and v2)
• Set up project structure

2. Build Docker Images

# Build both model versions
chmod +x build_versions.sh
./build_versions.sh

Verify images:

docker images | grep ml-grpc-service

Expected output:

ml-grpc-service   v2.0.0   ...
ml-grpc-service   v1.0.0   ...

3. Start Services

# Start Envoy proxy and both gRPC servers
docker compose -f docker-compose.envoy.yaml up -d

# Check services are running
docker ps

You should see three containers:

• ml-envoy (Envoy proxy)
• grpc-server-v1 (Model v1.0.0)
• grpc-server-v2 (Model v2.0.0)

4. Test the Service

# Run test client
python -m client.client

📊 Deployment

Manual Deployment

Execute canary stages manually:

# Stage 1: 10% traffic to v2
./canary_deploy.sh 10

# Wait and monitor metrics...
# If good, proceed to Stage 2

# Stage 2: 50% traffic to v2
./canary_deploy.sh 50

# Continue monitoring...
# If stable, proceed to Stage 3

# Stage 3: 100% traffic to v2
./canary_deploy.sh 100

Automated Deployment (CI/CD)

The GitHub Actions workflow automates the entire process:

# Trigger manually with specific stage
workflow_dispatch:
  inputs:
    stage: '10'  # or '50', '100', 'rollback'

Workflow Steps:

1. Checkout code
2. Build Docker images
3. Start services with docker-compose
4. Run canary deployment script
5. Execute traffic tests
6. Verify metrics

Secrets Required:

• MODEL_VERSION: Version tag for models
• CLOUD_TOKEN: Authentication token (if deploying to cloud)

GitHub Actions Usage

# Push to main/master triggers automated deployment
git push origin main

# Or manually trigger with GitHub UI:
# Actions → Deploy Canary ML Service → Run workflow
# Select stage: 10, 50, 100, or rollback

🧪 Testing

Traffic Distribution Test

# Test canary traffic routing
python test_canary.py

This script:

• Sends 100 prediction requests
• Counts responses from each version
• Verifies traffic distribution matches weights
• Reports success/failure

Example output:

Testing Canary Deployment...
Sending 100 requests...
v1.0.0: 89 requests (89%)
v2.0.0: 11 requests (11%)
✅ Traffic distribution matches expected 90/10 split

Manual Testing

# Single prediction test
python -m client.client

# Check which version handled the request (look for version in logs)
docker logs grpc-server-v1
docker logs grpc-server-v2

📈 Monitoring

Envoy Admin Interface

Access real-time metrics:

# Open in browser
http://localhost:9901

# Key endpoints:
# - /stats: All metrics
# - /clusters: Cluster health status
# - /config_dump: Current configuration

Check Cluster Status

# View cluster health
curl http://localhost:9901/clusters

# View traffic stats
curl http://localhost:9901/stats | grep cluster.service

Health Checks

Envoy automatically performs gRPC health checks:

• Interval: 10 seconds
• Timeout: 5 seconds
• Unhealthy threshold: 2 consecutive failures
• Healthy threshold: 2 consecutive successes

Unhealthy services are automatically removed from rotation.

Logs

# Envoy logs
docker logs ml-envoy

# gRPC server logs
docker logs grpc-server-v1
docker logs grpc-server-v2

# Follow logs in real-time
docker logs -f ml-envoy

🔄 Rollback

Quick Rollback

If issues are detected at any stage:

# Immediately return 100% traffic to v1.0.0
./canary_deploy.sh rollback

When to Rollback?

Trigger rollback if you observe:

• ❌ Error rate increase (>1% errors)
• ❌ Latency spikes (>500ms p95)
• ❌ Model accuracy degradation
• ❌ Health check failures
• ❌ Resource exhaustion (memory/CPU)

Automated Rollback

Add to CI/CD pipeline:

# In deploy.yml, add health check after deployment
- name: Health Check
  run: |
    if ! python health_check.py; then
      echo "Health check failed, rolling back..."
      ./canary_deploy.sh rollback
      exit 1
    fi

🛠 Configuration

Envoy Configuration

Edit envoy.yaml to modify:

• Traffic weights (default: 100/0)
• Health check intervals
• Timeouts
• Load balancing policy

weighted_clusters:
  clusters:
  - name: service_v1
    weight: 90  # Adjust this
  - name: service_v2
    weight: 10  # Adjust this

Model Versions

Update model versions in:

• docker-compose.envoy.yaml: Environment variables
• build_versions.sh: Docker build args
• Dockerfile: MODEL_FILE argument