Custom High-Concurrency Layer 7 Load Balancer

A Layer 7 reverse proxy and load balancer written in Go to explore concurrent systems programming and networking internals. The project supports host-based routing, Round Robin and Least Connections load balancing, automatic health checks, connection reuse, adaptive request admission, and graceful backend failover. Particular emphasis was placed on minimizing lock contention, benchmarking throughput, and validating behavior under backend failure scenarios.

---

🚀 Key Features

• Host-Based Routing — Routes incoming traffic to isolated backend clusters based on the request's Host header (e.g., api.localhost:8080 vs www.localhost:8080), allowing multiple services or environments to share a single proxy without path rewriting.

• Buffered Channel Admission Control — Uses a buffered chan struct{} as a lightweight semaphore to cap concurrent in-flight requests. Since Go's HTTP server already creates goroutines per request, this limits concurrency without introducing a worker pool.

• Timed Admission Queue — Rather than rejecting requests immediately when capacity is exhausted, requests wait up to 25ms for an available slot before returning HTTP 503 if the queue remains full.

• Adaptive Request Deadlines — Dynamically adjusts request read deadlines based on Content-Length, allowing larger uploads additional time while enforcing a 5-second limit for unknown payload sizes to reduce exposure to slowloris-style attacks.

• Context Propagation — Uses http.NewRequestWithContext so backend requests are cancelled immediately when the client disconnects, preventing unnecessary backend work.

• Atomic Backend Snapshots — Routing state is published through atomic.Value, allowing lock-free reads while health checks atomically replace failed backend snapshots without interrupting request handling.

• Connection Pool Management — Configures http.Transport idle connection limits based on backend count, ensuring each backend maintains reusable persistent connections and reducing repeated TCP connection setup.

• Streaming Response Forwarding — Streams backend responses directly to clients while preserving status codes and headers with minimal buffering and allocation overhead.

---

📊 Benchmarks

Benchmarks were performed on an AMD Ryzen 7 9800X3D using a custom Go flood tool issuing 50,000 HTTP requests through a shared keep-alive client with 500 concurrent goroutines. All tests were executed on localhost to measure proxy throughput under controlled conditions.

Round Robin — All backends healthy

Metric	Result
Throughput	~79,000 req/sec
Total requests	50,000
Concurrency	500 goroutines
Successful	49,855
Failed/Timeout	145 (queue capacity reached)
Distribution	16,618 / 16,618 / 16,619 across 3 backends

A one-request difference across 50,000 requests confirms the atomic Round Robin counter distributed traffic evenly. The 145 failed requests are expected because max_queue is intentionally limited to 100, causing requests beyond that admission limit to return HTTP 503.

---

Round Robin — One backend down

With one backend terminated during the benchmark, the load balancer automatically removed it from the routing pool and redistributed traffic across the remaining healthy servers without manual intervention.

Metric	Result
Throughput	~70,800 req/sec
Distribution	24,975 / 24,975 across 2 backends

---

Least Connections — All backends healthy

Metric	Result
Throughput	~73,100 req/sec
Distribution	~16,500 / ~16,981 / ~16,408 across 3 backends

Minor variance is expected because Least Connections routes each request to the backend with the fewest active requests at that moment. As requests complete at different times, the final request totals naturally diverge slightly.

---

Least Connections — One backend down

The remaining healthy backends continue serving requests after the failed node is removed by the health checker.

---

Health Check Detecting a Failed Backend

Health checks continuously monitor backend availability. When a server becomes unreachable, it is removed from the routing pool and later restored automatically once healthy again.

---

Graceful Backend Shutdown via Control Tool

The included control utility allows individual backends to be removed gracefully for testing or maintenance without restarting the load balancer.

---

🏗️ Design Decisions

Why atomic.Value?

Routing state is read on every request but updated infrequently by the health checker. Publishing immutable backend snapshots through atomic.Value allows request handlers to read routing information without lock contention while updates atomically replace the current snapshot.

Why buffered channels instead of a worker pool?

The Go HTTP server already creates a goroutine for every incoming request. The buffered channel serves only as an admission semaphore that limits concurrent work, avoiding the additional scheduling complexity of a dedicated worker pool.

Why reuse backend connections?

Opening a new TCP connection for every proxied request introduces unnecessary handshake overhead. Reusing persistent connections through http.Transport reduces latency and CPU utilization while improving throughput.

---

💻 Getting Started

Prerequisites

Make sure Go is installed. Developed and tested on:

$ go version
go version go1.26.0 darwin/arm64

Run the Load Balancer

go run .

Run the Test Servers

go run testing/servers/servers.go

Run the Flood Test

go run testing/flood/flood.go

---

⚙️ Configuration — config.json

All behavior is configured through a single JSON file.

{
    "host": "localhost",
    "port": "8080",
    "backends": {
        "api.localhost:8080": ["localhost:9000", "localhost:9001", "localhost:9002"],
        "www.localhost:8080": ["localhost:9050", "localhost:9051", "localhost:9052"]
    },
    "tls": {
        "enabled": false,
        "certfile": "./cert.pem",
        "keyfile": "./key.pem"
    },
    "timeouts": {
        "readheader_timeout": 7,
        "write_timeout": 7,
        "client_timeout": 7
    },
    "max_queue": 100,
    "max_idle_conns": 100,
    "mode": 0
}

Field Breakdown

Field	Description
backends	Maps Host headers to backend server pools.
max_queue	Maximum number of concurrent in-flight requests admitted through the semaphore.
max_idle_conns	Idle connection budget per backend.
tls.enabled	Enables TLS termination.
timeouts.*	HTTP timeout configuration.
mode	Load balancing algorithm selector.

Load Balancing Modes

Mode	Algorithm	Description
0	Atomic Round Robin	Sequential request distribution using an atomic counter.
1	Atomic Least Connections	Routes requests to the backend with the fewest active connections.

---

🤖 AI Usage Disclosure

Gemini was used as a sounding board for discussing Go concurrency tradeoffs, validating implementation ideas, and debugging portions of the project. All architecture, implementation, benchmarking, and testing were completed by the project author.