Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Experimental HTTP proxy (using node.js) for avoiding the dog pile effect.
JavaScript
Branch: master

Updated for Node 0.1.31

latest commit 9e69417f4f
Simon Willison authored
Failed to load latest commit information.
.gitignore dogproxy implementation
dogproxy.js Updated for Node 0.1.31
readme.txt fixed newlines, added project status note

readme.txt

dogproxy
========

A very simple dog pile preventing proxy.

The dog pile effect (also known as a thundering herd) affects caching systems
that use time-based expiry. In a heavy traffic system, a cached item expiring
could cause dozens of parallel "update" requests to be fired at once.

dogproxy assumes that these update requests go over HTTP. It joins together
simultaneous requests for the same URL, and only performs one actual fetch to
that resource.

STATUS: highly experimental. Known bugs (Content-Type is not correctly passed
through, for example). Expect to modify this code further before use.

Usage:

$ node dogproxy.js 
Server running at http://127.0.0.1:8009/

$ ab -n 10 -c 5 'http://127.0.0.1:8009/?url=http://example.com/'

dogproxy.js says:

Request for http://example.com/
... kicking off backend HTTP fetch
Request for http://example.com/
... already in flight, adding to queue
Request for http://example.com/
... already in flight, adding to queue
Request for http://example.com/
... already in flight, adding to queue
Request for http://example.com/
... already in flight, adding to queue
Fetched http://example.com/, alerting 5 waiting clients
Request for http://example.com/
... kicking off backend HTTP fetch
Request for http://example.com/
... already in flight, adding to queue
Request for http://example.com/
... already in flight, adding to queue
Request for http://example.com/
... already in flight, adding to queue
Request for http://example.com/
... already in flight, adding to queue
Fetched http://example.com/, alerting 5 waiting clients
Something went wrong with that request. Please try again.