Distributed timeouts for HTTP
Http-timeout is to HTTP what
It is great for triggering timed events in a distributed system:
- Http-timeout is an HTTP server that allows you to register HTTP requests to be made at a later time.
- Pending HTTP reqeusts are stored in a MongoDB database.
- Single instance of http-timeout runs several processes in a cluster to scale on multi-core machines.
- Http-timeout scales out to multiple machines by using the same MongoDB database.
- Peek-lock mechanism ensures a message will be delivered at least once even if an http-timout worker crashes.
- HTTP and HTTPS support, proxy support, re-delivery attempts, and redirect support.
- Windows, MacOS, or *nix (tested on Windows 7 & 2008 Server, MacOS Lion, Ubuntu 11.10)
- node.js v0.7.5 or greater.
- MongoDB. The database is used to store application metadata and must be accessible from all backends. (You can also get a free instance up and running quickly at MongoHQ)
npm install http-timeout
Make sure you are running unsecured MongoDB on localhost:
(Alternatively, use the
-m parameter to provide the URL of your MongoDB database when starting http-timeout in the next step).
sudo node ./node_modules/http-timeout/src/http-timeout.js
Issue a request directing http-timeout to send an HTTP GET request to https://www.google.com in 10 seconds:
About 10 seconds later, you should see one of the http-timeout worker processes reporting the request had been issued:
46706 (master): http-timeout started. Ctrl-C to terminate. 46709: Thu Mar 08 2012 17:11:18 GMT-0800 (PST) Registered: URL: https://www.google.com, Delay: 10 46709: Thu Mar 08 2012 17:11:28 GMT-0800 (PST) Delivered: id: 4f5958b6f607e275b6000001, URL: https://www.google.com, attempts: 1
Http-timeout is an HTTP and HTTPS server that expoes web APIs for scheduling HTTP or HTTPS requests to be
made at a later time. Requests to http-timeout service must use
POST methods. A
request must specify request options as URL query parameters. A
POST request can specify request options either
as URL query parameters or as properties of a JSON object pased in the body of the request.
Http-timeout accepts the following paramaters:
url(required) - an HTTP or HTTPS endpoint to call at a later time.
delay(required) - time in seconds to wait before the scheduled HTTP[S] request is made. Http-timeout guarantees that the request will be made at least once and not earlier than the scheduled time.
method(optional; default is
POST) - the HTTP method to use for the request.
body(optional; default none) - the entity body of the reqest to issue. It can only be specified with
headers(optional; default none) - HTTP request headers to include in the request. This property must be a JSON object whose property names are valid HTTP header names and property values are strings. It is best specified in the body of the
POSTrequest to http-timeout service.
Will issue an HTTP POST request to https://myservice.com with the HTTP request body value of
startBackup in about 30 minutes
Several configuration options can be specified when starting http-timeout:
Usage: node ./http-timeout.js Options: -m, --mongo Mongo DB connecton string [default: "mongodb://localhost/httpd"] -w, --workers Number of worker processes [default: 4] -p, --port HTTP listen port [default: 80] -s, --sslport HTTPS listen port [default: 443] -c, --cert Server certificate for SSL [default: "./cert.pem"] -k, --key Private key for SSL [default: "./key.pem"] -x, --proxy HTTP proxy in host:port format for outgoing requests [default: ""] -l, --lockDuration Peek lock duration (milliseconds) [default: 20000] -i, --peekInterval Cluster-wide peek interval (milliseconds) [default: 5000] -a, --maxPostSize Maximum size of a POST request in bytes [default: 8192] -t, --maxAttempts Maximum number of attempts before abandoning an event [default: 5] -r, --maxRedirects Maximum number of redirects when delivering the event [default: 0]
-m must specify the URL of the MongoDB database to use. By default it expects an unsecured instance to be running
on localhost. Http-timeout will create a
httpd collection in that database if it does not exist already.
-w option specifies the number of worker processes in a cluser. It defaults to the number of processors on the machine.
-s options specify the listen TCP ports for HTTP and HTTPS requests, respetively. In case of HTTPS,
server X.509 certificate and assocated private key files in PEM format are provided with
Sample cert and key file are checked in to get you started, but you will want to replace them with your own for any serious
-x option can be used to specify the HTTP proxy host and port for making outgoing HTTP[S] calls. The format is host:port,
-l option controls the duration of a peek lock a worker process creaets in the database when picking up requests
that are due for processing. The worker process subsequently has that much time to issue the requests and - if successful -
permanently remove the entry from the database. In case the worker crashes after picking up overdue requests from the database
or if the requests are unsuaccessful, another attempt to dispatch the requsts will only be made after the lock duration expires.
-i option controls the frequency with which the cluser of http-timeout workers polls the MongoDB database for
overdue notifications. With the value of 5000ms and 4 workers in the cluser, each worker will statistically poll every
-a option puts the limit on the maximum size of the HTTP POST request body the http-timeout service will accept.
-t option control how many delivery attempts are made for any single notification. Once this number is exceeded,
the notification is permanently removed from the database.
-r option controls how many redirects will be followed when issuing notifications. By default reirects are disabled.