A declarative HATEOAS API crawler for node.js
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
test stream responses, not bodies Sep 25, 2012
.gitignore Initial commit Aug 8, 2012
.travis.yml add travis-ci Aug 8, 2012
README.md update readme Sep 25, 2012
hyperspider.js stream responses, not bodies Sep 25, 2012
package.json [ci skip] Aug 9, 2012



Build Status

hyperspider is a declarative HATEOAS API crawler for node.js. Give it a list of url patterns, and it will recursively crawl your hypertext HTTP API, streaming back every matching endpoint.

hyperspider is great for folks that want to create clean, granular, and self-documenting hypertext APIs, but avoid the latency of remotely fetching hundreds of tiny HTTP resources.


Let's say you had a Twitter clone with a hypertext JSON API, including the following two sample endpoints:

GET /users/:id (fetch a user)

  "href": "/users/jedschmidt",
  "name": "Jed Schmidt",

  "location":  { "href": "/japan/tokyo" },
  "updates":   { "href": "/users/jedschmidt/updates" },
  "following": { "href": "/users/jedschmidt/following" },
  "followers": { "href": "/users/jedschmidt/followers" }

GET /users/:id/following (fetch the users a user is following)

  "href": "/users/jedschmidt/following",
  "items": [
    { "href": "/users/janl" },
    { "href": "/users/cramforce" },
    { "href": "/users/hblank" },
    { "href": "/users/theophani" }

To create a single following-detailed resource so that API consumers don't have to make a separate HTTP call for each resource, use hyperspider like this:

var hyperspider = require("hyperspider")

var options = {
  host: "mytwitterclone.biz",
  path: [

hyperspider(options, function(err, data) {
  // data is an array with the result entities of 6 endpoints:
  // 1. /users/jedschmidt
  // 2. /users/jedschmidt/following
  // 3. /users/janl
  // 4. /users/cramforce
  // 5. /users/hblank
  // 6. /users/theophani

See the tests for a working example.


Use npm to install hyperspider:

npm i hyperspider


req = hyperspider(options, [callback])

options: An object containing the same options as for node's http.request method, with one exception: path can be any of the following:

  • a normal url path, such as /users
  • a function that takes a path and returns a boolean
  • a RegExp that matches a path
  • a wildcard url path to turn into a RegExp, with * replaced by [^/]+ and ** replaced by .*?
  • an array containing any of the above

Note that path must contain at least one normal url path to serve as the starting point for the crawl.

callback(err, data): Buffers the event stream into a single callback. err is null if no errors occurred, and otherwise the first error emitted, with subsequent errors available from the next property of the error. data is an array of responses, one for each successful HTTP call. Omit this to listen for a stream of events.

req.extract (or hyperspider.prototype.extract)

This method takes a single argument, an HTTP response. Override this before adding listeners to customize how hyperspider should extract urls from each resource. By default, the response body is parsed into JSON, extracting the value of every nested href property, but you could roll your own, such as parsing Link headers, etc.