Bookmarklet and manual webcrawler to aid in web research
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
assets
build
client
common
config
server
.babelrc
.env.example
.gitignore
LICENSE.md
Procfile
Procfile.dev
README.md
assets.json
config.js
index.js
newrelic.js
npm-shrinkwrap.json
package.json
webpack.common.config.js
webpack.development.config.js
webpack.production.config.js
webpack.server.config.js

README.md

Pilgrim by Are.na

Pilgrim is a prototype tool for assisting in web-based research.

This project was initiated with generous support from the Knight Foundation Prototype Fund.

Pilgrim screenshot Pilgrim screenshot

Table of contents

Installation

  1. Install node modules: npm install
  2. Install Memcached: brew install memcached
  3. Install Redis: brew install redis
  4. Create a new .env file and copy contents from .env.example
  5. Run the server npm run dev

API

Routes can be found in server/api/index.js

/api/:url

Returns extracted content from a url, along with all inner links.

Example result

Request: /api/http%3A%2F%2Fwww.ribbonfarm.com%2F2016%2F02%2F11%2Fminimum-viable-superorganism%2F

Result:

{
  "html": (stripped html from webpage),
  "title": "Minimum Viable Superorganism",
  "hrefs": [
    "http://www.meltingasphalt.com/tears/",
    "https://en.wikipedia.org/wiki/Reciprocal_altruism",
    "https://en.wikipedia.org/wiki/Free_rider_problem",
    "http://www.ribbonfarm.com/2015/09/29/what-is-the-largest-collective-action-ever/",
    "http://peterturchin.com/blog/2016/01/19/naked-self-interest-is-a-recipe-for-social-dissolution-a-response-to-branko-milanovic/",
    "http://www.bartleby.com/17/1/67.html",
    "http://www.meltingasphalt.com/social-status-down-the-rabbit-hole/",
    "https://en.wikipedia.org/wiki/Competitive_altruism",
    "https://en.wikipedia.org/wiki/Enlightened_self-interest",
    "https://twitter.com/KevinSimler/status/650336275954860033",
    "https://twitter.com/mdc/status/573547255115968514",
    "https://en.wikipedia.org/wiki/Agent_detection",
    "http://billmoyers.com/2014/02/21/anatomy-of-the-deep-state/",
    "http://slatestarcodex.com/2014/09/14/does-class-warfare-have-a-free-rider-problem/",
    "https://en.wikipedia.org/wiki/Trade_association",
    "https://en.wikipedia.org/wiki/FWD.us",
    "http://www.amazon.com/Hierarchy-Forest-Evolution-Egalitarian-Behavior/dp/0674006917",
    "http://www.amazon.com/SuperCooperators-Altruism-Evolution-Other-Succeed/dp/1451626630",
    "http://www.amazon.com/Why-We-Talk-Evolutionary-Evolution/dp/0199276234"
  ],
  "url": "http://www.ribbonfarm.com/2016/02/11/minimum-viable-superorganism/",
  "host": "www.ribbonfarm.com",
  "cached": 1461013525246
}

Credits

Pilgrim is built on the shoulders of giants and we'd like to highlight a few of the open-source projects that made it possible (and enjoyable) to develop: