Pilgrim is a prototype tool for assisting in web-based research.
This project was initiated with generous support from the Knight Foundation Prototype Fund.
- Install node modules:
npm install - Install Memcached:
brew install memcached - Install Redis:
brew install redis - Create a new
.envfile and copy contents from.env.example - Run the server
npm run dev
Routes can be found in server/api/index.js
Returns extracted content from a url, along with all inner links.
Example result
Request: /api/http%3A%2F%2Fwww.ribbonfarm.com%2F2016%2F02%2F11%2Fminimum-viable-superorganism%2F
Result:
{
"html": (stripped html from webpage),
"title": "Minimum Viable Superorganism",
"hrefs": [
"http://www.meltingasphalt.com/tears/",
"https://en.wikipedia.org/wiki/Reciprocal_altruism",
"https://en.wikipedia.org/wiki/Free_rider_problem",
"http://www.ribbonfarm.com/2015/09/29/what-is-the-largest-collective-action-ever/",
"http://peterturchin.com/blog/2016/01/19/naked-self-interest-is-a-recipe-for-social-dissolution-a-response-to-branko-milanovic/",
"http://www.bartleby.com/17/1/67.html",
"http://www.meltingasphalt.com/social-status-down-the-rabbit-hole/",
"https://en.wikipedia.org/wiki/Competitive_altruism",
"https://en.wikipedia.org/wiki/Enlightened_self-interest",
"https://twitter.com/KevinSimler/status/650336275954860033",
"https://twitter.com/mdc/status/573547255115968514",
"https://en.wikipedia.org/wiki/Agent_detection",
"http://billmoyers.com/2014/02/21/anatomy-of-the-deep-state/",
"http://slatestarcodex.com/2014/09/14/does-class-warfare-have-a-free-rider-problem/",
"https://en.wikipedia.org/wiki/Trade_association",
"https://en.wikipedia.org/wiki/FWD.us",
"http://www.amazon.com/Hierarchy-Forest-Evolution-Egalitarian-Behavior/dp/0674006917",
"http://www.amazon.com/SuperCooperators-Altruism-Evolution-Other-Succeed/dp/1451626630",
"http://www.amazon.com/Why-We-Talk-Evolutionary-Evolution/dp/0199276234"
],
"url": "http://www.ribbonfarm.com/2016/02/11/minimum-viable-superorganism/",
"host": "www.ribbonfarm.com",
"cached": 1461013525246
}
Pilgrim is built on the shoulders of giants and we'd like to highlight a few of the open-source projects that made it possible (and enjoyable) to develop: