Skip to content
Functional streaming news aggregator. For educational purposes. And fun :-)
JavaScript
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config
helpers
models
setup
.gitignore
LICENSE
README.md
api.js
collector.js
error-processing.js
package.json
templates.js
websocket-api.js

README.md

Collectify

Introduction:

A functional streaming news aggregator with real-time distribution. Mostly for educative purposes. (uses babel-node to support ES6+).

  • collector.js will

    • store and update articles in a mongodb collection
    • publish data to applicable channels/listeners
  • api.js will

    • expose the data (articles) stored in the database as a HTTP API
  • websocket-api.js will

    • listen for data being piped into the articles:* channel
    • expose that data real-time as a websocket api
  • error-processing.js will

    • listen for data being piped to the errors channel
    • collect all error messages from all you processes
    • handle errors any way you want (just logging for now)

(More docs coming..)

Example of of a template for collecting data/articles

{
  "active": true,
  "type": "site",
  "name": "New Yorker",
  "url": "http://www.newyorker.com/",
  "format": "desktop",
  "template": {
    "containers": [{
      "selector": "article",
      "elements": [{
        "name": "guid",
        "type": "url",
        "occurence": "first",
        "required": true,
        "items": [{
          "selector": "section h2 a",
          "attribute": "href"
        }]
      }, {
        "name": "url",
        "type": "url",
        "occurence": "first",
        "required": true,
        "items": [{
          "selector": "section h2 a",
          "attribute": "href"
        }]
      }, {
        "name": "title",
        "required": true,
        "occurence": "first",
        "items": [{
          "selector": "section h2 a"
        }]
      }, {
        "name": "image",
        "type": "url",
        "occurence": "first",
        "fallback": null,
        "items": [{
          "selector": "figure a img",
          "attribute": "src"
        }, {
          "selector": "figure a img",
          "attribute": "data-lazy-src"
        }]
      }]
    }]
  }
}

Example of data collected

{
  _source: 'New Yorker',
  _origin: 'http://www.newyorker.com/',
  _host: 'www.newyorker.com',
  guid: 'http://www.newyorker.com/culture/richard-brody/movie-of-the-week-the-marriage-circle',
  url: 'http://www.newyorker.com/culture/richard-brody/movie-of-the-week-the-marriage-circle',
  title: 'Movie of the Week: “The Marriage Circle”',
  image: 'http://www.newyorker.com/wp-content/assets/dist/img/header_graphics/contributors/brody.png',
  _ranking: 26,
  createdAt: 1447428348324,
  shares: {
    facebook: {
      url: 'http://www.newyorker.com/culture/richard-brody/movie-of-the-week-the-marriage-circle',
      normalized_url: 'http://www.newyorker.com/culture/richard-brody/movie-of-the-week-the-marriage-circle',
      share_count: 5,
      like_count: 15,
      comment_count: 6,
      total_count: 26,
      click_count: 0,
      comments_fbid: '964017293671043',
      commentsbox_count: 0
    },
    twitter: {
      count: 8,
      url: 'http://www.newyorker.com/culture/richard-brody/movie-of-the-week-the-marriage-circle/' },
    linkedin: {
      count: 0,
      fCnt: '0',
      fCntPlusOne: '1',
      url: 'http://www.newyorker.com/culture/richard-brody/movie-of-the-week-the-marriage-circle'
    }
  },
  content: '(Content from the article)...arose in 1981 over the director Ernst Lubitsch and his second Hollywood feature, “The Marriage Circle...',
  keywords: [
    'splendid',
    'controversy',
    'arose',
    'director',
    'ernst'
  ]
}

Install dependencies (some might need to use sudo for various reasons):

  • brew/apt-get install iojs
  • brew/apt-get install redis
  • brew/apt-get install mongodb
  • npm install -g gulp
  • npm install -g babel
  • npm install

Development shellscript example for collector:

#!/bin/sh

export DEBUG="*" \
export NODE_ENV="development" \
export MONGO_URL="mongodb://localhost/collectify" \
export REDIS_URL='redis://localhost:6379' \
export REDIS_PREFIX='collectify'

babel-node --stage 0 collector

Development shellscript example for api:

#!/bin/sh

export DEBUG="*" \
export NODE_ENV="development" \
export MONGO_URL="mongodb://localhost/collectify" \

babel-node --stage 0 api

Development shellscript example for websocket-api:

#!/bin/sh

export DEBUG="*" \
export NODE_ENV="development" \
export REDIS_URL='redis://localhost:6379' \
export REDIS_PREFIX='collectify'

babel-node --stage 0 websocket-api

Development shellscript example for error-processing:

#!/bin/sh

export DEBUG="*" \
export NODE_ENV="development" \
export REDIS_URL='redis://localhost:6379' \
export REDIS_PREFIX='collectify'

babel-node --stage 0 error-processing
You can’t perform that action at this time.