Skip to content

Latest commit

 

History

History
247 lines (159 loc) · 8.93 KB

README.md

File metadata and controls

247 lines (159 loc) · 8.93 KB
Raw

Winds - An Open Source Personalized RSS Reader

Winds - An Open Source Personalized RSS Reader

Build Status

Open source & beautiful RSS reader built using React/Redux/Sails/Node 7 and Stream (getstream.io). Showcases personalized feeds (using machine learning similar to Facebook, Flipboard, Etsy, and Quora - powered by the getstream.io API).

This tutorial explains how the personalization API works (blogpost). Check out the hosted demo at http://winds.getstream.io.

Note: We launched this project in November of 2016. We're actively working on it and contributions are much appreciated!

Screenshots

Winds - Open source RSS reader - Personalized feed

Winds - Open source RSS reader - Chronological feed

Winds - Open source RSS reader - Select topics

Demo

Check out the hosted demo: winds.getstream.io

Installation

Step 0 - Git clone

git clone https://github.com/GetStream/Winds.git

Step 1 - Node & requirements

cd Winds
brew install nvm
nvm install 7.0.0
npm install .
npm install -g sails pm2

Step 2 - Secrets

Create a file called .env with the following secrets:

STREAM_APP_ID = ''
STREAM_API_KEY = ''
STREAM_API_SECRET = ''
STREAM_ANALYTICS_TOKEN = ''
SENDGRID_USERNAME = ''
SENDGRID_PASSWORD = ''
SENTRY_DSN = ''
MONGO_URI = ''
API_BASE_URL = 'http://localhost:1337'

Stream

Stream handles the feed personalization and storage. Accounts are free up to 3 million feed updates and handle personalization (machine learning) for up to 100 users.

Get started at getstream.io and visit the dashboard to get your credentials.

While you're in the dashboard, you'll also want to create the following feed types:

  • rss_feed (type = flat, realtime notifications = off)
  • timeline (type = flat, realtime notifications = on)
  • user (type = flat, realtime notificatons = off)
  • topic (type = flat, realtime notifications = on)

Email

To send email create an account on sendgrid.com and add your username and password. You can use other providers by customizing config/emails.js

Error Reporting (optional)

To track errors create an account on sentry.io. Next add your Sentry DSN to the .env file.

Database (optional)

Sails uses an ORM called Waterline, which supports many databases. If you don't provide the Mongo URI ,it will store your data on local disk. This is fine for trying out the app; however it is not a solution for a production level app. The full details are available in config/connections.js.

Step 3 - Scrape Some Data

A reader without any data isn't much fun though. Let's insert a few topics and RSS feeds into the database:

node load_initial_data.js

Next, we need to run 2 cronjobs to ensure we keep on reading RSS articles and update the site's favicons. To make it easy to keep these cronjobs up and running, we use the amazing PM2 library:

pm2 start process.json

Step 4 - Launch!

Next, run sails lift in the command line like so:

sails lift

You can now see your own RSS reader at: localhost:1337

Step 5 - Enjoy

Point your browser to: localhost:1337, follow topics, create an account and add feeds as you please.

Contributing to This Project

Project Layout

React

The React codebase is located in /assets/js. There you'll find the actions, components and reducers.

API

The API is located in /api. It uses Sails, so their documentation is a good place to start.

Design

The Sketch File files are available for download via Invision.

Ideas for Improvements

Contributions are much appreciated. Here are some ideas for improvements:

  • Secondary links (ie comments link for HNews and Lobsters)
  • Batch OPML import
  • Deploy to Heroku button
  • Follow suggestions (we're working on this)
  • Switching between feeds should be easier
  • Lightweight task queuing system for emails and discover endpoint
  • Keyboard shortcuts (vim style)
  • GraphQL style APIs so you have more flexibility for building your own mobile apps
  • Android & iOS apps
  • Support more sites (RSS data quality is pretty poor and often needs custom logic per site/feed)
  • Search article's you've read using Algolia
  • Folders/Groups
  • Sharing support (e.g. Buffer, Facebook, Twitter, etc.)

Roadmap

At the moment we're gathering feedback from the community before deciding on the changes for Winds 0.2

Adding Support for Your Favorite Feeds

Unfortunately, RSS is more of a guideline than a standard. There is a good chance that the feed you're trying to add isn't correctly parsed. If this happens, there are two things you can do:

  1. You can submit an issue. Be sure to specify the exact url you tried to add. Every now and then we will go over these outstanding issues and try to resolve them.

  2. If you're a developer you'll want to fork our project. As a starting point you can add a test in discover.test.js and run it

NODE_ENV=testing mocha test/bootstrap.test.js test/integration/**/*discover* -g sentry

Next you'll want to open up ScrapingService.js and DiscoverService.js. Most of the time you can resolve the problem by adding an if statement in these files. If statements are of course an ugly solution. After we learn more about the type of customization required per feed, we'll add support for subclassing and extending the scraping logic.

Running the Test Cycle

We use Mocha for the test cycle. It's a pretty default setup for Sails. The only tricky bit is that you to specify the NODE_ENV as testing. You also need to load test/bootstrap.test.js before executing other tests. Here is an example:

Run all tests:

NODE_ENV=testing mocha test/bootstrap.test.js test/integration/**/**

Style Guide

Running JS Beautify

find . -name '*.js' | grep -v node_modules | grep -v tmp | xargs -n 1 js-beautify -r

Details About Operations & Cron Jobs

RSS is quite a broken standard. It works, but barely so. You'll often have to customize the scraping logic for specific sites. Here are a few commands which make it easier to test your feeds:

Scrape the feeds containing avc in the URL. Scrape only 1 article at the time and run with concurrency 1.

node scrape_feeds.js -q avc -a 1 -c 1

Scrape all feeds that weren't updated in the last 3 minutes:

node scrape_feeds.js

Force all feeds to be scraped:

node scrape_feeds.js -f

Scrape the favicons:

node scrape_favicons.js -c 10 -q cnn

Native OS Support

Winds supports a native OSX client through the use of https://github.com/electron/electron. An example application (OSX) pointed at the hosted version can be found in the /native/Winds-darwin-x64 directory of this repo.

Building Your Own Native Client

To make the build process easier, we chose to go with the popular tool Nativefier, a command line tool that allows you to easily create a desktop application for any web site with succinct and minimal configuration. Apps are wrapped by Electron in an OS executable (.app, .exe, etc.) for use on Windows, OSX and Linux.

To start, you'll need to install the Nativefier module from NPM in your terminal:

npm install nativefier -g

Then, run the following command to build on OSX:

nativefier --name "Winds" "http://your-domain.com" --icon "icon.png" --show-menu-bar

The full API documentation can be found at: https://github.com/jiahaog/nativefier/blob/development/docs/api.md

Powered By