A river-of-news RSS aggregator in JS running in Node.
JavaScript HTML
Latest commit c05ba8c Feb 20, 2017 @scripting v0.47
Permalink
Failed to load latest commit information.
docs v0.44 Feb 26, 2016
lib v0.47 Feb 21, 2017
lists v0.47 Feb 10, 2017
misc v0.47 Feb 21, 2017
README.md v0.47 Feb 21, 2017
config.json v0.47 Feb 10, 2017
package.json v0.47 Feb 10, 2017
river5.js v0.43 Feb 9, 2016

README.md

River5

River5 is a river-of-news RSS aggregator in JavaScript running in Node, written by Dave Winer.

How River5 works

When it starts up, River5 reads files in the lists folder at the top level of the River5 folder.

Each list contains a set of URLs of feeds. The list files can be straight text, a JSON array of URLs of feeds, or a traditional OPML subscription list, with extensions .txt, .json and .opml. River5 ignores files in the lists folder that are not in one of these formats.

The feeds pointed to from the lists can be in RSS 0.9x, 1.0 or 2.0, or Atom 1.0.

Every 15 minutes, River5 reads all the feeds you're subscribed to, and routes new items to corresponding files in the rivers folder at the top level of the River5 folder. You're subscribed to a feed if it appears in one or more of your lists. The river files are a documented form of JSONP that we've been using since River3, in 2010.

To read the output of River5 on the machine it's running on, go to the home page. Assuming your river is running on the default port, 1337, you would go to this address to read the rivers. http://localhost:1337/. If it's running on a public server, just replace localhost with the domain name or IP address of the server.

Pretty much everything in this narrative is configurable.

Requirements

A current Node.js installation.

The machine running River5 ahould able to access the Internet.

It is possible to run the software on a non-public machine. Of course, you won't be able to access the home page remotely, or receive realtime notifications of feed updates. You'll also have to find a way to upload the output files in the rivers folder to a place where they can be accessed, if you want others to be able to access your river.

If you don't have Node.js installed on your machine, here are step by step instructions for installing Node.js on a Macintosh.

How to install

  1. Download the folder from the repository to your local computer.

  2. From the command line, go to this folder.

  3. npm install

  4. node river5.js

If it's working properly, you should see tons of stories scroll by in the console. River5 is reading each of the feeds in the lists for the first time. When you see things settle down, then you should open the home page of the RIver5 website, at http://localhost:1337/ and see the result of the first feed readings.

At first there will be a lot of old stories, but as the rivers run for hours and days and into weeks, you'll get the flow. It's a good way to read news.

Examples of lists

I have included several files in the lists folder to help you get started, so something actually happens the first time you run River5. You can edit, consolidate or delete them, as you wish.

  1. myJsonFeeds.json -- a list of URLs of feeds in the JSON format that River5 understands.

  2. myOpmlFeeds.opml -- a list of URLs of feeds in the standard OPML format for subscription lists.

  3. myTxtFeeds.txt -- a list of URLS in a text file.

  4. nyt.opml -- a list of a New York Times news feeds.

  5. hn.opml -- a list with just one feed in it, the Hacker News firehose. It's a good list for testing RIver5 because it updates so frequently.

To view your rivers

Go to http://localhost:1337/.

Configuring

There's only one way to configure it, via the config.json file in the same folder as the river5.js app.

The example config.json in the folder sets the max number of items in a river to 300.

All of the config.json options and their default values are listed on this page.

For River4 users

River5 does not support S3 storage, so if that's how you're running your rivers, you should continue to use River4.

  1. If you're running out of the filesystem, then River5 can replace River4.

  2. Follow the installation instructions above, and before you launch river5.js, remove the files from the lists folder, and replace them with the lists from your River4 installation.

  3. It will take a while before your rivers re-populate, so I recommend doing it in parallel with your existing River4 install.

You may want to review the Configuration settings page, there are options for setting new things, and in some cases the defaults have changed.

Where's the code?

The heart of River5 is in a Node package in the lib folder, called feedtools.js.

The plan is to eventually offer it through the Node.js package distribution system so we can easily include feed functionality in other apps. For right now, feedtools.js is set up just to work with river5.js.

Examples of working rivers

  1. Podcatch.com subscribes to my friends' podcast feeds.

  2. mlbriver.com and nbariver.com collect news for two sports I love.

  3. I have a River panel on the Scripting News home page.

  4. My rivers page puts all the news I care about on a single tabbed page, with panels for the NYT, Washington Post, Guardian, Movies, Tech, baseball and basketball.

Other docs

  1. The road to River5, reviews the previous versions of my river-of-news software going back to 1999, with a special focus on the transition from River4 to River5.

  2. The Hello World of Rivers shows you how to create a standalone river, one that people can read without accessing your RIver5 server.

  3. My blog post announcing River5.

Updates

v0.47 -- 2/9/17 by DW

Fixed two problems reported by Andrew Shell. Added some feeds to the default set up, and configured the default home page so it only has three panels and they have intelligent labels, using the relatively new homePage configuration option. Fixed a problem with NYT feeds and cookies. We now set the User-Agent header on requests. Apparently some servers won't respond unless this header is present.

v0.46 -- 11/16/16 by DW

Refreshed the admin app, relocated it, and fixed a couple of entry-points it depends on that didn't work with CORS.

v0.45 -- 3/14/16 by DW

Fixed a bug that prevented addToRiver callbacks from working correctly. When you made a change to an item in a river it wasn't actually being reflected back in the river.

As a result of this fix, River4 callbacks should now work in River5.

v0.44 -- 2/22/16 by DW

Fixed a bug that prevented text file lists from working. It would only subscribe to the last feed in the list.

v0.43 -- 2/4/16 by DW

You can now configure the panels on the server's home page.

The new configuration feature is documented in a new section on the configuration page.

v0.42 -- 2/4/16 by DW

We accept two new formats for files in the lists folder -- straight text and JSON, in addition to the traditional OPML subscription lists.

If a file in the lists folder has a .txt extension, we assume it's just a series of lines with the URLs of feeds. It's the simplest of all the list formats.

If it has a .json format, we look for a JSON file that's just an array of URLs.

Created a new top-level lists folder containing example lists in each of the three supported formats.

v0.41 -- 2/3/16 by DW

We now maintain a new data structure for each list, riverData.json, in the same folder as the listInfo.json file.

It contains an array of river items that are part of this river. To build the riverjs file, all we have to do is loop over that array.

It's kept in memory while River5 is running, so now building a river on reads a file from the disk the first time a river is built or an item is added to it.

One of the first bennies of rebuilding the river software, it's now quite a bit faster at building rivers, and it's no longer a function of how dense the river is. In previous versions, rivers that don't update often would require us to go back a long ways into the calendar structure to build the river.

Now the calendar structure is just an archive, it's no longer used to build anything. It could be made optional.

Questions, comments?

Please post a note on the River5 mail list.