Damjan Georgievski edited this page Oct 28, 2018 · 1 revision

Feed Pipes (braindump)

Input modules gather atom, rss (anything else) from different feed sources. Examples of feed sources would be polling of http resources, messages received by xmpp, notifications received by PubSubHubBub, http stream or longpoll or similar.

A feed generates feed entries. The entries are sanitized and normalized to a common JSON structure (like for ex. superfeedr).

New feed entries are routed to the appropriate pipes. The pipe defines a list of filter functions that the entry will go through. Each filter function receives the entry as JSON and outputs modified JSON. At the end the entry is submitted to one or more output modules.

a pipe example

declared in yaml: description: Pipe explanation feeds: - http://example.com/xyz.atom - xmpp:something ?? filters: - example1 - example2 publish: - http - xmpp - xmpp-pubsub

a filter

implemented in javascript: filter module: example1.js function init(System) { return state; } function filter(entry, state) { return [newEntry, newState]; } JSON in, JSON out

define the behaviour here, when is init called, what APIs are available, is state mutable or immutable...

output modules

  • http stream or longpoll
  • POST
  • xmpp message
  • xmpp pubsub
  • pubsubhubbub publisher
  • others could be specific modules for services like twitter, blogspot, etc?

input modules

  • typical poller of rss/atom feeds
  • pubsubhubbub subscriber
  • stream/longpoll listener
  • xmpp?


The system will have to have some persistent storage. It needs to store info about the feed sources (feedId -> feedSource), active pipes (uuid -> pipe definition), storage of rss/atom documents...


Python (with gevent), Erlang, Go, Rust or Node.js ?

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.