Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Feed Pipes (braindump)
Input modules gather atom, rss (anything else) from different feed sources. Examples of feed sources would be polling of http resources, messages received by xmpp, notifications received by PubSubHubBub, http stream or longpoll or similar.
A feed generates feed entries. The entries are sanitized and normalized to a common JSON structure (like for ex. superfeedr).
New feed entries are routed to the appropriate pipes. The pipe defines a list of filter functions that the entry will go through. Each filter function receives the entry as JSON and outputs modified JSON. At the end the entry is submitted to one or more output modules.
a pipe example
declared in yaml: description: Pipe explanation feeds: - http://example.com/xyz.atom - xmpp:something ?? filters: - example1 - example2 publish: - http - xmpp - xmpp-pubsub
define the behaviour here, when is init called, what APIs are available, is state mutable or immutable...
- http stream or longpoll
- xmpp message
- xmpp pubsub
- pubsubhubbub publisher
- others could be specific modules for services like twitter, blogspot, etc?
- typical poller of rss/atom feeds
- pubsubhubbub subscriber
- stream/longpoll listener
The system will have to have some persistent storage. It needs to store info about the feed sources (feedId -> feedSource), active pipes (uuid -> pipe definition), storage of rss/atom documents...
Python (with gevent), Erlang, Go, Rust or Node.js ?