Twitter Realtime Plugin

amygdala edited this page Apr 7, 2011 · 1 revision
Clone this wiki locally

Information on the ThinkUp Twitter Realtime (Streaming) Branch

The links below provide information on setting up and using the Twitter Realtime (User Streams) functionality in ThinkUp. The plugin sets up a persistent connection to the Twitter User Streams API, for each active Twitter account (instance) in your ThinkUp installation. The Twitter Realtime plugin runs fine in conjunction with the Twitter crawler, so you can continue to run the crawler as usual at the same time.

This functionality is in an alpha state and as of 2011-04-07 is not yet officially merged into the ThinkUp repository, though it will be added shortly as an 'experimental' plugin, at which time this information will be updated.

To run the code in its current state, you need to have access to and be comfortable using the command line— these scripts can not yet be triggered via the web browser. Eventually, things will become more shrink-wrapped. This branch is also missing some tests; more are in the process of being added, and more complete coverage will be in place before the plugin is added to the ThinkUp repository. (Currently, there is good model test coverage, but more plugin tests, particularly with respect to process management, will be added.)

It's not yet known whether this code runs as is on Windows (that's what alpha testers are for :) but for sure it will require Cygwin to be installed.

The streaming functionality is implemented via several (indefinitely-)long-running stream processes. So, an ISP such as Dreamhost, that kills long-running processes, may require some special attention. The script that launches the streams now checks to see which existing stream processes have shown recent activity, and kills and restarts those that have not. So, if you put this startup script on a cron, you can regularly restart any streams that have been killed off. Of course, there will be some gaps in the data stream if processes are regularly killed off. Such gaps will be filled in for the most part via the crawler.