Listens to a twitter stream and routes matching updates to HTTP posts.
Ruby
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
app_generators/twroute
bin
example_app
lib
test
.document
.gitignore
README.rdoc
Rakefile
VERSION
twroute.gemspec

README.rdoc

This project is completely abandonded don't use it.

I'm keeping it around just to keep a record of how I wrote it.

Twroute - Route Twitter status updates over http to your web app

Twroute uses the Twitter Stream API or the more stable Twitter Search API to watch status updates and convert them to HTTP post requests.

Twoute uses Delayed Job as a message queue ensuring that http requests are retried if the targeted web app is out of service for a bit.

As a Rails developer the motivation to write this gem is because I wanted a super simple way to write a twitter app where my app can recieve messages from twitter over http and respond to the sender by simply responding to the http request.

In other words an ordinary Rails/Merb/Sinatra controller can recieve a tweet and then reply to the sender by simply rendering a response.

Install

sudo gem install bhauman-twroute

Making your first Twroute app

Let's say we are going to create an app that rewards you for shooting someone in the ass and reprimands you for shooting someone in the arm, face, etc. It gives you a point for the bum shot and takes one away for the the others.

twroute bad_shot

This will create the following directory structure:

bad_shot
  |- config
       config.yml        # the configuration file
       twroutes.rb       # the routing file with sample routes
  |- db                  # holds the sqlite3 database
  |- log                 # holds the database log files
  |- test
       test_helper.rb    # defines the should_route_to macro     
       twroutes_test.rb  # sample tests for the sample routes

App Configuration

The config.yml file has 3 sections:

“Submit to” section

This is where you define the host, port and HTTP Auth inforomation for the web app that twroute will be routing requests to.

submit_to:
  host:               localhost
  port:               3000
# http_auth_user:     ''
# http_auth_password: ''

The port field is not required. Be sure to set the host to the target web app. For instance if you have a web application that lives at tweetrecorder.com, the config would look like:

submit_to:
  host:               tweetrecorder.com

Twitter section

This is where you specify the username and password for the twitter account that is making the requests to the Twitter Stream or Rest Search API and also authoring the replies to the senders.

Twitter Stream Api Configuration

This section also specifies which Twitter stream api to use. Most likely spritzer, follow or track unless you are privileged. See the Twitter Stream API docs for more information.

This section also describes the post query parameters that you want to send to the twitter-api call.

The following is an example configuration if you want to see all the tweets that have the word shoot in them.

twitter:
  user:            example_account_name
  password:        example_password
  stream_api:      track
  stream_api_args:  
    track: shoot
Twitter Search API Configuration

See the Twitter Search API for more information.

This section the post query parameters that you want to send to the Twitter search api call.

The following is an example configuration if you want to see all the tweets that have the word shoot in them.

twitter:
  user:            example_account_name
  password:        example_password
  search_params:
    q: 'shoot'  # search for updates with the text 'shoot'
    rpp: '99'     # return 99 results on one page : limit is 100

Database section

This section should be ready to go as is. If you want to use MySQL this configuration goes straight to ActiveRecord so set it up the way you would for a Rails project. I haven't used it for MySQL so no guarantees that it will work.

Routing

Testing First

We are going to use test driven development here. The system currently uses regex's. And even the best programmers have trouble getting regex's to work correctly.

For our example app bad_shot we have set of twitter updates that we would like to map to calls to our web app.

shoot @johnboy in the ass yeah buddy => /goodshot/create
shoot @johnboy in the arm            => /badshot/create
shoot @johnboy in the head           => /badshot/create
shoot @johnboy in the foot           => /badshot/create

Let's make these into tests first.

Open test/twroutes_test.rb it will have some sample tests in it. Delete them and replace them with these routing tests.

class TwroutesTest < Test::Unit::TestCase
  should_route_to "shoot @john_Boy3 in the ass yeah buddy", "/goodshot/create"
  should_route_to "shoot @john_Boy3 in the arm",            "/badshot/create"
  should_route_to "shoot @john_Boy3 in the head",           "/badshot/create"
  should_route_to "shoot @john_Boy3 in the foot",           "/badshot/create"
end

Now you can run rake test and see all of your tests fail. But that is a way better start than not having any tests at all.

Writing your routes

Let's try to make some of our tests pass.

First comment out all the tests except for the first one.

class TwroutesTest < Test::Unit::TestCase
  should_route_to "shoot @john_Boy3 in the ass yeah buddy", "/goodshot/create"
  # should_route_to "shoot @john_Boy3 in the arm",            "/badshot/create"
  # should_route_to "shoot @john_Boy3 in the head",           "/badshot/create"
  # should_route_to "shoot @john_Boy3 in the foot",           "/badshot/create"
end

We are going to only work on this one and then repeat the process for the other routes.

Open the config/twroutes.rb file in your editor of choice remove or comment the exiting routes. So it should read something like this:

Twroute::Routes.draw do |map|

end

Then add this first route:

Twroute::Routes.draw do |map|
  map.regex( {:whole_tweet => /shoot @john_Boy3 in the ass/},
             '/goodshot/create' )
end

Then execute rake test

You will notice that the test passed. (if not fix it). This is a good base to start from. Now lets refine the route one step at a time.

We want to match any username so add another test in twroutes_test.rb:

should_route_to "shoot @john_Boy3 in the ass yeah buddy", "/goodshot/create"
should_route_to "shoot @jannie_fly5 in the ass oh yeah", "/goodshot/create"

# the rest of the tests are commented out

Now the tests should fail. Change the Regex in the route like below:

map.regex( {:whole_tweet => /shoot @[\w\d_]+ in the ass/},
           '/goodshot/create' )

After each step check to see that the test passes. This should be passing now. We decided that white space shouldn't matter so we add a new test.

should_route_to "shoot @john_Boy3 in the ass yeah buddy", "/goodshot/create"
should_route_to "shoot @jannie_fly5 in the ass oh yeah", "/goodshot/create"
should_route_to "shoot   @jannie_fly5   in    the   ass     oh    yeah", "/goodshot/create"

# the rest of the tests are commented out

Tests fail, adjust the route:

map.regex( {:whole_tweet => /shoot\s+@[\w\d_]+\s+in\s+the\s+ass/},
           '/goodshot/create' )

So this is a passable route but it will match all kinds of things you wouldn't expect like “skeetshoot @marko in the ass” so more work could be done. Add a test:

should_route_to "shoot @john_Boy3 in the ass yeah buddy", "/goodshot/create"
should_route_to "shoot @jannie_fly5 in the ass oh yeah", "/goodshot/create"
should_route_to "shoot   @jannie_fly5   in    the   ass     oh    yeah", "/goodshot/create"
should_not_route_to "skeetshoot @marko in the ass oh yeah", "/goodshot/create"

Tests fail, adjust the routes:

map.regex( {:whole_tweet => /^shoot\s+@[\w\d_]+\s+in\s+the\s+ass/},
           '/goodshot/create' )

Alright you get the picture. This is the best way to go to ensure you are picking up the right tweets. As anomalies happen add tests and adjust the regex.

How the regex matcher works.

I am going to show you an advanced route.

map.regex( { :whole_tweet =>  /^shoot\s+@[\w\d_]+\s+in\s+the\s+(\w+).*/,
             :who_got_shot => [/^shoot\s+@([\w\d_]+)\s+/, 1]
             :shot_where =>   [/\s+in\s+the\s+(\w+).*/, 1]
            }, '/badshot/create' )

A few things are going on here. ALL the regexes defined have to be matched. The result of the match is a hash of values. :whole_tweet will be assigned the value of the whole tweet. The second match :who_got_shot has a back reference so it will only be assigned the name of the user who was shot. The third match :shot_where gets assigned the place where the person was shot.

Now what happens to the hash of matches? Two things

1 It gets posted along with the request as a hash named parsed[]. So for example in a Rails app you can refer to these parsed out goodies as

@name = params[:parsed][:who_got_shot]
@shot_where = params[:parsed][:shot_where]

2 The keys are available as substitutions in the target URL. So in the above example if we wanted to include some of the parsed items in the resulting url we could do this:

map.regex( { :whole_tweet =>  /^shoot\s+@[\w\d_]+\s+in\s+the\s+(\w+).*/,
             :who_got_shot => [/^shoot\s+@([\w\d_]+)\s+/, 1],
             :shot_where =>   [/\s+in\s+the\s+(\w+).*/, 1]
            }, '/badshot/create/name/:who_got_shot/where/:shot_where' )

Regex matcher and procs

We could rewrite the above example to use a proc as well.

map.regex( { :whole_tweet =>  /^shoot\s+@[\w\d_]+\s+in\s+the\s+(\w+).*/,
             :who_got_shot => [/^shoot\s+@([\w\d_]+)\s+/, 1],
             :shot_where =>   lambda { |tweet_text|
                match_data = tweet_text.match(/\s+in\s+the\s+(\w+).*/) 
                match_data ? match_data[1] : nil
              }
            },
            '/badshot/create/name/:who_got_shot/where/:shot_where')

If the proc returns nil it will not be considered a match and the next route will be tried.

Routing order

The routes are executed in a similar manner as Routes in Rails. They are tried in top down order. As soon as a match is found the rest of the routes are ignored.

What gets posted?

When a route is finally selected and a path is chosen three hashes get posted to the selected url:

The Parsed Hash

In our example this will be:

parsed[whole_tweet]:   shoot @johnny in the wild 
parsed[who_got_shot]:  johnny
parsed[shot_where]:    wild

The Twitter Tweet Hash - for Search API

tweet[id]:                       3317086732
tweet[source]:                web
tweet[profile_image_url]:     http://s.twimg.com/a/1252620925/images/default_profile_normal.png
tweet[to_user_id]: 
tweet[from_user]:             dailydid_dev
tweet[iso_language_code]:     en
tweet[text]:                  #test 123 #dailydid
tweet[from_user_id]:          50354397
tweet[created_at]:            Fri, 04 Sep 2009 17:03:56 +0000

The Twitter Tweet Hash - for Stream API

For example:

tweet[in_reply_to_screen_name]: 
tweet[id]:                       3317086732
tweet[created_at]:               Fri Aug 14 22:31:44 +0000 2009
tweet[in_reply_to_user_id]: 
tweet[favorited]:                false
tweet[truncated]:                false
tweet[source]:                   <a href="http://www.tweetdeck.com/" rel="nofollow">TweetDeck</a>
tweet[in_reply_to_status_id]: 
tweet[text]:                     shoot @johnny in the wild

The Twitter User Hash - for Stream API only

We pull this out of the tweet and post it as the sender[] hash:

sender[following]: 
sender[friends_count]: 86
sender[followers_count]: 113
sender[profile_link_color]: "990000"
sender[protected]: false
sender[profile_sidebar_border_color]: DFDFDF
sender[notifications]: 
sender[screen_name]: Sarahndipitea
sender[name]: Sarah
sender[profile_sidebar_fill_color]: F3F3F3
sender[created_at]: Tue Oct 21 03:06:15 +0000 2008
sender[id]: 16880192
sender[location]: Stumptown
sender[profile_image_url]: http://s3.amazonaws.com/twitter_production/profile_images/332336440/EllieUp_normal.jpg
sender[description]: You found me! Were you even *looking* for me?
sender[favourites_count]: 12
sender[profile_background_image_url]: http://static.twitter.com/images/themes/theme7/bg.gif
sender[statuses_count]: 9601
sender[profile_background_tile]: false
sender[verified]: false
sender[profile_background_color]: EBEBEB
sender[profile_text_color]: "333333"
sender[time_zone]: Pacific Time (US & Canada)
sender[utc_offset]: -28800
sender[url]: http://Sarahndipitea.wordpress.com

(_This tweet was completely chosen randomly. I didn't want to take time to come up with fake data. If you have some funny data. Send me a pull request._)

Gentlemen start your servers.

So now we setup the config file and we have have tests and routes.

First initialize the Sqlite3 DB

Execute the following rake command:

rake twroute:init

Create a sample app to receive the post requests

You might want to set up a sample Rails app and tail the development log at this point.

rails sample_twroute_app
cd sample_twroute_app
./script/server

No need to create controllers that do anything. This just so you can see that the requsts are being made.

Start the twroute_runner daemon

This is the daemon that pulls down the tweets from the twitter stream api, routes them and stores them as a delayed job.

Change to the bad_shot directory and execute:

twroute_runner start

At this point IF there are tweets that match your stream api query AND these tweets also match your routes defined in twroutes.rb then you should see some delayed_jobs being created in log/database.log

You can stop this daemon by executing:

twroute_runner stop

Start the twroute_worker daemon

Seeing as we are creating Delayed Jobs we have to have a daemon that executes the jobs. You can start it in a similar fashion.

Change to the bad_shot directory and execute:

twroute_worker start

At this point IF there are jobs stored in the Sqlite3 database then you should see activity in the log/delayed.log

You can stop this daemon by executing:

twroute_worker stop

Start/Stop Short Cut

You can start and stop both the daemons using the rake twroute:start and rake twroute:stop commands.

Troubleshooting

No output

You can check that your Twitter stream request is functioning by temporarily making a catchall route at the end of your routes like so:

map.regex( { :match_all_tweets =>  /.*/ }, '/badshot/create' )

This will force all tweets from the stream to be sent to your app. This is very useful to verify that tweets are coming down and going through the system.

Responses will be Tweeted!

When your web app responds with a 200 status and with the ContentType header set to text/twitter Twroute will tweet the response.

It won't respond to the sender. It will simply tweet the response. So if you want to respond you need to specify the @user at the beginning of the tweet.

Monit and God

Don't forget to setup either Monit, God or some other server process monitor because Twitter reserves the right to close the connection whenever they want. I would say that it's fairly safe to simply restart the daemons periodically as well.

License

(The MIT License)

Copyright © 2009 Bruce Hauman <bhauman@gmail.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Copyright © 2009 Bruce Hauman, released under the MIT license