Skip to content
Browse files

updated readme with info about the new interface and features

  • Loading branch information...
1 parent e5532a3 commit 6cf2f56b6bdc2a27057da8ec63f1378056bf924a @pauldix pauldix committed Oct 29, 2009
Showing with 115 additions and 120 deletions.
  1. +115 −120 README.textile
View
235 README.textile
@@ -12,14 +12,13 @@ h2. Summary
Like a modern code version of the mythical beast with 100 serpent heads, Typhoeus runs HTTP requests in parallel while cleanly encapsulating handling logic. To be a little more specific, it's a library for accessing web services in Ruby. It's specifically designed for building RESTful service oriented architectures in Ruby that need to be fast enough to process calls to multiple services within the client's HTTP request/response life cycle.
-Some of the awesome features are parallel request execution, memoization of request responses (so you don't make the same request multiple times in a single group), built in support for caching responses to memcached (or whatever), a nifty DSL for creating classes that make http calls and process responses, and mocking capability baked in. It uses libcurl and libcurl-multi to work this speedy magic. I wrote the c bindings myself so it's yet another Ruby libcurl library, but with some extra awesomeness added in.
+Some of the awesome features are parallel request execution, memoization of request responses (so you don't make the same request multiple times in a single group), built in support for caching responses to memcached (or whatever), and mocking capability baked in. It uses libcurl and libcurl-multi to work this speedy magic. I wrote the c bindings myself so it's yet another Ruby libcurl library, but with some extra awesomeness added in.
h2. Installation
-For now Typhoeus exists only on github. It requires you to have a current version of libcurl installed. I've tested this with 7.19.4.
+Typhoeus requires you to have a current version of libcurl installed. I've tested this with 7.19.4 and higher.
<pre>
-gem sources -a http://gems.github.com # if you haven't already
-gem install pauldix-typhoeus
+gem install typhoeus --source http://gemcutter.org
</pre>
If you're on Debian or Ubuntu and getting errors while trying to install, it could be because you don't have the latest version of libcurl installed. Do this to fix:
<pre>
@@ -30,139 +29,139 @@ Another problem could be if you are running Mac Ports and you have libcurl insta
If you're still having issues, please let me know on "the mailing list":http://groups.google.com/group/typhoeus.
-There's one other thing you should know. The Easy object (which is just a libcurl thing) allows you to set timeout values in milliseconds. However, for this to work you need to build libcurl with c-ares support built in. Unfortunately, I still haven't been able to get this to work on either my Mac or Linux. I'll have to figure that out.
+There's one other thing you should know. The Easy object (which is just a libcurl thing) allows you to set timeout values in milliseconds. However, for this to work you need to build libcurl with c-ares support built in.
h2. Usage
+*Deprecation Warning!*
+The old version of Typhoeus used a module that you included in your class to get functionality. That interface has been deprecated. Here is the new interface.
+The primary interface for Typhoeus is comprised of three classes: Request, Response, and Hydra. Request represents an HTTP request object, response represents an HTTP response, and Hydra manages making parallel HTTP connections.
<pre>
require 'rubygems'
require 'typhoeus'
-require 'json'
-
-# here's an example for twitter search
-# Including Typhoeus adds http methods like get, put, post, and delete.
-# What's more interesting though is the stuff to build up what I call
-# remote_methods.
-class Twitter
- include Typhoeus
- remote_defaults :on_success => lambda {|response| JSON.parse(response.body)},
- :on_failure => lambda {|response| puts "error code: #{response.code}"},
- :base_uri => "http://search.twitter.com"
-
- define_remote_method :search, :path => '/search.json'
- define_remote_method :trends, :path => '/trends/:time_frame.json'
-end
-
-tweets = Twitter.search(:params => {:q => "railsconf"})
-
-# if you look at the path argument for the :trends method, it has :time_frame.
-# this tells it to add in a parameter called :time_frame that gets interpolated
-# and inserted.
-trends = Twitter.trends(:time_frame => :current)
-
-# and then the calls don't actually happen until the first time you
-# call a method on one of the objects returned from the remote_method
-puts tweets.keys # it's a hash from parsed JSON
-
-# you can also do things like override any of the default parameters
-Twitter.search(:params => {:q => "hi"}, :on_success => lambda {|response| puts response.body})
+require 'json'
+
+# the request object
+request = Typhoeus::Request.new("http://www.pauldix.net",
+ :body => "this is a request body",
+ :method => :post,
+ :headers => {:Accepts => "text/html"},
+ :timeout => 100,
+ :cache_timeout => 60,
+ :params => {:field1 => "a field"})
+# we can see from this that the first argument is the url. the second is a set of options.
+# the options are all optional. The default for :method is :get. Timeout is measured in milliseconds.
+# cache_timeout is measured in seconds.
+
+# the response object will be set after the request is run
+response = request.response
+response.code # http status code
+response.time # time in seconds the request took
+response.headers # the http headers
+response.body # the response body
+</pre>
-# on_success and on_failure lambdas take a response object.
-# It has four accesssors: code, body, headers, and time
+*Making Quick Requests*
+The request object has some convenience methods for performing single HTTP requests. The arguments are the same as those you pass into the request constructor.
+<pre>
+response = Typhoeus::Request.get("http://www.pauldix.net")
+response = Typhoeus::Request.put("http://localhost:3000/posts/1", :body => "whoo, a body")
+response = Typhoeus::Request.post("http://localhost:3000/posts", :params => {:title => "test post", :content => "this is my test"})
+response = Typhoeus::Request.delete("http://localhost:3000/posts/1")
+</pre>
-# here's and example of memoization
-twitter_searches = []
-10.times do
- twitter_searches << Twitter.search(:params => {:q => "railsconf"})
+*Making Parallel Requests*
+<pre>
+# Generally, you should be running requests through hydra. Here is how that looks
+hydra = Typhoeus::Hydra.new
+
+first_request = Typhoeus::Request.new("http://localhost:3000/posts/1.json")
+first_request.on_complete do |response|
+ post = JSON.parse(response.body)
+ third_request = Typhoeus::Request.new(post.links.first) # get the first url in the post
+ third_request.on_complete do |response|
+ # do something with that
+ end
+ hydra.queue third_request
+ return post
end
-
-# this next part will actually make the call. However, it only makes one
-# http request and parses the response once. The rest are memoized.
-twitter_searches.each {|s| puts s.keys}
-
-# you can also have it cache responses and do gets automatically
-# here we define a remote method that caches the responses for 60 seconds
-klass = Class.new do
- include Typhoeus
-
- define_remote_method :foo, :base_uri => "http://localhost:3001", :cache_responses => 60
+second_request = Typhoeus::Request.new("http://localhost:3000/users/1.json")
+second_request.on_complete do |response|
+ JSON.parse(response.body)
end
+hydra.queue first_request
+hydra.queue second_request
+hydra.run # this is a blocking call that returns once all requests are complete
-klass.cache = some_memcached_instance_or_whatever
-response = klass.foo
-puts response.body # makes the request
-
-second_response = klass.foo
-puts response.body # pulls from the cache without making a request
+first_request.handled_resposne # the value returned from the on_complete block
+second_request.handled_resposne # the value returned from the on_complete block (parsed JSON)
+</pre>
+The execution of that code goes something like this. The first and second requests are built and queued. When hydra is run the first and second requests run in parallel. When the first request completes, the third request is then built and queued up. The moment it is queued Hydra starts executing it. Meanwhile the second request would continue to run (or it could have completed before the first). Once the third request is done, hydra.run returns.
-# you can also pass timeouts on the define_remote_method or as a parameter
-# Note that timeouts are in milliseconds.
-Twitter.trends(:time_frame => :current, :timeout => 2000)
+*Memoization*
+Hydra memoizes requests within a single run call. You can also disable memoization.
+<pre>
+hydra = Typhoeus::Hydra.new
+2.times do
+ r = Typhoeus::Request.new("http://localhost/3000/users/1")
+ hydra.queue r
+end
+hydra.run # this will result in a single request being issued. However, the on_complete handlers of both will be called.
+hydra.disable_memoization
+2.times do
+ r = Typhoeus::Request.new("http://localhost/3000/users/1")
+ hydra.queue r
+end
+hydra.run # this will result in a two requests.
+</pre>
-# you also get the normal get, put, post, and delete methods
-class Remote
- include Typhoeus
+*Caching*
+Hydra includes built in support for creating cache getters and setters. In the following example, if there is a cache hit, the cached object is passed to the on_complete handler of the request object.
+<pre>
+hydra = Typhoeus::Hydra.new
+hydra.cache_setter do |request|
+ @cache.set(request.cache_key, request.response, request.cache_timeout)
end
-Remote.get("http://www.pauldix.net")
-Remote.put("http://", :body => "this is a request body")
-Remote.post("http://localhost:3001/posts.xml",
- {:params => {:post => {:author => "paul", :title => "a title", :body => "a body"}}})
-Remote.delete("http://localhost:3001/posts/1")
+hydra.cache_getter do |request|
+ @cache.get(request.cache_key) rescue nil
+end
+</pre>
-# you also have the ability to set request headers. So you can set your user agent and manually
-Remote.get("http://www.pauldix.net", :headers => {"User-Agent" => "typhoeus", "If-None-Match" => "some etag"})
+*Stubbing*
+Hydra allows you to stub out specific urls and patters to avoid hitting remote servers while testing.
+<pre>
+hydra = Typhoeus::Hydra.new
+response = Response.new(:code => 200, :headers => "", :body => "{'name' : 'paul'}", :time => 0.3)
+hydra.stub(:get, "http://localhost:3000/users/1").and_return(response)
-# and do things like basic HTTP authentication
-require 'base64'
-Remote.get("http://twitter.com/statuses/followers.json",
- :headers => {"Authorization" => "Basic #{Base64.b64encode("login:password")}"})
-
-# body and headers arguments also get passed through on defined remote methods.
-class TwitterRestAPI
- include Typhoeus
- remote_defaults :on_success => lambda {|response| JSON.parse(response.body)},
- :on_failure => lambda {|response| puts "error code: #{response.code}"},
- :base_uri => "http://twitter.com"
-
- define_remote_method :followers,
- :path => '/statuses/followers.json',
- :headers => {"Authorization" => "Basic #{Base64.b64encode("twitter_id:password")}"}
+request = Typhoeus::Request.new("http://localhost:3000/users/1")
+request.on_complete do |response|
+ JSON.parse(response.body)
end
+hydra.queue request
+hydra.run
+</pre>
+The queued request will hit the stub. The on_complete handler will be called and will be passed the response object. You can also specify a regex to match urls.
+<pre>
+hydra.stub(:get, /http\:\/\/localhost\:3000\/users\/.*/).and_return(response)
+# any requests for a user will be stubbed out with the pre built response.
+</pre>
-# The response object returned by get, put, post, and delete is passed to the on_success
-# or on_failure lambda block if declared.
-# The return value of the lambda block is then what is returned by the remote method invocation.
-# The response object can do the following:
-response.code # the http return code
-response.body # the body of the response
-response.headers # the response headers
-response.time # the response time in seconds
-
-# Typhoeus also has a nifty mocking framework built in
-# mock all calls to get
-Remote.mock(:get, :code => 200, :body => "whatever")
-
-# here we mock calls to get for the url
-Remote.mock(:get, :url => "http://pauldix.net", :code => 200, :body => "hi", :headers => "there", :time => 2)
-
-# note that url, code, body, headers, and time are all optional parameters to mock.
-# the first parameter can be either :get, :put, :post, or :delete
-
-# you can also provide headers and body that are expected on the call. An exception will be raised if they don't match
-Remote.mock(:get, :url => "http://pauldix.net", :expected_headers => {"If-None-Match" => "sldfkj234"})
-Remote.mock(:put, :url => "http://pauldix.net", :expected_body => "this is a body!")
-
-# using that mocking you could mock out the Twitter client like so:
-Twitter.mock(:get, :body => '{"hi": "there"}')
-# now any calls to trends, or search will get the mock and call the on_success handler. the response object will have that body.
-# we could also mock out a failure like so
-Twitter.mock(:get, :body => '{"fail": "oh noes!"}', :code => 500)
-# now calls to a remote method will result in the on_failure handler being called
+*The Singleton*
+All of the quick requests are done using the singleton hydra object. If you want to enable caching or stubbing on the quick requests, set those options on the singleton.
+<pre>
+hydra = Typhoeus::Hydra.hydra
+hydra.stub(:get, "http://localhost:3000/users")
</pre>
-The best place to see the functionality of what including Typhoeus in a class gives you is to look at the "remote_spec.rb"
+*Basic Authentication*
+<pre>
+require 'base64'
+response = Typhoeus::Request.get("http://twitter.com/statuses/followers.json",
+ :headers => {"Authorization" => "Basic #{Base64.b64encode("#{username}:#{password}")}"})
+</pre>
h2. Benchmarks
@@ -177,12 +176,8 @@ We can see from this that NET::HTTP performs as expected, taking 10 seconds to r
h2. Next Steps
-* Write up some more examples.
-* Create a SimpleDB client library using Typhoeus.
-* Create or get someone to create a CouchDB client library using Typhoeus.
+* Add in ability to keep-alive requests and reuse them within hydra.
* Add support for automatic retry, exponential back-off, and queuing for later.
-* Add in the support for custom get and set methods on the cache.
-* Add in support for integrated HTTP caching with Memcached.
h2. LICENSE

0 comments on commit 6cf2f56

Please sign in to comment.
Something went wrong with that request. Please try again.