Skip to content

Parallel requests with Celluloid::IO

Tony Arcieri edited this page · 6 revisions
Clone this wiki locally

NOTE: Celluloid::IO support was removed in version 1.0 of this gem. We'd like to bring it back, but for now it's gone!

Want to make multiple requests in parallel like you can with the Typhoeus gem? Want to do it from a single thread? And do you want to avoid callback hell?

You can use Celluloid::IO, an evented programming library for Ruby, to accomplish this by leveraging the HTTP Gem's dependency injection APIs.

Here is how to write a single-threaded parallel HTTP fetcher using the HTTP Gem:

Taken from examples/parallel_requests_with_celluloid.rb

require 'celluloid/io'
require 'http'

class HttpFetcher
  include Celluloid::IO

  def fetch(url)
    # Note: For SSL support specify:
    # ssl_socket_class: Celluloid::IO::SSLSocket
    HTTP.get(url, socket_class: Celluloid::IO::TCPSocket)
  end
end

fetcher = HttpFetcher.new

urls = %w(http://www.ruby-lang.org/ http://www.rubygems.org/ http://celluloid.io/)

# Kick off a bunch of future calls to HttpFetcher to grab the URLs in parallel
futures = urls.map { |u| [u, fetcher.future.fetch(u)] }

# Consume the results as they come in
futures.each do |url, future|
  # Wait for HttpFetcher#fetch to complete for this request
  response = future.value
  puts "*** Got #{url}: #{response.inspect}\n\n"
end
Something went wrong with that request. Please try again.