Parallel requests

Andrew Radev edited this page Feb 17, 2014 · 5 revisions

Making parallel HTTP requests

Simple parallelism

When you initialize a new Faraday object, make sure to declare a proper adapter that supports parallel requests.

require 'typhoeus'
require 'typhoeus/adapters/faraday'

response1, response2 = nil

conn = Faraday.new(:url => "http://coolness.com") do |faraday|
  faraday.adapter :typhoeus
end

conn.in_parallel do
  response1 = conn.get('/one')
  response2 = conn.get('/two')

  # these will return nil here since the
  # requests haven't been completed
  response1.body
  response2.body
end

# at this point the response information you expected should be fully available to you.
response1.body # response1.status, etc
response2.body

All of the response data will be nil (resp.body, resp.status, etc.) until all of the requests in parallel are completed. Which means you'll want to wait until you are outside the block to handle any of the responses.

You'll also want to be aware of failed requests. Since they happen in parallel, you should check each response and make sure they went through successfully.

responses = []

conn.in_parallel do
  responses << conn.get('/one')
  responses << conn.get('/one')
end

responses.each { |r| p r.status }

Advanced use

Using the manager directly.

manager = Typhoeus::Hydra.new(:max_concurrency => 10) # (200 is default)
#manager.disable_memoization

conn.in_parallel(manager) do ...

Explicitly setting the default parallel manager

conn = Faraday.new :parallel_manager => manager

More Information

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.