Follow redirects returned from stubbed responses

[jarthod]

A request stubbed as 302 by webmock doesn't get followed using the em-http-request client despite the :redirect parameter.
My test works fine with real HTTP connection,

But if I try to stub the two requests, after the first is done (the 302) em-http-request calls the headers AND callback callbacks in a row with the same headers (the first response) and without doing the second request.

Here is a demonstrating example: https://gist.github.com/4294952

I'm using webmock 1.9.0 and em-http-request 1.0.3.

[jarthod]

Is this fixed ?

[bblimke]

ahh sorry, closed this one by mistake :)

[erwanlr]

Same with 301 and Typhoeus (302 not tested)

#!/usr/bin/env ruby

require 'rubygems'
require 'typhoeus'
require 'webmock'

include WebMock::API

url = 'https://updown.io:443/'
redirection = 'https://beta.updown.io:443/'

def call_url(url)
  response = Typhoeus.get(url, :followlocation => true)

  puts "#{url} : #{response.code}"
  puts "headers_hash : #{response.headers_hash}"
end

puts "Typhoues #{Typhoeus::VERSION}"
puts "webmock #{WebMock::VERSION}"
puts

puts "Real:"
WebMock.allow_net_connect!
call_url(url)

puts "Webmock stub:"
WebMock.disable_net_connect!
stub_request(:get, url).to_return(:status => 301, :headers => { 'Location' => redirection })
stub_request(:get, redirection)
call_url(url)

[erwanlr]

I was able to follow the redirection with the following hack/override:

module WebMock
  class StubRegistry
    def evaluate_response_for_request(response, request_signature)
      if [301, 302].include?(response.status[0])
        request_signature.uri = response.headers['Location']
        request_signature.method = :get

        ::WebMock::RequestRegistry.instance.requested_signatures.put(request_signature)
        response_for_request(request_signature)
      else
        response.dup.evaluate(request_signature)
      end
    end
  end
end

However, it raises an error about the request being unregistered even if it appears in the registered request stubs:

Unregistered request: GET https://beta.updown.io/ with headers {'User-Agent'=>'Typhoeus - https://github.com/typhoeus/typhoeus'} (WebMock::NetConnectNotAllowedError)

You can stub this request with the following snippet:

stub_request(:get, "https://beta.updown.io/").
  with(:headers => {'User-Agent'=>'Typhoeus - https://github.com/typhoeus/typhoeus'}).
  to_return(:status => 200, :body => "", :headers => {})

registered request stubs:

stub_request(:get, "https://beta.updown.io/")
stub_request(:get, "https://updown.io/")

Full test file is there: https://gist.github.com/erwanlr/9892510

[bblimke]

@erwanlr WebMock internally stores uri's as Addressable::URI.
Change line 5 to request_signature.uri = Addressable::URI.parse(response.headers['Location']) and it should work fine.

[erwanlr]

Humm, I thought I had tested this xD, it's working now.

However, Typhoeus (and maybe other Adapters) must have a specific response headers for the Typhoeus::Response#redirections to work properly

e.g, with Typhoeus, all redirected response headers must be in the final headers:

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.fr/?gfe_rd=cr&ei=KXM6U4H0JZG6jweshYG4Aw
Content-Length: 258
Date: Tue, 01 Apr 2014 08:04:57 GMT
Server: GFE/2.0
Alternate-Protocol: 80:quic

HTTP/1.1 200 OK
Date: Tue, 01 Apr 2014 08:04:58 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=76027976e032d6a9:FF=0:TM=1396339498:LM=1396339498:S=xLA12HVJwGwLlKzo; expires=Thu, 31-Mar-2016 08:04:58 GMT; path=/; domain=.google.fr
Set-Cookie: NID=67=RBXpvaCfM4yplRMpoQ6ypqzoQ8d5L7F3-I1LcOdenMrOe13HerFQ_MeEUFinIR6jOHB8hT44gW5VtLvBq1zZF5WWgEnPyjJTAJ1DSBNMYJzb-k3TSdpfXnKvq9dXmfwS; expires=Wed, 01-Oct-2014 08:04:58 GMT; path=/; domain=.google.fr; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Alternate-Protocol: 80:quic
Transfer-Encoding: chunked

But I have no idea how to do this in WebMock :/

[erwanlr]

Tried to implement this for Typhoeus but it wasn't successful as the WebMock::Response#headers are a hash and therefore we can't add the headers from the redirect's responses (as duplicate keys will be merged)

Could @i0rek or one of the Typhoeus' dev have a look at this please ?

[hanshasselberg]

I played around with it. I found a hacky solution to sneak in the response_headers which is a string. The tests are green with these changes. Not sure it helps.

diff --git a/lib/webmock/http_lib_adapters/typhoeus_hydra_adapter.rb b/lib/webmock/http_lib_adapters/typhoeus_hydra_adapter.rb
index bc36271..f4712c6 100644
--- a/lib/webmock/http_lib_adapters/typhoeus_hydra_adapter.rb
+++ b/lib/webmock/http_lib_adapters/typhoeus_hydra_adapter.rb
@@ -79,7 +79,11 @@ if defined?(Typhoeus)
           webmock_response = WebMock::Response.new
           webmock_response.status = [typhoeus_response.code, typhoeus_response.status_message]
           webmock_response.body = typhoeus_response.body
-          webmock_response.headers = typhoeus_response.headers
+          if typhoeus_response.response_headers && @disabled == true
+            webmock_response.headers = {"Raw-Headers-X-Typhoeus-012834701893274023" => typhoeus_response.response_headers.dup}
+          else
+            webmock_response.headers = typhoeus_response.headers
+          end
           webmock_response
         end

@@ -96,9 +100,14 @@ if defined?(Typhoeus)
             ::Typhoeus::Response.new(
               :code         => webmock_response.status[0],
               :status_message => webmock_response.status[1],
-              :body         => webmock_response.body,
-              :headers => webmock_response.headers
-            )
+              :body         => webmock_response.body
+            ).tap do |response|
+              if webmock_response.headers && response_headers = webmock_response.headers.delete("Raw-Headers-X-Typhoeus-012834701893274023")
+                response.instance_variable_set(:response_headers=, response_headers)
+              else
+                response.options[:headers] = webmock_response.headers
+              end
+            end
           end
           response.mock = :webmock
           response

I don't think I understand the problem yet.

[erwanlr]

The problem is that I am unable to find a clean solution for the implementation of the typhoeus' redirects in WebMock :/

[hanshasselberg]

In your example you have

stub_request(:get, "https://beta.updown.io/")
stub_request(:get, "https://updown.io/")

But Typhoeus only does one request; libcurl takes care of the redirection.

[erwanlr]

Not sure what you meant there :p

This is the spec file I currently use:

require 'typhoeus'
require 'webmock'
require 'webmock/rspec'

describe 'Typhoeus WebMock Redirects' do
  let(:url)         { 'http://gogle.com' }
  let(:redirection) { 'http://www.malicious.net' }
  # Typhoeus request params
  let(:params)      { {} }

  before do
    stub_request(:get, url).to_return(status: 301, headers: { location: redirection })
    stub_request(:get, redirection).to_return(body: 'Owned')

    @response = Typhoeus.get(url, params)
  end

  context 'when no --followlocation' do
    it 'does not process the redirection' do
      expect(@response.redirections).to be_empty
      expect(@response.code).to eq 301
    end
  end

  context 'when --followlocation' do
    let(:params) { { followlocation: true } }

    it 'follows the redirection' do
      expect(@response.redirections).to_not be_empty
      expect(@response.code).to eq 200
      expect(@response.body).to rq 'Owned'
    end
  end
end

[bblimke]

@i0rek I see. If the redirects are handled internally by libcurl then there is now way for WebMock to intercept them.
The only solution I could think of is WebMock intercepting request, removing followlocation flag, and then handling redirects on it's own. I'm not sure it's a good idea for WebMock to modify Typhoeus as far.

[hanshasselberg]

I don't think it is that hard @bblimke.
In your example @erwanlr you stub two responses, you should only stub one since Typhoeus only makes one request (libcurl hides everything else). The spec should be green like that:

require 'typhoeus'
require 'webmock'
require 'webmock/rspec'

describe 'Typhoeus WebMock Redirects' do
  let(:url)         { 'http://gogle.com' }
  let(:redirection) { 'http://www.malicious.net' }
  # Typhoeus request params
  let(:params)      { {} }

  context 'when no --followlocation' do
    before do
      stub_request(:get, url).to_return(status: 301, headers: { location: redirection })
      @response = Typhoeus.get(url, params)
    end
    it 'does not process the redirection' do
      expect(@response.redirections).to be_empty
      expect(@response.code).to eq 301
    end
  end

  context 'when --followlocation' do    before do
      stub_request(:get, url).to_return(body: 'Owned')
      @response = Typhoeus.get(url, params)
    end
    let(:params) { { followlocation: true } }

    it 'follows the redirection' do
      # expect(@response.redirections).to_not be_empty
      expect(@response.code).to eq 200
      expect(@response.body).to rq 'Owned'
    end
  end
end

Expect for # expect(@response.redirections).to_not be_empty which would fail.
The problem is how to sneak in the correct headers which make Typhoeus believe there was a redirection, right?!