Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

using proxy server instead of normal HTTP server #1212

Open
wants to merge 1 commit into from

4 participants

@mreinsch

I'd like to start a discussion with this pull request. It's still a bit rough, please bear with me.

This introduces a proxy server implementation (based on WEBrick) and configures the browser to use that proxy server. Thus the proxy server now receives any requests made by the browser. The proxy server internally uses the rack test framework to invoke the app to test. This allows us to easily test apps which use multiple domains.

Right now the proxy server forwards all requests to the web app (similar to the RackTest adapter), but it'd be relatively easy to extend this to actually forward some requests to remote servers or use net mocking frameworks to mock requests.

Besides the selenium proxy driver included in the pull request, I also created one for Poltergeist:

class Capybara::Poltergeist::ProxyDriver < Capybara::Poltergeist::Driver
  def initialize(app, opts={})
    phantomjs_options = opts.delete(:phantomjs_options) || []
    phantomjs_options.push(
      "--proxy=http://#{Capybara.server_host}:#{Capybara.server_port}",
      "--proxy-type=http",
      "--ignore-ssl-errors=true")
    super(app, opts.merge(phantomjs_options: phantomjs_options))
  end
  def needs_server?
    false
  end
  def needs_proxy?
    true
  end
end

Register those drivers like this:

Capybara.register_driver :poltergeist_proxy do |app|
  Capybara::Poltergeist::ProxyDriver.new(app)
end

Capybara.register_driver :selenium_proxy do |app|
  Capybara::Selenium::ProxyDriver.new(app)
end

One limitation, we need to configure the proxy driver's host and port to make sure they are available, for instance:

Capybara.configure do |config|
  config.javascript_driver = :selenium_proxy
  config.server_host = "127.0.0.1"
  config.server_port = 23798
end

I'd love to hear your comments on how we can get this into capybara, we need this mainly to test our multi-domain app. Thanks.

@abotalov
Collaborator

Have you looked at https://github.com/jarib/browsermob-proxy-rb?

Also I haven't experienced any issues with testing multi-domain apps when I used Capybara + Selenium.

@jnicklas
Owner

As an idea, I really like this. It would be very nice to have a simple way to mock out specific URLs via the proxy, it seems that this would be a fluent candidate for creating something like that, or even making it possible to use existing mocking frameworks like WebMock in the client. Interesting stuff, the idea of using a proxy is neat.

Implementation wise, I think this needs some work though, but I realize you just wanted to throw this out there. I'm not too happy about all the SSL stuff in there, and I also don't like that it comes with a preconfigured list of hosts it ignores. There's also a lot of copy-paste with Capybara::Server going on. I'm guessing you know all of that though, so let's work on making this into something we can merge.

I'd also liket to discuss the needs_proxy? method. Somehow this seems a bit strange to me. Some drivers will be capable of providing a proxy setting and some won't (terminus comes to mind). It seems more in line that some drivers are capable of using a proxy and some aren't, so maybe we should somehow have an option to enable the proxy, and a way for Capybara to instruct the driver to use a certain proxy server. What do you think?

@mreinsch

Thanks for the feedback and sorry for the long delay!

I finally got some time to revise the pull request based on your feedback. The drivers now have a supports_proxy_protocol? method, which together with Capybara.use_proxy_protocol determines whether to use http or proxy. If proxy protocol is chosen, we'll tell the driver which host/port we're using, so it can start the browser with the correct settings.

So to use it, you only need the following

Capybara.configure do |config|
  config.use_proxy_protocol = true
  config.javascript_driver = :selenium
end

For poltergeist, the following patch works fine for me:

class Capybara::Poltergeist::Driver
  def supports_proxy_protocol?
    true
  end

  def setup_proxy_host(host, port)
    opts = (options[:phantomjs_options] ||= [])
    opts.delete_if {|o| o =~ /^--proxy/ }
    opts.push("--proxy=http://#{host}:#{port}", "--proxy-type=http")
    opts.push("--ignore-ssl-errors=true") unless opts.include?("--ignore-ssl-errors=true")
  end
end

I still need to make the list of ignored hosts configurable - idea there is to allow everyone to hook in their own code, so we can hopefully see stubs for some of the common sites people integrate with...

Anyway, please let me know if that is moving into a direction you'd be happy with. Thanks!

@mreinsch

Regarding the SSL stuff, I'd prefer to not have that in there either, but it's required to test any SSL sites. The browsers use the CONNECT mechanism to tunnel the raw SSL stream through the proxy...

lib/capybara/selenium/driver.rb
@@ -75,6 +75,13 @@ def find_css(selector)
def wait?; true; end
def needs_server?; true; end
+ def supports_proxy_protocol?; true; end
+
+ def setup_proxy_host(host, port)
+ proxy_server = "#{host}:#{port}"
+ options[:profile] ||= Selenium::WebDriver::Firefox::Profile.new
@twalpole Collaborator

selenium supports more than just Firefox - how will this work if the user is using selenium with chrome, ie, etc?

good question, I suppose all browsers will somehow support setting up a proxy. If they don't, we can have the supports_proxy_protocol return false for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mreinsch

I've completed the configuration, so it's now possible to plug in different rack apps for different hosts to mock out third party services - or to deliver 404 errors in case you just don't want to bother (like for google analytics).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 21, 2014
  1. @mreinsch
This page is out of date. Refresh to see the latest.
View
35 lib/capybara.rb
@@ -22,7 +22,7 @@ class << self
attr_accessor :asset_host, :app_host, :run_server, :default_host, :always_include_port
attr_accessor :server_port, :exact, :match, :exact_options, :visible_text_only
attr_accessor :default_selector, :default_wait_time, :ignore_hidden_elements
- attr_accessor :save_and_open_page_path, :automatic_reload, :raise_server_errors
+ attr_accessor :save_and_open_page_path, :automatic_reload, :raise_server_errors, :use_proxy_protocol
attr_writer :default_driver, :current_driver, :javascript_driver, :session_name, :server_host
attr_accessor :app
@@ -46,6 +46,7 @@ class << self
# [ignore_hidden_elements = Boolean] Whether to ignore hidden elements on the page (Default: true)
# [automatic_reload = Boolean] Whether to automatically reload elements as Capybara is waiting (Default: true)
# [save_and_open_page_path = String] Where to put pages saved through save_and_open_page (Default: Dir.pwd)
+ # [use_proxy_protocol = Boolean] Use the proxy protocol to send requests to capybara's proxy server instead of normal web server. Using the proxy server stubs all requests and allows you to use subdomains (Default: false)
#
# === DSL Options
#
@@ -134,6 +135,23 @@ def server(&block)
##
#
+ # Register a proc that Capybara will use to start a proxy server which will receive any requests from the browser.
+ # The proxy method is used in case the browser supports proxy servers.
+ #
+ # By default, Capybara will setup it's internal proxy.
+ #
+ # @yield [app, port] This block receives a rack app and port and should run a Rack handler
+ #
+ def proxy_server(&block)
+ if block_given?
+ @proxy_server = block
+ else
+ @proxy_server
+ end
+ end
+
+ ##
+ #
# Wraps the given string, which should contain an HTML document or fragment
# in a {Capybara::Node::Simple} which exposes all {Capybara::Node::Matchers},
# {Capybara::Node::Finders} and {Capybara::Node::DocumentMatchers}. This allows you to query
@@ -175,6 +193,19 @@ def run_default_server(app, port)
##
#
+ # Runs Capybara's default proxy server for the given application and port
+ # under most circumstances you should not have to call this method
+ # manually.
+ #
+ # @param [Rack Application] app The rack application to run
+ # @param [Fixnum] port The port to run the application on
+ #
+ def run_default_proxy_server(app, port)
+ Capybara::ProxyServer.run(app, :Host => server_host, :Port => port, :AccessLog => [], :Logger => WEBrick::Log::new(nil, 0))
+ end
+
+ ##
+ #
# @return [Symbol] The name of the driver to use by default
#
def default_driver
@@ -322,6 +353,7 @@ module Selenium; end
require 'capybara/dsl'
require 'capybara/window'
require 'capybara/server'
+ require 'capybara/proxy_server'
require 'capybara/selector'
require 'capybara/result'
require 'capybara/version'
@@ -357,6 +389,7 @@ module Selenium; end
config.always_include_port = false
config.run_server = true
config.server {|app, port| Capybara.run_default_server(app, port)}
+ config.proxy_server {|app, port| Capybara.run_default_proxy_server(app, port)}
config.default_selector = :css
config.default_wait_time = 2
config.ignore_hidden_elements = true
View
4 lib/capybara/driver/base.rb
@@ -133,4 +133,8 @@ def reset!
def needs_server?
false
end
+
+ def supports_proxy_protocol?
+ false
+ end
end
View
114 lib/capybara/proxy_server.rb
@@ -0,0 +1,114 @@
+require 'webrick'
+require 'webrick/httpproxy'
+require 'webrick/https'
+require 'rack/test'
+
+module Capybara
+ class ProxyServer < WEBrick::HTTPProxyServer
+
+ class SSLHandler < WEBrick::HTTPServer
+ def initialize(app, config)
+ super(config.merge(:Port => 443, :DoNotListen => true))
+ @rack_handler = Rack::Handler::WEBrick.new(self, app)
+ end
+
+ def service(req, res)
+ if (host_handler = Capybara::ProxyServer.host_mapping[req.host])
+ Rack::Handler::WEBrick.new(self, host_handler).service(req, res)
+ else
+ @rack_handler.service(req, res)
+ end
+ rescue => err
+ $stderr.puts err
+ $stderr.puts err.backtrace.join("\n")
+ end
+ end
+
+ DEFAULT_404_HANDLER = Proc.new {|env| [404, {}, []] }
+
+ class << self
+ attr_accessor :host_mapping
+
+ ##
+ #
+ # Configure Capybara ProxyServer to suit your needs.
+ #
+ # Capybara::ProxyServer.configure do |config|
+ # config.route "i.kissmetrics.com" => Capybara::ProxyServer::DEFAULT_404_HANDLER
+ # end
+ #
+ def configure
+ yield self
+ end
+
+ ##
+ #
+ # Route a request to a specific host to a separate Rack application.
+ # This allows you to mock external services without actually doing external requests.
+ #
+ # You can use the Capybara::ProxyServer::DEFAULT_404_HANDLER to simply return a 404 error code, or create your own rack applications.
+ #
+ def route(host_map)
+ @host_mapping ||= {}
+ @host_mapping.merge!(host_map)
+ end
+
+ def run(app, config)
+ server = new(app, config.merge(:OutputBufferSize => 5))
+ server.start
+ server
+ end
+ end
+
+ def initialize(app, config)
+ super(config, WEBrick::Config::HTTP)
+ @rack_handler = Rack::Handler::WEBrick.new(self, app)
+ @ssl_context = generate_ssl_context
+ @ssl_server = SSLHandler.new(app, config)
+ end
+
+ def service(req, res)
+ if req.request_method == "CONNECT"
+ host, port = req.unparsed_uri.split(":", 2)
+ if port == '443'
+ ua = Thread.current[:WEBrickSocket]
+ res.status = WEBrick::HTTPStatus::RC_OK
+ res.send_response(ua)
+ req.parse(NullReader) rescue nil
+ ssl = OpenSSL::SSL::SSLSocket.new(ua, @ssl_context)
+ ssl.sync_close = true
+ ssl.accept
+ @ssl_server.run(ssl)
+ else
+ res.status = 501
+ end
+ elsif (host_handler = Capybara::ProxyServer.host_mapping[req.host])
+ Rack::Handler::WEBrick.new(self, host_handler).service(req, res)
+ else
+ @rack_handler.service(req, res)
+ end
+ rescue => err
+ $stderr.puts err
+ $stderr.puts err.backtrace.join("\n")
+ end
+
+ private
+
+ def generate_ssl_context
+ OpenSSL::SSL::SSLContext.new('SSLv23_server').tap do |ssl_context|
+ self_signed_cert, self_signed_cert_key =
+ WEBrick::Utils.create_self_signed_cert(512, [ [ "CN", "localhost" ] ], "Generated by Ruby/OpenSSL")
+ ssl_context.cert = self_signed_cert
+ ssl_context.key = self_signed_cert_key
+ end
+ end
+ end
+end
+
+Capybara::ProxyServer.configure do |config|
+ config.route "www.google-analytics.com" => Capybara::ProxyServer::DEFAULT_404_HANDLER,
+ "ssl.google-analytics.com" => Capybara::ProxyServer::DEFAULT_404_HANDLER,
+ "fonts.googleapis.com" => Capybara::ProxyServer::DEFAULT_404_HANDLER,
+ "connect.facebook.net" => Capybara::ProxyServer::DEFAULT_404_HANDLER,
+ "platform.twitter.com" => Capybara::ProxyServer::DEFAULT_404_HANDLER
+end
View
8 lib/capybara/selenium/driver.rb
@@ -33,7 +33,6 @@ def initialize(app, options={})
raise e
end
end
-
@app = app
@browser = nil
@exit_status = nil
@@ -75,6 +74,13 @@ def find_css(selector)
def wait?; true; end
def needs_server?; true; end
+ def supports_proxy_protocol?; options[:browser] == :firefox; end
+
+ def setup_proxy_host(host, port)
+ proxy_server = "#{host}:#{port}"
+ options[:profile] ||= Selenium::WebDriver::Firefox::Profile.new
+ options[:profile].proxy = Selenium::WebDriver::Proxy.new(:http => proxy_server, :ssl => proxy_server)
+ end
def execute_script(script)
browser.execute_script script
View
11 lib/capybara/server.rb
@@ -33,8 +33,9 @@ def ports
attr_reader :app, :port, :host
- def initialize(app, port=Capybara.server_port, host=Capybara.server_host)
+ def initialize(app, driver, port=Capybara.server_port, host=Capybara.server_host)
@app = app
+ @driver = driver
@middleware = Middleware.new(@app)
@server_thread = nil # suppress warnings
@host, @port = host, port
@@ -65,9 +66,15 @@ def responsive?
def boot
unless responsive?
Capybara::Server.ports[@app.object_id] = @port
+ if Capybara.use_proxy_protocol && @driver.supports_proxy_protocol?
+ server_proc = Capybara.proxy_server
+ @driver.setup_proxy_host(@host, @port)
+ else
+ server_proc = Capybara.server
+ end
@server_thread = Thread.new do
- Capybara.server.call(@middleware, @port)
+ server_proc.call(@middleware, @port)
end
Timeout.timeout(60) { @server_thread.join(0.1) until responsive? }
View
2  lib/capybara/session.rb
@@ -63,7 +63,7 @@ def initialize(mode, app=nil)
@mode = mode
@app = app
if Capybara.run_server and @app and driver.needs_server?
- @server = Capybara::Server.new(@app).boot
+ @server = Capybara::Server.new(@app, driver).boot
else
@server = nil
end
View
2  lib/capybara/spec/session/current_url_spec.rb
@@ -1,6 +1,6 @@
Capybara::SpecHelper.spec '#current_url, #current_path, #current_host' do
before :all do
- @servers = 2.times.map { Capybara::Server.new(TestApp.clone).boot }
+ @servers = 2.times.map { Capybara::Server.new(TestApp.clone, nil).boot }
# sanity check
expect(@servers[0].port).not_to eq(@servers[1].port)
expect(@servers.map { |s| s.port }).not_to include 80
View
28 spec/server_spec.rb
@@ -4,7 +4,7 @@
it "should spool up a rack server" do
@app = proc { |env| [200, {}, ["Hello Server!"]]}
- @server = Capybara::Server.new(@app).boot
+ @server = Capybara::Server.new(@app, nil).boot
@res = Net::HTTP.start(@server.host, @server.port) { |http| http.get('/') }
@@ -13,7 +13,7 @@
it "should do nothing when no server given" do
expect do
- @server = Capybara::Server.new(nil).boot
+ @server = Capybara::Server.new(nil, nil).boot
end.not_to raise_error
end
@@ -22,12 +22,12 @@
app = proc { |env| [200, {}, ['Hello Server!']] }
Capybara.server_host = '127.0.0.1'
- server = Capybara::Server.new(app).boot
+ server = Capybara::Server.new(app, nil).boot
res = Net::HTTP.get(URI("http://127.0.0.1:#{server.port}"))
expect(res).to eq('Hello Server!')
Capybara.server_host = '0.0.0.0'
- server = Capybara::Server.new(app).boot
+ server = Capybara::Server.new(app, nil).boot
res = Net::HTTP.get(URI("http://127.0.0.1:#{server.port}"))
expect(res).to eq('Hello Server!')
ensure
@@ -39,7 +39,7 @@
Capybara.server_port = 22789
@app = proc { |env| [200, {}, ["Hello Server!"]]}
- @server = Capybara::Server.new(@app).boot
+ @server = Capybara::Server.new(@app, nil).boot
@res = Net::HTTP.start(@server.host, 22789) { |http| http.get('/') }
expect(@res.body).to include('Hello Server')
@@ -49,7 +49,7 @@
it "should use given port" do
@app = proc { |env| [200, {}, ["Hello Server!"]]}
- @server = Capybara::Server.new(@app, 22790).boot
+ @server = Capybara::Server.new(@app, nil, 22790).boot
@res = Net::HTTP.start(@server.host, 22790) { |http| http.get('/') }
expect(@res.body).to include('Hello Server')
@@ -61,8 +61,8 @@
@app1 = proc { |env| [200, {}, ["Hello Server!"]]}
@app2 = proc { |env| [200, {}, ["Hello Second Server!"]]}
- @server1 = Capybara::Server.new(@app1).boot
- @server2 = Capybara::Server.new(@app2).boot
+ @server1 = Capybara::Server.new(@app1, nil).boot
+ @server2 = Capybara::Server.new(@app2, nil).boot
@res1 = Net::HTTP.start(@server1.host, @server1.port) { |http| http.get('/') }
expect(@res1.body).to include('Hello Server')
@@ -75,10 +75,10 @@
@app1 = proc { |env| [200, {}, ["Hello Server!"]]}
@app2 = proc { |env| [200, {}, ["Hello Second Server!"]]}
- @server1a = Capybara::Server.new(@app1).boot
- @server1b = Capybara::Server.new(@app1).boot
- @server2a = Capybara::Server.new(@app2).boot
- @server2b = Capybara::Server.new(@app2).boot
+ @server1a = Capybara::Server.new(@app1, nil).boot
+ @server1b = Capybara::Server.new(@app1, nil).boot
+ @server2a = Capybara::Server.new(@app2, nil).boot
+ @server2b = Capybara::Server.new(@app2, nil).boot
@res1 = Net::HTTP.start(@server1b.host, @server1b.port) { |http| http.get('/') }
expect(@res1.body).to include('Hello Server')
@@ -98,7 +98,7 @@
end
expect do
- Capybara::Server.new(proc {|e|}).boot
+ Capybara::Server.new(proc {|e|}, nil).boot
end.to raise_error(RuntimeError, 'kaboom')
ensure
# TODO refactor out the defaults so it's reliant on unset state instead of
@@ -109,7 +109,7 @@
it "is not #responsive? when Net::HTTP raises a SystemCallError" do
app = lambda { [200, {}, ['Hello, world']] }
- server = Capybara::Server.new(app)
+ server = Capybara::Server.new(app, nil)
expect(Net::HTTP).to receive(:start).and_raise(SystemCallError.allocate)
expect(server.responsive?).to eq false
end
Something went wrong with that request. Please try again.