using proxy server instead of normal HTTP server #1212

Open
wants to merge 23 commits into
from

Conversation

Projects
None yet
9 participants
@mreinsch

I'd like to start a discussion with this pull request. It's still a bit rough, please bear with me.

This introduces a proxy server implementation (based on WEBrick) and configures the browser to use that proxy server. Thus the proxy server now receives any requests made by the browser. The proxy server internally uses the rack test framework to invoke the app to test. This allows us to easily test apps which use multiple domains.

Right now the proxy server forwards all requests to the web app (similar to the RackTest adapter), but it'd be relatively easy to extend this to actually forward some requests to remote servers or use net mocking frameworks to mock requests.

Besides the selenium proxy driver included in the pull request, I also created one for Poltergeist:

class Capybara::Poltergeist::ProxyDriver < Capybara::Poltergeist::Driver
  def initialize(app, opts={})
    phantomjs_options = opts.delete(:phantomjs_options) || []
    phantomjs_options.push(
      "--proxy=http://#{Capybara.server_host}:#{Capybara.server_port}",
      "--proxy-type=http",
      "--ignore-ssl-errors=true")
    super(app, opts.merge(phantomjs_options: phantomjs_options))
  end
  def needs_server?
    false
  end
  def needs_proxy?
    true
  end
end

Register those drivers like this:

Capybara.register_driver :poltergeist_proxy do |app|
  Capybara::Poltergeist::ProxyDriver.new(app)
end

Capybara.register_driver :selenium_proxy do |app|
  Capybara::Selenium::ProxyDriver.new(app)
end

One limitation, we need to configure the proxy driver's host and port to make sure they are available, for instance:

Capybara.configure do |config|
  config.javascript_driver = :selenium_proxy
  config.server_host = "127.0.0.1"
  config.server_port = 23798
end

I'd love to hear your comments on how we can get this into capybara, we need this mainly to test our multi-domain app. Thanks.

@abotalov

This comment has been minimized.

Show comment Hide comment
@abotalov

abotalov Dec 15, 2013

Collaborator

Have you looked at https://github.com/jarib/browsermob-proxy-rb?

Also I haven't experienced any issues with testing multi-domain apps when I used Capybara + Selenium.

Collaborator

abotalov commented Dec 15, 2013

Have you looked at https://github.com/jarib/browsermob-proxy-rb?

Also I haven't experienced any issues with testing multi-domain apps when I used Capybara + Selenium.

@jnicklas

This comment has been minimized.

Show comment Hide comment
@jnicklas

jnicklas Dec 16, 2013

Collaborator

As an idea, I really like this. It would be very nice to have a simple way to mock out specific URLs via the proxy, it seems that this would be a fluent candidate for creating something like that, or even making it possible to use existing mocking frameworks like WebMock in the client. Interesting stuff, the idea of using a proxy is neat.

Implementation wise, I think this needs some work though, but I realize you just wanted to throw this out there. I'm not too happy about all the SSL stuff in there, and I also don't like that it comes with a preconfigured list of hosts it ignores. There's also a lot of copy-paste with Capybara::Server going on. I'm guessing you know all of that though, so let's work on making this into something we can merge.

I'd also liket to discuss the needs_proxy? method. Somehow this seems a bit strange to me. Some drivers will be capable of providing a proxy setting and some won't (terminus comes to mind). It seems more in line that some drivers are capable of using a proxy and some aren't, so maybe we should somehow have an option to enable the proxy, and a way for Capybara to instruct the driver to use a certain proxy server. What do you think?

Collaborator

jnicklas commented Dec 16, 2013

As an idea, I really like this. It would be very nice to have a simple way to mock out specific URLs via the proxy, it seems that this would be a fluent candidate for creating something like that, or even making it possible to use existing mocking frameworks like WebMock in the client. Interesting stuff, the idea of using a proxy is neat.

Implementation wise, I think this needs some work though, but I realize you just wanted to throw this out there. I'm not too happy about all the SSL stuff in there, and I also don't like that it comes with a preconfigured list of hosts it ignores. There's also a lot of copy-paste with Capybara::Server going on. I'm guessing you know all of that though, so let's work on making this into something we can merge.

I'd also liket to discuss the needs_proxy? method. Somehow this seems a bit strange to me. Some drivers will be capable of providing a proxy setting and some won't (terminus comes to mind). It seems more in line that some drivers are capable of using a proxy and some aren't, so maybe we should somehow have an option to enable the proxy, and a way for Capybara to instruct the driver to use a certain proxy server. What do you think?

@mreinsch

This comment has been minimized.

Show comment Hide comment
@mreinsch

mreinsch Jul 19, 2014

Thanks for the feedback and sorry for the long delay!

I finally got some time to revise the pull request based on your feedback. The drivers now have a supports_proxy_protocol? method, which together with Capybara.use_proxy_protocol determines whether to use http or proxy. If proxy protocol is chosen, we'll tell the driver which host/port we're using, so it can start the browser with the correct settings.

So to use it, you only need the following

Capybara.configure do |config|
  config.use_proxy_protocol = true
  config.javascript_driver = :selenium
end

For poltergeist, the following patch works fine for me:

class Capybara::Poltergeist::Driver
  def supports_proxy_protocol?
    true
  end

  def setup_proxy_host(host, port)
    opts = (options[:phantomjs_options] ||= [])
    opts.delete_if {|o| o =~ /^--proxy/ }
    opts.push("--proxy=http://#{host}:#{port}", "--proxy-type=http")
    opts.push("--ignore-ssl-errors=true") unless opts.include?("--ignore-ssl-errors=true")
  end
end

I still need to make the list of ignored hosts configurable - idea there is to allow everyone to hook in their own code, so we can hopefully see stubs for some of the common sites people integrate with...

Anyway, please let me know if that is moving into a direction you'd be happy with. Thanks!

Thanks for the feedback and sorry for the long delay!

I finally got some time to revise the pull request based on your feedback. The drivers now have a supports_proxy_protocol? method, which together with Capybara.use_proxy_protocol determines whether to use http or proxy. If proxy protocol is chosen, we'll tell the driver which host/port we're using, so it can start the browser with the correct settings.

So to use it, you only need the following

Capybara.configure do |config|
  config.use_proxy_protocol = true
  config.javascript_driver = :selenium
end

For poltergeist, the following patch works fine for me:

class Capybara::Poltergeist::Driver
  def supports_proxy_protocol?
    true
  end

  def setup_proxy_host(host, port)
    opts = (options[:phantomjs_options] ||= [])
    opts.delete_if {|o| o =~ /^--proxy/ }
    opts.push("--proxy=http://#{host}:#{port}", "--proxy-type=http")
    opts.push("--ignore-ssl-errors=true") unless opts.include?("--ignore-ssl-errors=true")
  end
end

I still need to make the list of ignored hosts configurable - idea there is to allow everyone to hook in their own code, so we can hopefully see stubs for some of the common sites people integrate with...

Anyway, please let me know if that is moving into a direction you'd be happy with. Thanks!

@mreinsch

This comment has been minimized.

Show comment Hide comment
@mreinsch

mreinsch Jul 19, 2014

Regarding the SSL stuff, I'd prefer to not have that in there either, but it's required to test any SSL sites. The browsers use the CONNECT mechanism to tunnel the raw SSL stream through the proxy...

Regarding the SSL stuff, I'd prefer to not have that in there either, but it's required to test any SSL sites. The browsers use the CONNECT mechanism to tunnel the raw SSL stream through the proxy...

lib/capybara/selenium/driver.rb
+
+ def setup_proxy_host(host, port)
+ proxy_server = "#{host}:#{port}"
+ options[:profile] ||= Selenium::WebDriver::Firefox::Profile.new

This comment has been minimized.

Show comment Hide comment
@twalpole

twalpole Jul 19, 2014

Collaborator

selenium supports more than just Firefox - how will this work if the user is using selenium with chrome, ie, etc?

@twalpole

twalpole Jul 19, 2014

Collaborator

selenium supports more than just Firefox - how will this work if the user is using selenium with chrome, ie, etc?

This comment has been minimized.

Show comment Hide comment
@mreinsch

mreinsch Jul 20, 2014

good question, I suppose all browsers will somehow support setting up a proxy. If they don't, we can have the supports_proxy_protocol return false for now.

@mreinsch

mreinsch Jul 20, 2014

good question, I suppose all browsers will somehow support setting up a proxy. If they don't, we can have the supports_proxy_protocol return false for now.

@mreinsch

This comment has been minimized.

Show comment Hide comment
@mreinsch

mreinsch Aug 3, 2014

I've completed the configuration, so it's now possible to plug in different rack apps for different hosts to mock out third party services - or to deliver 404 errors in case you just don't want to bother (like for google analytics).

mreinsch commented Aug 3, 2014

I've completed the configuration, so it's now possible to plug in different rack apps for different hosts to mock out third party services - or to deliver 404 errors in case you just don't want to bother (like for google analytics).

twalpole and others added some commits Aug 14, 2014

Always convert visit's URL to a String before manipulation
It is convenient to be able to pass in URI objects to `visit`, but
these need to be coerced to a string before actual usage. Most of
visit's code paths would convert the incoming url to a string
as-needed. However, the path used when `always_include_port` is
enabled did not, causing an error where `URI.parse` would try to parse
an existing URI object.

This change consolidates the coercion to string at the entrypoint of
the method.
Merge pull request #1380 from jnicklas/composable_version_test
check correct version string for rspec composable
Replace hardcoded `localhost` in spec helper.
This test had a hardcoded values for the URI to visit which varies for different test cases. When these specs are integrated in custom driver's test suite like capybara-mechanize, it leads to failures.

This replaces #1227, which was rejected for changing the original purpose of the test. However, this patch only changes the hardcoding. It will allow jeroenvandijk/capybara-mechanize#55 to proceed and bump it's compatibility to beyond capybara 2.1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment