Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Problems with redirect to HTTPS #121

Open
andrewhavens opened this Issue · 30 comments
@andrewhavens

I have a cucumber test suite using Capybara and Selenium which is all passing. I'm trying to switch to Poltergeist to speed up my tests, however I'm having problems getting it to work.

The first step of my test is to go to the login page and log in, but it's failing. The browser should first navigate to the insecure http://dev.example.com/auth/login, then redirect to the secure https://dev.myapp.com/auth/login. We are using self-signed certificates, but I enabled ignore-ssl-errors and it's still failing:

# in my env.rb
Capybara.register_driver :poltergeist do |app|
  Capybara::Poltergeist::Driver.new(app, debug: true, phantomjs_options: ['--ignore-ssl-errors=yes', '--local-to-remote-url-access=yes'])
end

# output of running test
{"name"=>"visit", "args"=>["http://dev.myapp.com/auth/login"]}
{"response"=>"fail"}
{"name"=>"find", "args"=>[".//*[@id = 'login']"]}
{"response"=>{"page_id"=>0, "ids"=>[]}}
...and so on

What do I need to do to get this working?

@jonleighton
Owner

I think this may be a problem with not having the correct SSL libraries on your system.

  • What platform are you using?
  • Have you tried to connect to a HTTPS address using just phantomjs (not poltergeist)?
@andrewhavens

I'm running Cucumber (and subsequently Capybara, Poltergeist, and PhantomJS) on my Mac OS X 10.6. I haven't tried using PhantomJS directly because I don't know how to use it (I only learned about PhantomJS from finding Poltergeist while looking for a headless browser to use with Capybara). If you could give me some steps to take I can try to track this down further.

@jonleighton
Owner

create a test.js file like this:

page = require('webpage').create()
page.open('https://somewebsite.com/', function(status) {
  console.log(status)
})

(obviously fill in with a real url)

then run with phantomjs test.js and see if it works...

@andrewhavens

Okay, so I tried a few different things and PhantomJS is working as expected. When I use my insecure http url, it redirects me to the HTTPS version and succeeds. However, I had to make sure to pass the command line option correctly:

phantomjs test.js   # fails
phantomjs test.js --ignore-ssl-errors=yes     # fails
phantomjs --ignore-ssl-errors=yes test.js    #succeeds
@jonleighton
Owner

Okay, so HTTPS is working. What comes up in the rails log when you run your test?

@andrewhavens

The application I am testing is not a Rails app. It's not even Ruby. It's PHP. Sorry if I left that part out. =] So I'm not sure what you want me to look for. Are you interested in the Apache access logs? My app has a development log, but it only logs errors of which there are none that I know of. My question is what is Poltergeist doing with PhantomJS that causes this to fail in Poltergeist but not PhantomJS. Are the command line options being passed to PhantomJS correctly?

@jonleighton
Owner

Yes, Apache access logs would be interesting. I just want to see what requests actually get made. You could also print the output of page.driver.network_traffic.

@andrewhavens

At first, I wasn't able to get page.driver.network_traffic to work. I thought maybe it was recently added to master, but not released (or my Gemfile.lock had an earlier gem). Anyway, I changed my Gemfile to pull directly from master and started getting different errors. Once I updated, I could puts page.driver.network_traffic.inspect before "visiting" the page, but it was empty, and I couldn't inspect it after visiting because the error caused it to skip ahead to the next test.

The next difference I noticed when I switched to master is that rather than getting a {"response"=>"fail"} message, I was getting a TimeoutError in addition to a bunch of "PhantomJS client died while processing" errors.

The apache logs show that PhantomJS is actually redirecting to the HTTPS page and making requests for javascript files:

"GET /auth/login HTTP/1.1" 302 20 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.0 Safari/534.34"
"GET /auth/login HTTP/1.1" 200 3267 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.0 Safari/534.34"
"GET /js/dojo/dijit/themes/tundra/tundra.css HTTP/1.1" 200 10962 "https://dev.example.com/auth/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.0 Safari/534.34"
"GET /js/dojo/dojo/dojo.js HTTP/1.1" 200 27607 "https://dev.example.com/auth/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.0 Safari/534.34"
"GET /js/dojo/dojo/mysite_dojo.js HTTP/1.1" 200 76503 "https://dev.example.com/auth/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.0 Safari/534.34"

I am aware that there are a few areas of the site that have javascript errors (this is a legacy site that I am trying to start testing) but they aren't noticeable when you are viewing it from a browser so we haven't tried to fix them. When I ran the test.js I remembered that there were a few warnings, but it ended in success, so I didn't think anything of it:

Error: Could not load 'dojo.nls.mysite_dojo_en-us'; last tried './nls/mysite_dojo_en-us.js'

  https://dev.example.com/js/dojo/dojo/dojo.js:16
  https://dev.example.com/js/dojo/dojo/mysite_dojo.js:16
  https://dev.example.com/js/dojo/dojo/mysite_dojo.js:16
  https://dev.example.com/js/dojo/dojo/mysite_dojo.js:16 in _37d
  https://dev.example.com/js/dojo/dojo/mysite_dojo.js:16
  https://dev.example.com/js/dojo/dojo/mysite_dojo.js:16
success

So I started looking around in the other issues to see if maybe my issue is related to something else. That's when I came across issue #124 and pull request #125. It sounds like a very similar issue. So I decided to try and see if their fix would help me. So I switched to their fork:

gem 'poltergeist', require: 'capybara/poltergeist', git: 'https://github.com/Virtualmaster/poltergeist'

...and got a message very similar to the one from my PhantomJS test. Then setting the driver's js_errors: false option, I was able to get a little further, but now every time I run the test I get a "PhantomJS has crashed" error. It also mentions that I should file it as a bug report with PhantomJS but I don't know if it's really the fault of PhantomJS or this pull request.

So short story made long, I think my issue is more likely related to javascript errors and Poltergeist not handling them apropriately and/or a bug in PhantomJS.

@jonleighton
Owner

Regarding the crashing - PhantomJS should never crash so yeah, do file a bug report against PhantomJS. I'll look at #125 in due course...

@raphaelcm

I'm having similar problems. I had cucumber working fine without javascript, but when I added poltergeist, any https redirects that happen for @javascript scenarios fail. These redirects work fine when I remove the @javascript tag.

It appears that after being redirected to https://127.0.0.1:7787/login/new something fails and current_url points to about:blank instead of the login page.

Here's my Capybara config:

  Capybara.configure do |config|
    config.default_driver = :poltergeist
    config.javascript_driver = :poltergeist
    config.app_host = "http://127.0.0.1"
    config.run_server = true
    config.server_port = 7787
    config.default_wait_time = 10
  end

  Capybara.register_driver :poltergeist do |app|
    Capybara::Poltergeist::Driver.new(app,
                                      :phantomjs_options => ['--debug=yes', '--load-images=no', '--ignore-ssl-errors=yes'],
                                      :debug => true)
  end

  Before '@javascript' do
    Capybara.app_host = "http://127.0.0.1:#{Capybara.server_port}"
  end

Here's what the test does:

When /^I am logged in as "(.*?)" with password "(.*?)"$/ do |email, password|
  visit '/login/new'
  puts page.driver.network_traffic.inspect
  puts current_url
  puts page.body

  fill_in 'Email:', :with => email
  fill_in 'Password:', :with => password
  click_button 'Login'

  page.should have_content('Logged In') # Note: unicode space
end

Here's the output from cucumber:

Examples: 
      | customer type |
      |{"args"=>["http://127.0.0.1:7787/login/new"], "name"=>"visit"}
{"response"=>{"status"=>"fail"}}
{"args"=>[], "name"=>"network_traffic"}
{"response"=>{"2"=>{"responseParts"=>[{"headers"=>[], "time"=>"2012-12-21T20:42:29.954Z", "stage"=>"end", "redirectURL"=>nil, "id"=>2, "statusText"=>nil, "url"=>"https://127.0.0.1:7787/login/new", "contentType"=>nil, "status"=>nil}], "request"=>{"headers"=>[{"value"=>"Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.7.0 Safari/534.34", "name"=>"User-Agent"}, {"value"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "name"=>"Accept"}], "time"=>"2012-12-21T20:42:29.937Z", "id"=>2, "method"=>"GET", "url"=>"https://127.0.0.1:7787/login/new"}}, "1"=>{"responseParts"=>[{"headers"=>[{"value"=>"3.423566", "name"=>"X-Runtime"}, {"value"=>"Keep-Alive", "name"=>"Connection"}, {"value"=>"text/html; charset=utf-8", "name"=>"Content-Type"}, {"value"=>"miss", "name"=>"X-Rack-Cache"}, {"value"=>"Fri, 21 Dec 2012 20:42:29 GMT", "name"=>"Date"}, {"value"=>"IE=Edge,chrome=1", "name"=>"X-Ua-Compatible"}, {"value"=>"WEBrick/1.3.1 (Ruby/1.8.7/2012-02-08)", "name"=>"Server"}, {"value"=>"https://127.0.0.1:7787/login/new", "name"=>"Location"}, {"value"=>"98", "name"=>"Content-Length"}, {"value"=>"no-cache", "name"=>"Cache-Control"}], "time"=>"2012-12-21T20:42:29.936Z", "bodySize"=>98, "stage"=>"start", "redirectURL"=>"https://127.0.0.1:7787/login/new", "id"=>1, "statusText"=>"Found ", "url"=>"http://127.0.0.1:7787/login/new", "contentType"=>"text/html; charset=utf-8", "status"=>302}, {"headers"=>[{"value"=>"3.423566", "name"=>"X-Runtime"}, {"value"=>"Keep-Alive", "name"=>"Connection"}, {"value"=>"text/html; charset=utf-8", "name"=>"Content-Type"}, {"value"=>"miss", "name"=>"X-Rack-Cache"}, {"value"=>"Fri, 21 Dec 2012 20:42:29 GMT", "name"=>"Date"}, {"value"=>"IE=Edge,chrome=1", "name"=>"X-Ua-Compatible"}, {"value"=>"WEBrick/1.3.1 (Ruby/1.8.7/2012-02-08)", "name"=>"Server"}, {"value"=>"https://127.0.0.1:7787/login/new", "name"=>"Location"}, {"value"=>"98", "name"=>"Content-Length"}, {"value"=>"no-cache", "name"=>"Cache-Control"}], "time"=>"2012-12-21T20:42:29.937Z", "stage"=>"end", "redirectURL"=>"https://127.0.0.1:7787/login/new", "id"=>1, "statusText"=>"Found ", "url"=>"http://127.0.0.1:7787/login/new", "contentType"=>"text/html; charset=utf-8", "status"=>302}], "request"=>{"headers"=>[{"value"=>"Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.7.0 Safari/534.34", "name"=>"User-Agent"}, {"value"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "name"=>"Accept"}], "time"=>"2012-12-21T20:42:26.401Z", "id"=>1, "method"=>"GET", "url"=>"http://127.0.0.1:7787/login/new"}}}}
{"args"=>[], "name"=>"current_url"}
{"response"=>"about:blank"}
{"args"=>[], "name"=>"body"}
{"response"=>"<html><head></head><body></body></html>"}

then this is repeated a gazillion times:

{"args"=>[".//*[self::input | self::textarea][not(./@type = 'submit' or ./@type = 'image' or ./@type = 'radio' or ./@type = 'checkbox' or ./@type = 'hidden' or ./@type = 'file')][((./@id = 'Email:' or ./@name = 'Email:') or ./@id = //label[contains(normalize-space(string(.)), 'Email:')]/@for)] | .//label[contains(normalize-space(string(.)), 'Email:')]//.//*[self::input | self::textarea][not(./@type = 'submit' or ./@type = 'image' or ./@type = 'radio' or ./@type = 'checkbox' or ./@type = 'hidden' or ./@type = 'file')]"], "name"=>"find"}
{"response"=>{"ids"=>[], "page_id"=>0}}

lastly, this:

 listing       |{"args"=>[], "name"=>"reset"}
{"response"=>true}
  [#<Capybara::Poltergeist::NetworkTraffic::Request:0x11172bf70 @response_parts=[#<Capybara::Poltergeist::NetworkTraffic::Response:0x11172bf48 @data={"headers"=>[], "time"=>"2012-12-21T20:42:29.954Z", "stage"=>"end", "redirectURL"=>nil, "id"=>2, "statusText"=>nil, "url"=>"https://127.0.0.1:7787/login/new", "contentType"=>nil, "status"=>nil}>], @data={"headers"=>[{"value"=>"Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.7.0 Safari/534.34", "name"=>"User-Agent"}, {"value"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "name"=>"Accept"}], "time"=>"2012-12-21T20:42:29.937Z", "id"=>2, "method"=>"GET", "url"=>"https://127.0.0.1:7787/login/new"}>, #<Capybara::Poltergeist::NetworkTraffic::Request:0x11172be58 @response_parts=[#<Capybara::Poltergeist::NetworkTraffic::Response:0x11172be08 @data={"headers"=>[{"value"=>"3.423566", "name"=>"X-Runtime"}, {"value"=>"Keep-Alive", "name"=>"Connection"}, {"value"=>"text/html; charset=utf-8", "name"=>"Content-Type"}, {"value"=>"miss", "name"=>"X-Rack-Cache"}, {"value"=>"Fri, 21 Dec 2012 20:42:29 GMT", "name"=>"Date"}, {"value"=>"IE=Edge,chrome=1", "name"=>"X-Ua-Compatible"}, {"value"=>"WEBrick/1.3.1 (Ruby/1.8.7/2012-02-08)", "name"=>"Server"}, {"value"=>"https://127.0.0.1:7787/login/new", "name"=>"Location"}, {"value"=>"98", "name"=>"Content-Length"}, {"value"=>"no-cache", "name"=>"Cache-Control"}], "time"=>"2012-12-21T20:42:29.936Z", "bodySize"=>98, "stage"=>"start", "redirectURL"=>"https://127.0.0.1:7787/login/new", "id"=>1, "statusText"=>"Found ", "url"=>"http://127.0.0.1:7787/login/new", "contentType"=>"text/html; charset=utf-8", "status"=>302}>, #<Capybara::Poltergeist::NetworkTraffic::Response:0x11172bde0 @data={"headers"=>[{"value"=>"3.423566", "name"=>"X-Runtime"}, {"value"=>"Keep-Alive", "name"=>"Connection"}, {"value"=>"text/html; charset=utf-8", "name"=>"Content-Type"}, {"value"=>"miss", "name"=>"X-Rack-Cache"}, {"value"=>"Fri, 21 Dec 2012 20:42:29 GMT", "name"=>"Date"}, {"value"=>"IE=Edge,chrome=1", "name"=>"X-Ua-Compatible"}, {"value"=>"WEBrick/1.3.1 (Ruby/1.8.7/2012-02-08)", "name"=>"Server"}, {"value"=>"https://127.0.0.1:7787/login/new", "name"=>"Location"}, {"value"=>"98", "name"=>"Content-Length"}, {"value"=>"no-cache", "name"=>"Cache-Control"}], "time"=>"2012-12-21T20:42:29.937Z", "stage"=>"end", "redirectURL"=>"https://127.0.0.1:7787/login/new", "id"=>1, "statusText"=>"Found ", "url"=>"http://127.0.0.1:7787/login/new", "contentType"=>"text/html; charset=utf-8", "status"=>302}>], @data={"headers"=>[{"value"=>"Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.7.0 Safari/534.34", "name"=>"User-Agent"}, {"value"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "name"=>"Accept"}], "time"=>"2012-12-21T20:42:26.401Z", "id"=>1, "method"=>"GET", "url"=>"http://127.0.0.1:7787/login/new"}>], about:blank, <html><head></head><body></body></html>
      cannot fill in, no text field, text area or password field with id, name, or label 'Email:' found (Capybara::ElementNotFound)
      (eval):2:in `send'
      (eval):2:in `fill_in'
      ./features/step_definitions/authentication_steps.rb:7:in `/^I am logged in as "(.*?)" with password "(.*?)"$/'
      features/edit_listing.feature:17:in `Given I am a <customer type> customer'

Failing Scenarios:
cucumber features/edit_listing.feature:16 # Scenario: Edit the application deadline of a current listing

1 scenario (1 failed)
4 steps (1 failed, 2 skipped, 1 undefined)
0m25.986s

Here's what I see in test.log:

Started GET "/login/new" for 127.0.0.1 at Fri Dec 21 12:42:26 -0800 2012
  Processing by LoginController#new as HTML
Geokit is using the domain: 
Redirected to https://127.0.0.1:7787/login/new
Completed 302 Found in 1ms

I'm using the bartt-ssl_requirement gem. The relevant line in LoginController is:

ssl_required :create, :new
@rpanachi

I have the same problem as @raphaelcm

@gummybears

Also having the same problem.

@dmacvicar

I am seeing the same: empty screenshot/blank age for a site with a redirect.

@joevandyk

I see the same, empty page when trying to load a https:// site with poltergeist.

Works in capybara-webkit, fwiw.

@LoneWanderer04

I'm seeing the same on my end. It's proving problematic, since we want a headless setup on our automated build process.

As soon as I change my drivers to selenium rather then poltergeist, the page seems to redirect properly. The site I'm trying to hit basically has a URL that immediately is supposed to redirect the user to a sign in page, but it seems that redirect is never occurring, making all the tests fail since that page never loads.

If anybody needs anything to provide in order to help them track this down, I'm more then willing to provide. The same goes for if anybody knows a solution I might be missing to get this to work properly.

@streetlight

@joevandyk what OS are you running it on? Got it working on Ubuntu but failing on CentOS 6.5 and driving me crazy!

@dmacvicar

BTW, I just revisited this a year later and because the problem still happens with current phantomjs, I looked further and found a workaround: --ssl-protocol=TLSv1

Capybara.register_driver :poltergeist do |app|
  Capybara::Poltergeist::Driver.new(app,
    :phantomjs_options => ['--debug=no', '--load-images=no', '--ignore-ssl-errors=yes', '--ssl-protocol=TLSv1'], :debug => false)
  end
end

and it works!

(got the idea from here ariya/phantomjs#11239 )

@yaauie
Owner

:+1: thanks for the followup, @dmacvicar

@darkness51

No more information about this?? I am having this problem too with poltergeist 1.5.0 and phantomjs 1.9.0

@bobbysteel

Thanks @dmacvicar That solved the problem perfectly for me. Apparently some update seems to have arbitrarily broken SSL for me this week, but this fixed it right up

@KrauseFx

@dmacvicar Thanks, that solved my problem!

@brandoncc

@dmacvicar Thanks for that! After the POODLE exploit, SSL3 is going away so I think you have saved a lot of people's days!

@spiffxp spiffxp referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@portovep

@dmacvicar, many thanks!, that solved the problem for us.

@TriptiMittal

I am too facing this issue :
I have created an env.rb file as:

require "Capybara"
require "Capybara/cucumber"
require "rspec"
require "capybara/poltergeist"


 Capybara.default_selector = :xpath

Capybara.configure do |config|
    config.default_driver = :poltergeist
    config.javascript_driver = :poltergeist
    config.app_host = "http://127.0.0.1"
    config.run_server = true
    config.server_port = 7787
    config.default_wait_time = 10
  end

  Capybara.register_driver :poltergeist do |app|
  Capybara::Poltergeist::Driver.new(app,
    :phantomjs_options => ['--debug=no', '--load-images=no', '--ignore-ssl-errors=yes', '--ssl-protocol=TLSv1'], :debug => false)
  end
end

  Before '@javascript' do
    Capybara.app_host = "http://127.0.0.1:7787"
  end

My test does as

Given /^I am on "(.*?)" instance$/ do |sncurl|

 visit ('https://leanpub.com/whatsnewinjava8/read')

 puts page.driver.network_traffic.inspect
  puts current_url
  puts page.body
end

When I run this code I am receiving:

[#<Capybara::Poltergeist::NetworkTraffic::Request:0x419d888 @data={"headers"=>[{"name"=>"User-Agent", "value"=>"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Safari/534.34"}, {"na
me"=>"Accept", "value"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}], "id"=>1, "method"=>"GET", "time"=>"2014-11-28T10:38:41.066Z", "url"=>"https://leanpub.com/whatsnewinjava8/read"}, @response_parts=[#<Capy
bara::Poltergeist::NetworkTraffic::Response:0x41a0d08 @data={"contentType"=>nil, "headers"=>[], "id"=>1, "redirectURL"=>nil, "stage"=>"end", "status"=>nil, "statusText"=>nil, "time"=>"2014-11-28T10:38:41.670Z", "url"=>"https://lea
npub.com/whatsnewinjava8/read"}>]>]
      about:blank
      <html><head></head><body></body></html>

And when I redirect to gooogle it is working fine but not for other URLs.

@bobbysteel

@TriptiMittal turn on the debugging and see what you get. That was helpful for me once I enabled. You may want to experiment with using --ssl-protocol=auto as well, which seemed to also work for me.

@TriptiMittal

Hi.. I am new to this technology and don't know how to open the debugger.. Please help me know that how to do the same.

@bobbysteel

Change it in the obvious place:

:phantomjs_options => ['--debug=no', '--load-images=no', '--ignore-ssl-errors=yes', '--ssl-protocol=TLSv1'], :debug => false)
@TriptiMittal

Hi,
I received this as a debug

[#<Capybara::Poltergeist::NetworkTraffic::Request:0x409d448 @data={"headers"=>[{"name"=>"User-Agent", "value"=>"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.7 Safari/534.34"}, {"na
me"=>"Accept", "value"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}], "id"=>1, "method"=>"GET", "time"=>"2014-12-02T06:20:32.526Z", "url"=>"https://leanpub.com/whatsnewinjava8/read"}, @response_parts=[#<Capy
bara::Poltergeist::NetworkTraffic::Response:0x40a08c8 @data={"contentType"=>nil, "headers"=>[], "id"=>1, "redirectURL"=>nil, "stage"=>"end", "status"=>nil, "statusText"=>nil, "time"=>"2014-12-02T06:20:33.711Z", "url"=>"https://lea
npub.com/whatsnewinjava8/read"}>]>]
  "  <html><head></head><body></body></html>"
      about:blank

Moreover when I am using :phantomjs_options => ['--debug=yes', '--load-images=no', '--ignore-ssl-errors=yes', '--ssl-protocol=TLSv1'], :debug => true) to debug then nothing is coming up.

What changes need I to make to get an HTML page and proceed further with the tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.