New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

webdrivers gem not consistent with MRI #4939

Open
boris-petrov opened this Issue Jan 5, 2018 · 16 comments

Comments

Projects
None yet
3 participants
@boris-petrov

boris-petrov commented Jan 5, 2018

Environment

% jruby -v
jruby 9.1.15.0 (2.3.3) 2017-12-07 929fde8 Java HotSpot(TM) 64-Bit Server VM 25.152-b16 on 1.8.0_152-b16 +jit [linux-x86_64]
% uname -a
Linux Boris-Desktop 4.14.11-1-ARCH #1 SMP PREEMPT Wed Jan 3 07:02:42 UTC 2018 x86_64 GNU/Linux

With MRI:

% sudo gem install webdrivers watir
% irb
irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-05 11:53:02 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-05 11:53:02 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-05 11:53:03 DEBUG Webdrivers Versions now located on downloads site: [0.191, 0.19, 0.18, 0.17, 0.161, 0.16, 0.15, 0.14, 0.13, 0.12]
2018-01-05 11:53:03 DEBUG Webdrivers Checking current version
2018-01-05 11:53:03 DEBUG Webdrivers File is already downloaded: false
2018-01-05 11:53:03 DEBUG Webdrivers Deleting /home/boris/.webdrivers/geckodriver
2018-01-05 11:53:04 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-05 11:53:04 DEBUG Webdrivers Versions previously located on downloads site: [0.191, 0.19, 0.18, 0.17, 0.161, 0.16, 0.15, 0.14, 0.13, 0.12]
2018-01-05 11:53:05 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-05 11:53:05 DEBUG Webdrivers Versions previously located on downloads site: [0.191, 0.19, 0.18, 0.17, 0.161, 0.16, 0.15, 0.14, 0.13, 0.12]
2018-01-05 11:53:06 DEBUG Webdrivers Redirected to https://github-production-release-asset-2e65be.s3.amazonaws.com/25354393/e31e4c22-be6f-11e7-9bc7-dedc3490a7fd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180105%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180105T095306Z&X-Amz-Expires=300&X-Amz-Signature=7393826d1fabdfd7de18b5ad256d7cc5b8eb239f5f97f765fb9717d236c426c3&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dgeckodriver-v0.19.1-linux64.tar.gz&response-content-type=application%2Foctet-stream
2018-01-05 11:53:08 DEBUG Webdrivers Successfully downloaded geckodriver-v0.19.1-linux64.tar.gz
2018-01-05 11:53:08 DEBUG Webdrivers Decompressing tar
2018-01-05 11:53:08 DEBUG Webdrivers Decompression Complete
2018-01-05 11:53:08 DEBUG Webdrivers Deleting geckodriver-v0.19.1-linux64.tar.gz
2018-01-05 11:53:08 DEBUG Webdrivers Completed download and processing of /home/boris/.webdrivers/geckodriver
=> #<Watir::Browser:0x6b888fedfef1ca98 url="about:blank" title="">

With JRuby:

% sudo jruby -S gem install webdrivers watir
% jruby -S irb
irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-05 11:51:12 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-05 11:51:13 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-05 11:51:14 DEBUG Webdrivers Versions now located on downloads site: []
Selenium::WebDriver::Error::WebDriverError: unexpected response, code=500, content-type=""
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/remote/http/common.rb:90:in `create_response'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/remote/http/default.rb:107:in `request'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/remote/http/common.rb:61:in `call'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/remote/bridge.rb:170:in `execute'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/remote/bridge.rb:103:in `create_session'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/firefox/marionette/driver.rb:57:in `initialize'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/firefox/driver.rb:33:in `new'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver/common/driver.rb:54:in `for'
	from /opt/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.5.0/lib/selenium/webdriver.rb:86:in `for'
	from /opt/jruby/lib/ruby/gems/shared/gems/watir-6.10.2/lib/watir/browser.rb:48:in `initialize'
	from (irb):4:in `<eval>'
	from org/jruby/RubyKernel.java:994:in `eval'
	from org/jruby/RubyKernel.java:1292:in `loop'
	from org/jruby/RubyKernel.java:1114:in `catch'
	from org/jruby/RubyKernel.java:1114:in `catch'
	from /opt/jruby/bin/jirb:13:in `<main>'

P.S. issue in the gem.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jan 25, 2018

Member

@boris-petrov Can you narrow down what it is we're doing wrong within Watir and its dependencies? Perhaps at the point of the error, compare the request/etc with what you'd see in MRI?

Member

headius commented Jan 25, 2018

@boris-petrov Can you narrow down what it is we're doing wrong within Watir and its dependencies? Perhaps at the point of the error, compare the request/etc with what you'd see in MRI?

@boris-petrov

This comment has been minimized.

Show comment
Hide comment
@boris-petrov

boris-petrov Jan 25, 2018

@headius - thanks for the answer! I don't think I'll have the time to debug this any time soon. The steps I've given above are simple to reproduce so hopefully someone can try figuring it out. It's not urgent at all, it's just something to be fixed at some point. Sorry I can't be of more help right now and thanks for the hard work on JRuby!

boris-petrov commented Jan 25, 2018

@headius - thanks for the answer! I don't think I'll have the time to debug this any time soon. The steps I've given above are simple to reproduce so hopefully someone can try figuring it out. It's not urgent at all, it's just something to be fixed at some point. Sorry I can't be of more help right now and thanks for the hard work on JRuby!

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jan 25, 2018

Member

Hmm, I got a different result. What did I do wrong?

$ jruby blah.rb
2018-01-25 12:00:48 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:00:48 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:00:49 DEBUG Webdrivers Versions now located on downloads site: []
StandardError: Unable to find the latest version of geckodriver; try downloading manually from https://github.com/mozilla/geckodriver/releases and place in /Users/headius/.webdrivers
       update at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/webdrivers-3.2.4/lib/webdrivers/common.rb:19
  driver_path at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/webdrivers-3.2.4/lib/webdrivers/selenium.rb:13
   initialize at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/firefox/marionette/driver.rb:37
          new at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/firefox/driver.rb:31
          for at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/common/driver.rb:52
          for at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver.rb:84
   initialize at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/watir-6.10.2/lib/watir/browser.rb:48
       <main> at blah.rb:4
Member

headius commented Jan 25, 2018

Hmm, I got a different result. What did I do wrong?

$ jruby blah.rb
2018-01-25 12:00:48 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:00:48 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:00:49 DEBUG Webdrivers Versions now located on downloads site: []
StandardError: Unable to find the latest version of geckodriver; try downloading manually from https://github.com/mozilla/geckodriver/releases and place in /Users/headius/.webdrivers
       update at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/webdrivers-3.2.4/lib/webdrivers/common.rb:19
  driver_path at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/webdrivers-3.2.4/lib/webdrivers/selenium.rb:13
   initialize at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/firefox/marionette/driver.rb:37
          new at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/firefox/driver.rb:31
          for at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver/common/driver.rb:52
          for at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/selenium-webdriver-3.8.0/lib/selenium/webdriver.rb:84
   initialize at /Users/headius/projects/jruby/lib/ruby/gems/shared/gems/watir-6.10.2/lib/watir/browser.rb:48
       <main> at blah.rb:4
@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jan 25, 2018

Member

I followed the instructions in the error, and I still get a different result:

$ jruby blah.rb
2018-01-25 12:02:58 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:02:58 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:02:59 DEBUG Webdrivers Versions now located on downloads site: []

And that's it.

Member

headius commented Jan 25, 2018

I followed the instructions in the error, and I still get a different result:

$ jruby blah.rb
2018-01-25 12:02:58 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:02:58 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-25 12:02:59 DEBUG Webdrivers Versions now located on downloads site: []

And that's it.

@boris-petrov

This comment has been minimized.

Show comment
Hide comment
@boris-petrov

boris-petrov Jan 26, 2018

@headius - thanks for trying! Now I also receive a different result from the one I pasted before. But in any case the result from MRI is different from the one with JRuby - MRI says:

irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-26 13:47:51 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:47:52 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:47:54 DEBUG Webdrivers Versions now located on downloads site: [0.191, 0.19, 0.18, 0.17, 0.161, 0.16, 0.15, 0.14, 0.13, 0.12]
2018-01-26 13:47:54 DEBUG Webdrivers Checking current version
2018-01-26 13:47:54 DEBUG Webdrivers File is already downloaded: true
2018-01-26 13:47:54 DEBUG Webdrivers Current version of /home/boris/.webdrivers/geckodriver is geckodriver 0.19.1

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

=> #<Watir::Browser:0x7d1eda28bc5bf2ee url="about:blank" title="">

And JRuby says:

irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-26 13:46:36 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:46:37 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:46:39 DEBUG Webdrivers Versions now located on downloads site: []
=> #<Watir::Browser:0x51dd1058 url="about:blank" title="">

Which doesn't make sense as the commands were exactly the same (the ones from my original post). You can reproduce that same thing, yes?

boris-petrov commented Jan 26, 2018

@headius - thanks for trying! Now I also receive a different result from the one I pasted before. But in any case the result from MRI is different from the one with JRuby - MRI says:

irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-26 13:47:51 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:47:52 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:47:54 DEBUG Webdrivers Versions now located on downloads site: [0.191, 0.19, 0.18, 0.17, 0.161, 0.16, 0.15, 0.14, 0.13, 0.12]
2018-01-26 13:47:54 DEBUG Webdrivers Checking current version
2018-01-26 13:47:54 DEBUG Webdrivers File is already downloaded: true
2018-01-26 13:47:54 DEBUG Webdrivers Current version of /home/boris/.webdrivers/geckodriver is geckodriver 0.19.1

The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.

This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

=> #<Watir::Browser:0x7d1eda28bc5bf2ee url="about:blank" title="">

And JRuby says:

irb(main):001:0> require 'webdrivers'
=> true
irb(main):002:0> Webdrivers.logger.level = :debug
=> :debug
irb(main):003:0> require 'watir'
=> true
irb(main):004:0> browser = Watir::Browser.new :firefox
2018-01-26 13:46:36 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:46:37 DEBUG Webdrivers Found Site: https://github.com/mozilla/geckodriver/releases
2018-01-26 13:46:39 DEBUG Webdrivers Versions now located on downloads site: []
=> #<Watir::Browser:0x51dd1058 url="about:blank" title="">

Which doesn't make sense as the commands were exactly the same (the ones from my original post). You can reproduce that same thing, yes?

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Feb 13, 2018

Member

@boris-petrov Yeah I get the same result. I'm not sure what to look into though...I have no idea what any of this output even means. Perhaps there's someone from watir we can get involved here to help us narrow down what's different? There's a lot of code here.

Member

headius commented Feb 13, 2018

@boris-petrov Yeah I get the same result. I'm not sure what to look into though...I have no idea what any of this output even means. Perhaps there's someone from watir we can get involved here to help us narrow down what's different? There's a lot of code here.

@boris-petrov

This comment has been minimized.

Show comment
Hide comment
@boris-petrov

boris-petrov Feb 14, 2018

I don't have an idea either unfortunately. Should I open an issue in watir? Not sure how to approach this.

The idea of webdrivers is that it downloads the latest available version of a Selenium driver from the corresponding website. I'm not sure which of those lines actually does that though.

boris-petrov commented Feb 14, 2018

I don't have an idea either unfortunately. Should I open an issue in watir? Not sure how to approach this.

The idea of webdrivers is that it downloads the latest available version of a Selenium driver from the corresponding website. I'm not sure which of those lines actually does that though.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Feb 17, 2018

Member

@boris-petrov Opening a bug with watir would be a good place to start. We can make it clear we don't think it's their bug, but we need help diagnosing the actual problem.

Member

headius commented Feb 17, 2018

@boris-petrov Opening a bug with watir would be a good place to start. We can make it clear we don't think it's their bug, but we need help diagnosing the actual problem.

@titusfortner

This comment has been minimized.

Show comment
Hide comment
@titusfortner

titusfortner Feb 19, 2018

@boris-petrov can you throw a debug point in your code and see what the output is of this: https://github.com/titusfortner/webdrivers/blob/master/lib/webdrivers/common.rb#L69

Webdrivers is calling http.get_response on https://github.com/mozilla/geckodriver/releases, and then using nokogiri to parse out the values of the versions found on that page.

My guess is that jruby is returning something different with that call than mri so it isn't finding the same thing to parse, but that's the information that is needed to debug it.

titusfortner commented Feb 19, 2018

@boris-petrov can you throw a debug point in your code and see what the output is of this: https://github.com/titusfortner/webdrivers/blob/master/lib/webdrivers/common.rb#L69

Webdrivers is calling http.get_response on https://github.com/mozilla/geckodriver/releases, and then using nokogiri to parse out the values of the versions found on that page.

My guess is that jruby is returning something different with that call than mri so it isn't finding the same thing to parse, but that's the information that is needed to debug it.

@boris-petrov

This comment has been minimized.

Show comment
Hide comment
@boris-petrov

boris-petrov Feb 23, 2018

@titusfortner - sorry for the late response. I'll try this next week as I had my hands full this one.

boris-petrov commented Feb 23, 2018

@titusfortner - sorry for the late response. I'll try this next week as I had my hands full this one.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Feb 23, 2018

Member

@titusfortner Thanks for the assistance!

Member

headius commented Feb 23, 2018

@titusfortner Thanks for the assistance!

@boris-petrov

This comment has been minimized.

Show comment
Hide comment
@boris-petrov

boris-petrov Feb 28, 2018

@titusfortner, @headius - sorry for the delay. The response thing is exactly the same in both MRI and JRuby - that is, the response's body is the same. So I guess the issue is somewhere later when that body is parsed. Any further ideas? Actually you could try it out yourselves with the steps from the first post - it's pretty easy to reproduce.

boris-petrov commented Feb 28, 2018

@titusfortner, @headius - sorry for the delay. The response thing is exactly the same in both MRI and JRuby - that is, the response's body is the same. So I guess the issue is somewhere later when that body is parsed. Any further ideas? Actually you could try it out yourselves with the steps from the first post - it's pretty easy to reproduce.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Mar 19, 2018

Member

@boris-petrov Is it easy to isolate "when that body is parsed" and narrow it down that way? I'm not familiar with the code. I have been able to reproduce but got a bit lost trying to sort out why the bug happens.

Member

headius commented Mar 19, 2018

@boris-petrov Is it easy to isolate "when that body is parsed" and narrow it down that way? I'm not familiar with the code. I have been able to reproduce but got a bit lost trying to sort out why the bug happens.

@titusfortner

This comment has been minimized.

Show comment
Hide comment
@titusfortner

titusfortner Mar 20, 2018

@headius yeah, whatever the output of the response is, it's just getting parsed it with Nokogiri here: https://github.com/titusfortner/webdrivers/blob/master/lib/webdrivers/geckodriver.rb#L26.

Actually, if the response is the same, is it possible that this case statement takes a different path between ruby implementations? Should I have a different conditional in there?

titusfortner commented Mar 20, 2018

@headius yeah, whatever the output of the response is, it's just getting parsed it with Nokogiri here: https://github.com/titusfortner/webdrivers/blob/master/lib/webdrivers/geckodriver.rb#L26.

Actually, if the response is the same, is it possible that this case statement takes a different path between ruby implementations? Should I have a different conditional in there?

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Mar 26, 2018

Member

@titusfortner I wouldn't expect that to be different. More likely the XML parsing is producing slightly different results so webdrivers can't fine the URL it's looking for.

I think the next step is to figure whether the response content is the same. If it is, then it's an issue of parsing in nokogiri, and we should be able to just proceed from the point in the code you highlighted.

Member

headius commented Mar 26, 2018

@titusfortner I wouldn't expect that to be different. More likely the XML parsing is producing slightly different results so webdrivers can't fine the URL it's looking for.

I think the next step is to figure whether the response content is the same. If it is, then it's an issue of parsing in nokogiri, and we should be able to just proceed from the point in the code you highlighted.

@titusfortner

This comment has been minimized.

Show comment
Hide comment
@titusfortner

titusfortner Oct 22, 2018

@boris-petrov looks like I was trying to parse the geckodriver page with XML instead of HTML. Please try with 3.4.3.

titusfortner commented Oct 22, 2018

@boris-petrov looks like I was trying to parse the geckodriver page with XML instead of HTML. Please try with 3.4.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment