Running on Ubuntu 20.04 gives chromedriver error

[ylluminarious]

I wrote a scraper that works just fine on macOS. When I run the same scraper on Ubuntu 20.04, I get an error:

/home/username/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/service.rb:104:in `start': already started: #<URI::HTTP http://127.0.0.1:9515/> "/usr/local/bin/chromedriver" (RuntimeError)

To make sure it wasn't just my own script, I tried running:

$ kimurai console --engine selenium_chrome --url https://github.com/vifreefly/kimuraframework

This also failed with the same error:

D, [2020-11-13 15:34:54 -0500#7086] [M: 47354020103620] DEBUG -- : BrowserBuilder (selenium_chrome): created browser instance
D, [2020-11-13 15:34:54 -0500#7086] [M: 47354020103620] DEBUG -- : BrowserBuilder (selenium_chrome): enabled native headless_mode
I, [2020-11-13 15:34:54 -0500#7086] [M: 47354020103620]  INFO -- : Browser: started get request to: https://github.com/vifreefly/kimuraframework
Traceback (most recent call last):
	22: from /home/username/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in `<main>'
	21: from /home/username/.rvm/gems/ruby-2.5.3/bin/ruby_executable_hooks:24:in `eval'
	20: from /home/username/.rvm/gems/ruby-2.5.3/bin/kimurai:23:in `<main>'
	19: from /home/username/.rvm/gems/ruby-2.5.3/bin/kimurai:23:in `load'
	18: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/exe/kimurai:6:in `<top (required)>'
	17: from /home/username/.rvm/gems/ruby-2.5.3/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	16: from /home/username/.rvm/gems/ruby-2.5.3/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	15: from /home/username/.rvm/gems/ruby-2.5.3/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	14: from /home/username/.rvm/gems/ruby-2.5.3/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	13: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/lib/kimurai/cli.rb:123:in `console'
	12: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/lib/kimurai/base.rb:201:in `request_to'
	11: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/lib/kimurai/capybara_ext/session.rb:52:in `visit'
	10: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/lib/kimurai/capybara_ext/session.rb:51:in `ensure in visit'
	 9: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/lib/kimurai/capybara_ext/driver/base.rb:16:in `current_memory'
	 8: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/lib/kimurai/capybara_ext/selenium/driver.rb:28:in `pid'
	 7: from /home/username/.rvm/gems/ruby-2.5.3/gems/kimurai-1.4.0/lib/kimurai/capybara_ext/selenium/driver.rb:32:in `port'
	 6: from /home/username/.rvm/gems/ruby-2.5.3/gems/capybara-3.13.2/lib/capybara/selenium/driver.rb:32:in `browser'
	 5: from /home/username/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver.rb:88:in `for'
	 4: from /home/username/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:46:in `for'
	 3: from /home/username/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:46:in `new'
	 2: from /home/username/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/chrome/driver.rb:40:in `initialize'
	 1: from /home/username/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:303:in `service_url'
/home/username/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/service.rb:104:in `start': already started: #<URI::HTTP http://127.0.0.1:9515/> "/usr/local/bin/chromedriver" (RuntimeError)

Again, this error only occurs on Ubuntu 20.04. When I try the same command on macOS, it works perfectly. I followed the Ubuntu 18.04 installation instructions in the README file, with the excption that I have installed the most recent versions of chromedriver and geckodriver like so:

cd /tmp && wget https://chromedriver.storage.googleapis.com/87.0.4280.20/chromedriver_linux64.zip
sudo unzip chromedriver_linux64.zip -d /usr/local/bin
cd /tmp && wget https://github.com/mozilla/geckodriver/releases/download/v0.28.0/geckodriver-v0.28.0-linux64.tar.gz
sudo tar -xvzf geckodriver-v0.28.0-linux64.tar.gz -C /usr/local/bin
rm -f geckodriver-v0.28.0-linux64.tar.gz

The reason I did this was because other people reported similar errors and suggested upgrading chromedriver to eliminate them. This did not fix the issue for me, but I figured that it's better to have more recent versions anyway.

When I looked in htop to see why this error was occurring, I noticed that on Ubuntu, there were multiple instances of chromedriver opening during this command, whereas in macOS only one instance of it opened. I am not sure why this is happening, but I suspect it is related to the error because it's apparently complaining about more than one chromedriver instance being open.

[GarnicaJR]

Try kill the process in that port as shown in this link https://stackoverflow.com/questions/11583562/how-to-kill-a-process-running-on-particular-port-in-linux

Example

fuser -k 9515/tcp

[ylluminarious]

@GarnicaJR Thanks for the advice! Strangely, this error is now also occurring on macOS. I've been running it fine on macOS for some weeks, but today this error suddenly appeared on macOS too.

So I found a macOS equivalent for the command you sent me: sudo lsof -nP -iTCP:9515 | grep LISTEN. These were the results I got:

COMMAND   PID   USER        FD  TYPE DEVICE                  SIZE/OFF  NODE NAME
...
chromedri 64183 username    6u  IPv4 0xf750323f8e3fb91f      0t0       TCP  127.0.0.1:9515 (LISTEN)
chromedri 64183 username    7u  IPv6 0xf750323f5877b74f      0t0       TCP  [::1]:9515 (LISTEN)

So it seems there are two connections being formed at startup. FYI, I can only get this output right before chromedriver starts. I have to keep running the command, because the output is empty until chromedriver actually runs for a split second before it crashes.

It seems it's spawning both an IPv4 and an IPv6 connection at the same time. Could this be the root of the problem? If so, how can I get chromedriver to stop doing that?

[ylluminarious]

@GarnicaJR Oh, I solved the issue. It turns out I just needed to update chromedriver. I had been running version 85.0.4183.87. I updated to 87.0.4280.20 and the issue went away (at least, on my macOS computer).

I saw some other people discussing this issue here and elsewhere; they suggested using the --whitelisted-ips flag, but that didn't seem to work for me. I also couldn't figure out a way to pass flags to chromedriver in Kimurai, so that made things rather complicated. I ended up creating a little shim script which called chromedriver with the --whitelisted-ips flag and set config.chromedriver_path to this script, but that didn't work.

So it turned out the main issue was just that my chromedriver was out of date. Go figure 🤷‍♂️. Perhaps the same issue exists for my Ubuntu installation? Not sure...

[jethrodaniel]

Had a similar issue on Mac (Montery): already started: #<URI::HTTP http://127.0.0.1:9515/> "/usr/local/bin/chromedriver".

Turns out it was due to mismatched versions of chrome and chromedriver:

 % /usr/local/bin/chromedriver --version && /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version                                                                                                                     
                                                                                                                                    
ChromeDriver 97.0.4692.71 (adefa7837d02a07a604c1e6eff0b3a09422ab88d-refs/branch-heads/4692@{#1247})
Google Chrome 96.0.4664.110 

Updating chrome to the same major version fixed the issue.