Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parallelization issues #77

Closed
beauraF opened this issue Apr 28, 2019 · 18 comments
Closed

parallelization issues #77

beauraF opened this issue Apr 28, 2019 · 18 comments
Assignees
Milestone

Comments

@beauraF
Copy link

beauraF commented Apr 28, 2019

Hello,

Wedrivers is having some problems with parallelization. This makes it unusable on Heroku CI and under Rails 6.
One of the threads downloads the driver while the others try to access the resource. Which leads to this error.

Error:
AuthenticationsTest#test_successful_login:
Zip::Error: File /home/circleci/.webdrivers/chromedriver_linux64.zip has zero size. Did you mean to pass the create flag?
    test/system/authentications_test.rb:7:in `block in <class:AuthenticationsTest>'


rails test test/system/authentications_test.rb:6
Error:
AuthenticationsTest#test_failed_login:
ChildProcess::LaunchError: Text file busy - /home/circleci/.webdrivers/chromedriver
    test/system/authentications_test.rb:14:in `block in <class:AuthenticationsTest>'


rails test test/system/authentications_test.rb:13

Traceback (most recent call last):
	13: from /home/circleci/doctopod/vendor/bundle/gems/minitest-5.11.3/lib/minitest.rb:63:in `block in autorun'
	12: from /home/circleci/doctopod/vendor/bundle/gems/minitest-5.11.3/lib/minitest.rb:133:in `run'
	11: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `start'
	10: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `map'
	 9: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `each'
	 8: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `times'
	 7: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:71:in `block in start'
	 6: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:71:in `fork'
	 5: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/platform.rb:150:in `block in exit_hook'
	 4: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/service.rb:105:in `stop'
	 3: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/service.rb:170:in `stop_server'
	 2: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/service.rb:180:in `process_exited?'
	 1: from /home/circleci/doctopod/vendor/bundle/gems/childprocess-1.0.1/lib/childprocess/unix/process.rb:31:in `exited?'
/home/circleci/doctopod/vendor/bundle/gems/childprocess-1.0.1/lib/childprocess/abstract_process.rb:188:in `assert_started': process not started (ChildProcess::Error)
	14: from /home/circleci/doctopod/vendor/bundle/gems/minitest-5.11.3/lib/minitest.rb:63:in `block in autorun'
	13: from /home/circleci/doctopod/vendor/bundle/gems/minitest-5.11.3/lib/minitest.rb:133:in `run'
	12: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `start'
	11: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `map'
	10: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `each'
	 9: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:70:in `times'
	 8: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:71:in `block in start'
	 7: from /home/circleci/doctopod/vendor/bundle/gems/activesupport-6.0.0.rc1/lib/active_support/testing/parallelization.rb:71:in `fork'
	 6: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/platform.rb:150:in `block in exit_hook'
	 5: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/service.rb:110:in `stop'
	 4: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/service.rb:110:in `ensure in stop'
	 3: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/service.rb:163:in `stop_process'
	 2: from /home/circleci/doctopod/vendor/bundle/gems/selenium-webdriver-3.142.0/lib/selenium/webdriver/common/service.rb:180:in `process_exited?'
	 1: from /home/circleci/doctopod/vendor/bundle/gems/childprocess-1.0.1/lib/childprocess/unix/process.rb:31:in `exited?'
/home/circleci/doctopod/vendor/bundle/gems/childprocess-1.0.1/lib/childprocess/abstract_process.rb:188:in `assert_started': process not started (ChildProcess::Error)

These are logs under Rails 6 and Circle CI. But I get the same result on Rails 5.2.3 with Heroku CI.

If you have any idea how we can handle this, I'm ready to work on it.

Thanks! Keep up the good work!

@Govan
Copy link

Govan commented Apr 29, 2019

@beauraF When running under parallel_test we're able to solve this by downloading one driver per process....

require 'webdrivers'
Webdrivers::Chromedriver.version = '2.35'
Webdrivers.install_dir = File.expand_path('~/.webdrivers/' + ENV['TEST_ENV_NUMBER'].to_s)

@kevindew
Copy link

kevindew commented May 1, 2019

I've just experienced the same problem. We have a Rails app that is started with foreman with web and worker processes. These hit the same empty zip file issue.

Some ideas to work around this could be:

  1. Giving the zip a UUID based file name (or dealing with it as a temp file) so two processes can't be
  2. Keeping the zip around afterwards rather than deleting it, so a second process decompressing it isn't fatal
  3. Less nice, but could have a rake task that could be run before parallelisation to initialise webdrivers

@titusfortner
Copy link
Owner

Rake tasks are a good idea, regardless. I'll see if I can figure out how to add those intelligently.

As for collisions, I don't want to have someone running 30 tests in parallel that each download a new driver, regardless of how we would keep them from overwriting each other. I'm thinking maybe we can create a local file and use its timestamp to "lock" new downloads. It's also how I'm planning on implementing #29 so can probably pull double duty. We'll see what we can do.

@titusfortner
Copy link
Owner

just merged #84 which will address the immediate issue if not the underlying problem.

@beauraF
Copy link
Author

beauraF commented May 2, 2019

Awesome. It's a first step. Are you planning a release soon?

@titusfortner
Copy link
Owner

Yes, I'm loading up my old Windows machine to validate everything is working there first.

@titusfortner titusfortner self-assigned this May 5, 2019
@titusfortner titusfortner modified the milestones: 4.0, 3.9 May 5, 2019
This was referenced May 5, 2019
Closed
@titusfortner
Copy link
Owner

@beauraF @Govan @kevindew can you try 3.9 and set a cache value (recommended Webdrivers.cache_time = 86_400) and let me know if this is still an issue?

@beauraF
Copy link
Author

beauraF commented May 8, 2019

Same issue on my side: https://circleci.com/gh/beauraF/doctopod/57

@titusfortner
Copy link
Owner

Oh, I think I see what is happening. I'll get a point release with the change and we can reevaluate.

@beauraF
Copy link
Author

beauraF commented May 9, 2019

@titusfortner If you need access to my project/my CI to perform tests. No worries, I can provide you an access.

@ryan-plated
Copy link

I just run Webdrivers::Chromedriver.update before fork

@titusfortner
Copy link
Owner

titusfortner commented May 9, 2019

Yeah, I think that's going to need to be the recommended solution, but we'll create some rake tasks in 4.0 to give people the option of doing it from command line instead of forcing everyone to execute it in their code.

@titusfortner
Copy link
Owner

Rake tasks have been merged with #117

@kevindew
Copy link

Thanks for your work on this @titusfortner 👍

@mike1o1
Copy link

mike1o1 commented Jun 2, 2019

It wasn't quite clear to me what the fix was, so I'm just adding this here incase others have the same issue.

To fix this with, for example, Circle CI, I added a new run task before my system tests run to execute the rake tasks mentioned above to update the drivers.

- run:
    name: Run tests
    environment:
      - CIRCLE_TEST_REPORTS: test/reports/test
    command: bundle exec rails test
    when: always

# This is the new task, run before the system tests execute
- run:
    name: Extract browser
    command: bundle exec rake webdrivers:chromedriver:update
    when: always

- run:
    name: Run system tests
    environment:
      - CIRCLE_TEST_REPORTS: test/reports/system
    command: bundle exec rails test:system
    when: always

@map7
Copy link

map7 commented Jul 1, 2019

I'm still on Rails 5.2.3 using the parallel_tests gem and ran into the same problem. The only fix which worked for me is;

Webdrivers.install_dir = File.expand_path('~/.webdrivers/' + ENV['TEST_ENV_NUMBER'].to_s)

barrucadu added a commit to alphagov/publishing-e2e-tests that referenced this issue Sep 29, 2021
We've started seeing the error

    ChildProcess::LaunchError:
      Text file busy - /root/.webdrivers/chromedriver

Which sounds a lot like
titusfortner/webdrivers#77, but that fix was
applied in 2019 and it solved the issues we had then.  This is a new
occurrence, possibly related to a new version of Chrome (as one came
out recently).

Downloading one copy of chrome for each test runner isn't great, but
it's acceptable if it fixes the issue.
barrucadu added a commit to alphagov/publishing-e2e-tests that referenced this issue Sep 29, 2021
We've started seeing the error

    ChildProcess::LaunchError:
      Text file busy - /root/.webdrivers/chromedriver

Which sounds a lot like
titusfortner/webdrivers#77, but that fix was
applied in 2019 and it solved the issues we had then.  This is a new
occurrence, possibly related to a new version of Chrome (as one came
out recently).

Downloading one copy of chrome for each test runner isn't great, but
it's acceptable if it fixes the issue.
@kevx86
Copy link

kevx86 commented Jun 20, 2023

Webdrivers.install_dir = File.expand_path('~/.webdrivers/' + ENV['TEST_ENV_NUMBER'].to_s)

@map7 Where did you place this line? Thanks,

@map7
Copy link

map7 commented Jun 20, 2023

@kevx86 I put it in the specs/rails_helper.rb and set environment variables in my .bashrc/.zshrc file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants