Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Test suite random failures #55

Closed
voxik opened this Issue Jul 24, 2012 · 1 comment

Comments

Projects
None yet
3 participants

voxik commented Jul 24, 2012

Running test suite just with polling it sometimes (like 2 times out of 10 tries) fails with the following error:

rspec spec
Run options: include {:focus=>true}
All examples were filtered out; ignoring {:focus=>true}
..........................................................F......................................................................................................................................
Failures:
  1) Listen::DirectoryRecord#fetch_changes with single file changes when a file is modified during the same second at which we are checking for changes doesn't detects the modified file the second time if the content haven't changed
     Failure/Error: modified.should be_empty
       expected empty? to return true, got false
     # /usr/share/gems/gems/rspec-expectations-2.8.0/lib/rspec/expectations/fail_with.rb:32:in `fail_with'
     # /usr/share/gems/gems/rspec-expectations-2.8.0/lib/rspec/expectations/handler.rb:21:in `handle_matcher'
     # /usr/share/gems/gems/rspec-expectations-2.8.0/lib/rspec/expectations/extensions/kernel.rb:12:in `should'
     # /builddir/build/BUILD/rubygem-listen-0.4.7/usr/share/gems/gems/listen-0.4.7/spec/listen/directory_record_spec.rb:437:in `block (7 levels) in <top (required)>'
     # /builddir/build/BUILD/rubygem-listen-0.4.7/usr/share/gems/gems/listen-0.4.7/spec/support/fixtures_helper.rb:24:in `fixtures'
     # /builddir/build/BUILD/rubygem-listen-0.4.7/usr/share/gems/gems/listen-0.4.7/spec/listen/directory_record_spec.rb:425:in `block (6 levels) in <top (required)>'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example.rb:80:in `instance_eval'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example.rb:80:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example.rb:173:in `with_around_hooks'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example.rb:77:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:355:in `block in run_examples'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:351:in `map'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:351:in `run_examples'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:337:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `map'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `map'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `map'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `map'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/example_group.rb:338:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/command_line.rb:28:in `block (2 levels) in run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/command_line.rb:28:in `map'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/command_line.rb:28:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/reporter.rb:34:in `report'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/command_line.rb:25:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/runner.rb:80:in `run_in_process'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/runner.rb:69:in `run'
     # /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/runner.rb:10:in `block in autorun'
Finished in 22.33 seconds
193 examples, 1 failure
Failed examples:
rspec ./spec/listen/directory_record_spec.rb:424 # Listen::DirectoryRecord#fetch_changes with single file changes when a file is modified during the same second at which we are checking for changes doesn't detects the modified file the second time if the content haven't changed
Randomized with seed 17364
Member

Maher4Ever commented Jul 25, 2012

Thanks for the report. This particular test case needs to be run in under a second for it to pass. As you might imagine, sometimes your CPU might be busy doing something else and can pause the rspec process. That's why we cannot really guarantee it'll always pass, although I know that tests should always be deterministic. We also can't freeze the time because then we won't be testing the functionality!

I'm open for suggestions on how to tackle this problem :)

@thibaudgg thibaudgg closed this Feb 4, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment