Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests fail randomly #212

Closed
terceiro opened this issue Jun 29, 2017 · 4 comments
Closed

tests fail randomly #212

terceiro opened this issue Jun 29, 2017 · 4 comments

Comments

@terceiro
Copy link

Hi,

Running the rake tests fails depending on the (random) order in which they are executed. One can easily reproduce this by forcing the value of random seed with the SEED environment:

$ ruby -v
ruby 2.3.3p222 (2016-11-21) [x86_64-linux-gnu]
$ rake SEED=58133
/usr/bin/ruby2.3 -w -I"lib:test" -I"/home/terceiro/src/rake/lib" "/home/terceiro/src/rake/lib/rake/rake_test_loader.rb" "test/test_private_reader.rb" "test/test_rake.rb" "test/test_rake_application.rb" "test/test_rake_application_options.rb" "test/test_rake_backtrace.rb" "test/test_rake_clean.rb" "test/test_rake_cpu_counter.rb" "test/test_rake_definitions.rb" "test/test_rake_directory_task.rb" "test/test_rake_dsl.rb" "test/test_rake_early_time.rb" "test/test_rake_extension.rb" "test/test_rake_file_creation_task.rb" "test/test_rake_file_list.rb" "test/test_rake_file_list_path_map.rb" "test/test_rake_file_task.rb" "test/test_rake_file_utils.rb" "test/test_rake_functional.rb" "test/test_rake_invocation_chain.rb" "test/test_rake_late_time.rb" "test/test_rake_linked_list.rb" "test/test_rake_makefile_loader.rb" "test/test_rake_multi_task.rb" "test/test_rake_name_space.rb" "test/test_rake_package_task.rb" "test/test_rake_path_map.rb" "test/test_rake_path_map_explode.rb" "test/test_rake_path_map_partial.rb" "test/test_rake_pseudo_status.rb" "test/test_rake_rake_test_loader.rb" "test/test_rake_reduce_compat.rb" "test/test_rake_require.rb" "test/test_rake_rules.rb" "test/test_rake_scope.rb" "test/test_rake_task.rb" "test/test_rake_task_argument_parsing.rb" "test/test_rake_task_arguments.rb" "test/test_rake_task_manager.rb" "test/test_rake_task_manager_argument_resolution.rb" "test/test_rake_task_with_arguments.rb" "test/test_rake_test_task.rb" "test/test_rake_thread_pool.rb" "test/test_rake_top_level_functions.rb" "test/test_rake_win32.rb" "test/test_thread_history_display.rb" "test/test_trace_output.rb" 
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's default settings.
Run options: --seed 58133

# Running:

...........................................................................................................................................................................................................................................F................................................................................................E....F..............................................................................................................................................................................................................................................

Finished in 13.642206s, 42.2219 runs/s, 114.9374 assertions/s.

  1) Failure:
TestRakeRakeTestLoader#test_load_error [/home/terceiro/src/rake/test/test_rake_rake_test_loader.rb:31]:
[SystemExit] exception expected, not
Class: <NoMethodError>
Message: <"undefined method `name' for nil:NilClass">
---Backtrace---
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:189:in `block in to_spec'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `each'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `find'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `to_spec'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:148:in `missing_extensions?'
/usr/lib/ruby/2.3.0/rubygems/basic_specification.rb:71:in `contains_requirable_file?'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1037:in `block in find_by_path'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `each'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `find'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `find_by_path'
/usr/lib/ruby/2.3.0/rubygems.rb:189:in `try_activate'
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:125:in `rescue in require'
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
/home/terceiro/src/rake/lib/rake/rake_test_loader.rb:17:in `block in <top (required)>'
/home/terceiro/src/rake/lib/rake/rake_test_loader.rb:5:in `select'
/home/terceiro/src/rake/lib/rake/rake_test_loader.rb:5:in `<top (required)>'
/home/terceiro/src/rake/test/test_rake_rake_test_loader.rb:32:in `load'
/home/terceiro/src/rake/test/test_rake_rake_test_loader.rb:32:in `block (2 levels) in test_load_error'
/usr/lib/ruby/vendor_ruby/minitest/assertions.rb:313:in `assert_raises'
/home/terceiro/src/rake/test/test_rake_rake_test_loader.rb:31:in `block in test_load_error'
/usr/lib/ruby/vendor_ruby/minitest/assertions.rb:422:in `block in capture_io'
/usr/lib/ruby/vendor_ruby/minitest/assertions.rb:145:in `_synchronize'
/usr/lib/ruby/vendor_ruby/minitest/assertions.rb:415:in `capture_io'
/home/terceiro/src/rake/test/test_rake_rake_test_loader.rb:28:in `test_load_error'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:107:in `block (3 levels) in run'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:204:in `capture_exceptions'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:104:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:255:in `time_it'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:103:in `block in run'
/usr/lib/ruby/vendor_ruby/minitest.rb:348:in `on_signal'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:275:in `with_info_handler'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:102:in `run'
/usr/lib/ruby/vendor_ruby/minitest.rb:799:in `run_one_method'
/usr/lib/ruby/vendor_ruby/minitest.rb:322:in `run_one_method'
/usr/lib/ruby/vendor_ruby/minitest.rb:310:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/minitest.rb:309:in `each'
/usr/lib/ruby/vendor_ruby/minitest.rb:309:in `block in run'
/usr/lib/ruby/vendor_ruby/minitest.rb:348:in `on_signal'
/usr/lib/ruby/vendor_ruby/minitest.rb:335:in `with_info_handler'
/usr/lib/ruby/vendor_ruby/minitest.rb:308:in `run'
/usr/lib/ruby/vendor_ruby/minitest.rb:158:in `block in __run'
/usr/lib/ruby/vendor_ruby/minitest.rb:158:in `map'
/usr/lib/ruby/vendor_ruby/minitest.rb:158:in `__run'
/usr/lib/ruby/vendor_ruby/minitest.rb:135:in `run'
/usr/lib/ruby/vendor_ruby/minitest.rb:62:in `block in autorun'
---------------


  2) Error:
TestRakeApplicationOptions#test_require:
NoMethodError: undefined method `name' for nil:NilClass
    /usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:189:in `block in to_spec'
    /usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `each'
    /usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `find'
    /usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `to_spec'
    /usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:148:in `missing_extensions?'
    /usr/lib/ruby/2.3.0/rubygems/basic_specification.rb:71:in `contains_requirable_file?'
    /usr/lib/ruby/2.3.0/rubygems/specification.rb:1037:in `block in find_by_path'
    /usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `each'
    /usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `find'
    /usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `find_by_path'
    /usr/lib/ruby/2.3.0/rubygems.rb:189:in `try_activate'
    /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:125:in `rescue in require'
    /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
    /home/terceiro/src/rake/lib/rake/application.rb:513:in `block in standard_rake_options'
    /usr/lib/ruby/2.3.0/optparse.rb:1578:in `block in parse_in_order'
    /usr/lib/ruby/2.3.0/optparse.rb:1534:in `catch'
    /usr/lib/ruby/2.3.0/optparse.rb:1534:in `parse_in_order'
    /usr/lib/ruby/2.3.0/optparse.rb:1528:in `order!'
    /usr/lib/ruby/2.3.0/optparse.rb:1620:in `permute!'
    /usr/lib/ruby/2.3.0/optparse.rb:1642:in `parse!'
    /usr/lib/ruby/2.3.0/optparse.rb:1631:in `parse'
    /home/terceiro/src/rake/lib/rake/application.rb:637:in `handle_options'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:463:in `block in command_line'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:462:in `instance_eval'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:462:in `command_line'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:450:in `block (2 levels) in flags'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:450:in `catch'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:450:in `block in flags'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:447:in `each'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:447:in `flags'
    /home/terceiro/src/rake/test/test_rake_application_options.rb:175:in `test_require'
    /usr/lib/ruby/vendor_ruby/minitest/test.rb:107:in `block (3 levels) in run'
    /usr/lib/ruby/vendor_ruby/minitest/test.rb:204:in `capture_exceptions'
    /usr/lib/ruby/vendor_ruby/minitest/test.rb:104:in `block (2 levels) in run'
    /usr/lib/ruby/vendor_ruby/minitest/test.rb:255:in `time_it'
    /usr/lib/ruby/vendor_ruby/minitest/test.rb:103:in `block in run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:348:in `on_signal'
    /usr/lib/ruby/vendor_ruby/minitest/test.rb:275:in `with_info_handler'
    /usr/lib/ruby/vendor_ruby/minitest/test.rb:102:in `run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:799:in `run_one_method'
    /usr/lib/ruby/vendor_ruby/minitest.rb:322:in `run_one_method'
    /usr/lib/ruby/vendor_ruby/minitest.rb:310:in `block (2 levels) in run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:309:in `each'
    /usr/lib/ruby/vendor_ruby/minitest.rb:309:in `block in run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:348:in `on_signal'
    /usr/lib/ruby/vendor_ruby/minitest.rb:335:in `with_info_handler'
    /usr/lib/ruby/vendor_ruby/minitest.rb:308:in `run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:158:in `block in __run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:158:in `map'
    /usr/lib/ruby/vendor_ruby/minitest.rb:158:in `__run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:135:in `run'
    /usr/lib/ruby/vendor_ruby/minitest.rb:62:in `block in autorun'


  3) Failure:
TestRakeApplicationOptions#test_missing_require [/home/terceiro/src/rake/test/test_rake_application_options.rb:187]:
[LoadError] exception expected, not
Class: <NoMethodError>
Message: <"undefined method `name' for nil:NilClass">
---Backtrace---
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:189:in `block in to_spec'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `each'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `find'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:188:in `to_spec'
/usr/lib/ruby/2.3.0/rubygems/stub_specification.rb:148:in `missing_extensions?'
/usr/lib/ruby/2.3.0/rubygems/basic_specification.rb:71:in `contains_requirable_file?'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1037:in `block in find_by_path'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `each'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `find'
/usr/lib/ruby/2.3.0/rubygems/specification.rb:1036:in `find_by_path'
/usr/lib/ruby/2.3.0/rubygems.rb:189:in `try_activate'
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:125:in `rescue in require'
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
/home/terceiro/src/rake/lib/rake/application.rb:513:in `block in standard_rake_options'
/usr/lib/ruby/2.3.0/optparse.rb:1547:in `block in parse_in_order'
/usr/lib/ruby/2.3.0/optparse.rb:1534:in `catch'
/usr/lib/ruby/2.3.0/optparse.rb:1534:in `parse_in_order'
/usr/lib/ruby/2.3.0/optparse.rb:1528:in `order!'
/usr/lib/ruby/2.3.0/optparse.rb:1620:in `permute!'
/usr/lib/ruby/2.3.0/optparse.rb:1642:in `parse!'
/usr/lib/ruby/2.3.0/optparse.rb:1631:in `parse'
/home/terceiro/src/rake/lib/rake/application.rb:637:in `handle_options'
/home/terceiro/src/rake/test/test_rake_application_options.rb:463:in `block in command_line'
/home/terceiro/src/rake/test/test_rake_application_options.rb:462:in `instance_eval'
/home/terceiro/src/rake/test/test_rake_application_options.rb:462:in `command_line'
/home/terceiro/src/rake/test/test_rake_application_options.rb:450:in `block (2 levels) in flags'
/home/terceiro/src/rake/test/test_rake_application_options.rb:450:in `catch'
/home/terceiro/src/rake/test/test_rake_application_options.rb:450:in `block in flags'
/home/terceiro/src/rake/test/test_rake_application_options.rb:447:in `each'
/home/terceiro/src/rake/test/test_rake_application_options.rb:447:in `flags'
/home/terceiro/src/rake/test/test_rake_application_options.rb:188:in `block in test_missing_require'
/usr/lib/ruby/vendor_ruby/minitest/assertions.rb:313:in `assert_raises'
/home/terceiro/src/rake/test/test_rake_application_options.rb:187:in `test_missing_require'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:107:in `block (3 levels) in run'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:204:in `capture_exceptions'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:104:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:255:in `time_it'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:103:in `block in run'
/usr/lib/ruby/vendor_ruby/minitest.rb:348:in `on_signal'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:275:in `with_info_handler'
/usr/lib/ruby/vendor_ruby/minitest/test.rb:102:in `run'
/usr/lib/ruby/vendor_ruby/minitest.rb:799:in `run_one_method'
/usr/lib/ruby/vendor_ruby/minitest.rb:322:in `run_one_method'
/usr/lib/ruby/vendor_ruby/minitest.rb:310:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/minitest.rb:309:in `each'
/usr/lib/ruby/vendor_ruby/minitest.rb:309:in `block in run'
/usr/lib/ruby/vendor_ruby/minitest.rb:348:in `on_signal'
/usr/lib/ruby/vendor_ruby/minitest.rb:335:in `with_info_handler'
/usr/lib/ruby/vendor_ruby/minitest.rb:308:in `run'
/usr/lib/ruby/vendor_ruby/minitest.rb:158:in `block in __run'
/usr/lib/ruby/vendor_ruby/minitest.rb:158:in `map'
/usr/lib/ruby/vendor_ruby/minitest.rb:158:in `__run'
/usr/lib/ruby/vendor_ruby/minitest.rb:135:in `run'
/usr/lib/ruby/vendor_ruby/minitest.rb:62:in `block in autorun'
---------------

576 runs, 1568 assertions, 2 failures, 1 errors, 0 skips
/usr/lib/ruby/vendor_ruby/simplecov.rb:56: warning: instance variable @track_files_glob not initialized
[Coveralls] Outside the Travis environment, not sending data.
rake aborted!
Command failed with status (1): [ruby -w -I"lib:test" -I"/home/terceiro/src/rake/lib" "/home/terceiro/src/rake/lib/rake/rake_test_loader.rb" "test/test_private_reader.rb" "test/test_rake.rb" "test/test_rake_application.rb" "test/test_rake_application_options.rb" "test/test_rake_backtrace.rb" "test/test_rake_clean.rb" "test/test_rake_cpu_counter.rb" "test/test_rake_definitions.rb" "test/test_rake_directory_task.rb" "test/test_rake_dsl.rb" "test/test_rake_early_time.rb" "test/test_rake_extension.rb" "test/test_rake_file_creation_task.rb" "test/test_rake_file_list.rb" "test/test_rake_file_list_path_map.rb" "test/test_rake_file_task.rb" "test/test_rake_file_utils.rb" "test/test_rake_functional.rb" "test/test_rake_invocation_chain.rb" "test/test_rake_late_time.rb" "test/test_rake_linked_list.rb" "test/test_rake_makefile_loader.rb" "test/test_rake_multi_task.rb" "test/test_rake_name_space.rb" "test/test_rake_package_task.rb" "test/test_rake_path_map.rb" "test/test_rake_path_map_explode.rb" "test/test_rake_path_map_partial.rb" "test/test_rake_pseudo_status.rb" "test/test_rake_rake_test_loader.rb" "test/test_rake_reduce_compat.rb" "test/test_rake_require.rb" "test/test_rake_rules.rb" "test/test_rake_scope.rb" "test/test_rake_task.rb" "test/test_rake_task_argument_parsing.rb" "test/test_rake_task_arguments.rb" "test/test_rake_task_manager.rb" "test/test_rake_task_manager_argument_resolution.rb" "test/test_rake_task_with_arguments.rb" "test/test_rake_test_task.rb" "test/test_rake_thread_pool.rb" "test/test_rake_top_level_functions.rb" "test/test_rake_win32.rb" "test/test_thread_history_display.rb" "test/test_trace_output.rb" ]
/home/terceiro/src/rake/lib/rake/testtask.rb:129:in `block (3 levels) in define'
/home/terceiro/src/rake/lib/rake/testtask.rb:116:in `block (2 levels) in define'
/home/terceiro/src/rake/lib/rake/testtask.rb:110:in `block in define'
Tasks: TOP => default => test
(See full trace by running task with --trace)

Other seed values will guarantee a pass, e.g.

$ rake SEED=2118
/usr/bin/ruby2.3 -w -I"lib:test" -I"/home/terceiro/src/rake/lib" "/home/terceiro/src/rake/lib/rake/rake_test_loader.rb" "test/test_private_reader.rb" "test/test_rake.rb" "test/test_rake_application.rb" "test/test_rake_application_options.rb" "test/test_rake_backtrace.rb" "test/test_rake_clean.rb" "test/test_rake_cpu_counter.rb" "test/test_rake_definitions.rb" "test/test_rake_directory_task.rb" "test/test_rake_dsl.rb" "test/test_rake_early_time.rb" "test/test_rake_extension.rb" "test/test_rake_file_creation_task.rb" "test/test_rake_file_list.rb" "test/test_rake_file_list_path_map.rb" "test/test_rake_file_task.rb" "test/test_rake_file_utils.rb" "test/test_rake_functional.rb" "test/test_rake_invocation_chain.rb" "test/test_rake_late_time.rb" "test/test_rake_linked_list.rb" "test/test_rake_makefile_loader.rb" "test/test_rake_multi_task.rb" "test/test_rake_name_space.rb" "test/test_rake_package_task.rb" "test/test_rake_path_map.rb" "test/test_rake_path_map_explode.rb" "test/test_rake_path_map_partial.rb" "test/test_rake_pseudo_status.rb" "test/test_rake_rake_test_loader.rb" "test/test_rake_reduce_compat.rb" "test/test_rake_require.rb" "test/test_rake_rules.rb" "test/test_rake_scope.rb" "test/test_rake_task.rb" "test/test_rake_task_argument_parsing.rb" "test/test_rake_task_arguments.rb" "test/test_rake_task_manager.rb" "test/test_rake_task_manager_argument_resolution.rb" "test/test_rake_task_with_arguments.rb" "test/test_rake_test_task.rb" "test/test_rake_thread_pool.rb" "test/test_rake_top_level_functions.rb" "test/test_rake_win32.rb" "test/test_thread_history_display.rb" "test/test_trace_output.rb" 
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's default settings.
Run options: --seed 2118

# Running:

................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Finished in 13.767306s, 41.8383 runs/s, 114.9826 assertions/s.

576 runs, 1583 assertions, 0 failures, 0 errors, 0 skips
/usr/lib/ruby/vendor_ruby/simplecov.rb:56: warning: instance variable @track_files_glob not initialized
[Coveralls] Outside the Travis environment, not sending data.
@terceiro
Copy link
Author

Hm, nevermind. I can't reproduce this in a clean environment.

@PPed72
Copy link

PPed72 commented Aug 14, 2017

I have the same problem and I can (more or less) reproduce it.
SEED=2118 and 2389 work consistently on rake-10.5.0, rake-11.2.2, rake-11.3.0 and rake-12.0.0, SEED=30397 works up to 11.2.2, others usually do not work. This is on multiple machines (same OS, though) and has survived several reboots. Is it possible to reopen the issue to investigate some more, or should I open a new one?

@hsbt hsbt reopened this Aug 14, 2017
@hsbt
Copy link
Member

hsbt commented Aug 14, 2017

@PPed72 I reopened. Can you show your results?

@hsbt
Copy link
Member

hsbt commented Nov 10, 2017

I couldn't reproduce this with 5 times.

@hsbt hsbt closed this as completed Nov 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hsbt @terceiro @PPed72