rake test:units breaks _after_ doing the tests #4145

Closed
Rhywden opened this Issue Dec 23, 2011 · 15 comments

Projects

None yet
@Rhywden
Rhywden commented Dec 23, 2011

This happens on both Windows and Linx, with both 1.9.2p290 and 1.9.3p0 on Rails 3.1.3 with both the "minitest" and the "unit-test" gem.

I created a nearly empty project, save for a "Page" model with a "name" string column.

The following code in page_test.rb will yield no errors:

require 'test_helper'

class PageTest < ActiveSupport::TestCase
  test 'page must have a name' do
    assert true
  end
end

The following code will break rake test:units after having run the test:

require 'test_helper'

class PageTest < ActiveSupport::TestCase
  test 'page must have a name' do
    page = Page.new
    assert !page.save
  end
end

The output will be as follows (using rake test:units --trace):

** Invoke test:units (first_time)
** Invoke test:prepare (first_time)
** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:abort_if_pending_migrations
** Execute db:test:prepare
** Invoke db:test:load (first_time)
** Invoke db:test:purge (first_time)
** Invoke environment 
** Execute db:test:purge
** Execute db:test:load
** Invoke db:schema:load (first_time)
** Invoke environment 
** Execute db:schema:load
** Execute test:prepare
** Execute test:units
Loaded suite /usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_test_loader
Started

PageTest:
 FAIL page must have a name (0.73s) 
      Failed assertion, no message given.
      /home/rhywden/RubymineProjects/boerssum/test/unit/page_test.rb:6:in `block in <class:PageTest>'


Finished in 0.735883 seconds.

1 tests, 1 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1): [/usr/local/bin/ruby -I"lib:test" "/usr/loc...]
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/file_utils.rb:53:in `block in create_shell_runner'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/file_utils.rb:45:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/file_utils.rb:45:in `sh'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/file_utils_ext.rb:39:in `sh'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/file_utils.rb:82:in `ruby'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/file_utils_ext.rb:39:in `ruby'
/usr/local/lib/ruby/gems/1.9.1/gems/railties-3.1.3/lib/rails/test_unit/testing.rake:26:in `block (2 levels) in define'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/file_utils_ext.rb:60:in `verbose'
/usr/local/lib/ruby/gems/1.9.1/gems/railties-3.1.3/lib/rails/test_unit/testing.rake:11:in `block in define'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block (2 levels) in top_level'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block in top_level'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `block in run'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/usr/local/lib/ruby/gems/1.9.1/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/usr/local/bin/rake:19:in `load'
/usr/local/bin/rake:19:in `<main>'
Tasks: TOP => test:units

The error given under Windows looks a bit different. Still fails, though (same code, same gems):

PageHelperTest
PageTest
     test_page_should_not_save_without_a_name                             FAIL
        Failed assertion, no message given.
        Assertion at C:/Ruby192/lib/ruby/gems/1.9.1/gems/activesupport-3.1.3/lib/active_support/testing/setup_and_teardown.rb:35:in `block in run'
Test::Unit::TestCase
==============================================================================
  pass: 0,  fail: 1,  error: 0
  total: 1 tests with 1 assertions in 0.491 seconds
==============================================================================
rake aborted!
Command failed with status (1): [C:/Ruby192/bin/ruby.exe -I"lib;test" "C:/R...]
C:/Ruby192/lib/ruby/1.9.1/rake.rb:993:in `block in sh'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:1008:in `call'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:1008:in `sh'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:1092:in `sh'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:1027:in `ruby'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:1092:in `ruby'
C:/Ruby192/lib/ruby/gems/1.9.1/gems/railties-3.1.3/lib/rails/test_unit/testing.rake:26:in `block (2 levels) in define'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:1110:in `verbose'
C:/Ruby192/lib/ruby/gems/1.9.1/gems/railties-3.1.3/lib/rails/test_unit/testing.rake:11:in `block in define'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:634:in `call'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:634:in `block in execute'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:629:in `each'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:629:in `execute'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:595:in `block in invoke_with_call_chain'
C:/Ruby192/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:588:in `invoke_with_call_chain'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:581:in `invoke'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:2041:in `invoke_task'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:2019:in `block (2 levels) in top_level'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:2019:in `each'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:2019:in `block in top_level'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:2058:in `standard_exception_handling'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:2013:in `top_level'
C:/Ruby192/lib/ruby/1.9.1/rake.rb:1992:in `run'
C:/Ruby192/bin/rake:31:in `<main>'

This also means that test integration into IDEs like Rubymine will NOT work as well which is highly annyoing.

@nfm
nfm commented Dec 23, 2011

This is intended behavior - it's just the way rake works.

Rake invokes a task (running a test) which fails, and that propagates upwards, so that you get a non-zero exit code from rake. This behavior also prevents other tasks from running. So if you have a 3 step rake task that fails on step 2, step 3 will not be run.

I haven't used Rubymine but rake behaving like this shouldn't be a problem. Do you have a specific problem caused by this behavior?

@Rhywden
Rhywden commented Dec 23, 2011

Please tell me you're not serious. First of all, this behaviour is mentioned nowhere - maybe there should be a hint of this being "intended" in, oh, the official documents?
I mean, just look at http://guides.rubyonrails.org/testing.html#unit-testing-your-models

There's NO mention of a line like Assertion at C:/Ruby192/lib/ruby/gems/1.9.1/gems/activesupport-3.1.3/lib/active_support/testing/setup_and_teardown.rb:35:in 'block in run' after this:

1) Failure:
test_should_not_save_post_without_title(PostTest) [/test/unit/post_test.rb:6]:
<false> is not true.

1 tests, 1 assertions, 1 failures, 0 errors

That's what it's supposed to look like, according to the docs. There's also NO mention of rake exploding after that.

Secondly, I'm pretty sure that there is some way to stop execution of a chain of commands in a proper manner - instead of letting it simply look like your application just exploded.
Thirdly, it seems as if the Rubymine IDE does not look stuff exploding in this way either - since that way I only get a list of whatever broke first. And not ALL the errors. I don't like this piecemeal approach, to be honest.
Fourthly, when I was using Rails 2, I also don't remember exploding unit tests.

@dgm
dgm commented Dec 24, 2011

I haven't started to use rails 3.1 yet, but under rails 3 it doesn't explode like that at the end. It should only show the tests that fail, or if the app has a serious error, it may show the stack trace from the error, but this isn't an error in the app.

@nfm
nfm commented Dec 24, 2011

@Rhywden Yep, I am serious. Rake isn't breaking or exploding - this is just how it operates. This isn't specific to Rails.

You can --trace back to the rake task that aborted. This is useful for debugging. The output about what task failed won't cause you any harm. If you don't want the extra info, skip the --trace flag.

Rake will run all of your tests and than exit with status code 1 if any of them failed. If you have other rake tasks that depend on test:units, they won't be invoked if any tests failed. This is the correct behavior.

@freyr
freyr commented Jan 19, 2012

this is done by gem called "turn" (pretty print test results) if you disable it, everything gonna be fine :D

@mrampton

Have to say I agree with Rhywden -- just spent 40 minutes trying to figure out what was causing this.

@carlosantoniodasilva

Hey, have you guys been able to test this against latest Rails 3.2? I'd love some feedback on this, to see if it's working or the issue just got stale, thanks!

@mrampton
mrampton commented May 6, 2012

Looks good on my end! Awesome!

@carlosantoniodasilva

@mrampton ok, great, thanks for reporting back.

I'm closing the issue for now, but if someone still find this is an issue, please let me know and I'll gladly reopen. Thanks!

@mrichman

I have this issue with Rails 3.2.3 and Ruby 1.9.3p194 on OS X Lion. Running rake test:functionals --trace gives:

1 tests, 1 assertions, 1 failures, 0 errors, 0 skips
rake aborted!
Command failed with status (1): [/Users/mark/.rvm/rubies/ruby-1.9.3-p194/bi...]
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/file_utils.rb:53:in `block in create_shell_runner'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/file_utils.rb:45:in `call'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/file_utils.rb:45:in `sh'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/file_utils_ext.rb:39:in `sh'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/file_utils.rb:82:in `ruby'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/file_utils_ext.rb:39:in `ruby'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/testtask.rb:99:in `block (2 levels) in define'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/file_utils_ext.rb:60:in `verbose'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/testtask.rb:98:in `block in define'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
/Users/mark/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block (2 levels) in top_level'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block in top_level'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `block in run'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/bin/rake:19:in `load'
/Users/mark/.rvm/gems/ruby-1.9.3-p194@global/bin/rake:19:in `<main>'
Tasks: TOP => test:functionals

I am NOT using the turn gem. The only test-related gems in my Gemfile are ZenTest and autotest-rails. Perhaps those use turn.

@hminaya
hminaya commented Apr 3, 2013

I'm having the same issue with Rails 3.2.11 and ruby 1.9.3p194 on OSX 10.8.3

@reagleton

I'm having a similar issue with rake test:units using Rails 3.2.13 and ruby 1.9.3p392. My output is mainly different with respect to the status code returned.

24 tests, 46 assertions, 3 failures, 0 errors, 2 skips
rake aborted!
Command failed with status (3): [ruby -I"lib:test" -I"/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib" "/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/rake_test_loader.rb" "test/unit/**/*_test.rb" ]
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/testtask.rb:104:in `block (3 levels) in define'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/file_utils.rb:45:in `call'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/file_utils.rb:45:in `sh'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/file_utils_ext.rb:37:in `sh'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/file_utils.rb:82:in `ruby'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/file_utils_ext.rb:37:in `ruby'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/testtask.rb:100:in `block (2 levels) in define'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/file_utils_ext.rb:58:in `verbose'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/testtask.rb:98:in `block in define'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/task.rb:246:in `call'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/task.rb:246:in `block in execute'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/task.rb:241:in `each'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/task.rb:184:in `block in invoke_with_call_chain'
/home/reagleton/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:101:in `block (2 levels) in top_level'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:101:in `each'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:101:in `block in top_level'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level' 
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:73:in `block in run'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/gems/rake-10.0.4/bin/rake:33:in `<top (required)>'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/bin/rake:19:in `load'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/bin/rake:19:in `<main>'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/bin/ruby_noexec_wrapper:14:in `eval'
/home/reagleton/.rvm/gems/ruby-1.9.3-p392@cms_rails3/bin/ruby_noexec_wrapper:14:in `<main>'
Tasks: TOP => test:units

I have to agree with the people before me that this is poor error handling in many ways. I'm not sure whether it needs to be addressed in the test scripts or in rake but failing tests are not the same as an exception occurring in code. A failing test should output the results gracefully (which it does) and then exit with a non-zero status code (that is also fine and commonly done). But Rake shouldn't explode and die just because it gets a non-zero status code. It should handle that much more gracefully.

@anup-pareek

Did anyone got solution for this ? I am facing the same issue without using the turn gem.
I am using the ruby 1.8.7 with rails 2.3.18

@ilyakogan

I was having the same issue, but it seems to be solved by error catching as described here: https://benbiddington.wordpress.com/2009/05/16/rake-shell-task-failure/

@stratigos

@nfm tysm for this. Ive been scouring old threads on ruby-forum.com for a while now and couldnt find an answer. After reading your comment, I just went and changed all my tests to assert true and that worrying stack trace disappeared!

It is some very interesting behavior for Rake. I thought I broke something in my local dev environment, but I guess this is "working as expected." So now I feel I can ignore the stack trace and exit code from Rake until my tests are passing. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment