New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Test Runner's Minitest integration. #19571

Merged
merged 1 commit into from Jun 8, 2015

Conversation

Projects
None yet
9 participants
@kaspth
Member

kaspth commented Mar 28, 2015

I've tried to improve the Rails new test running system to take better advantage of Minitest's extension system.

The build will break as I haven't ported over the Runner tests yet. But the tests running the tests commands should pass.

@zenspider, was this more inline with your comments on the original pull request?

cc @senny @arthurnn

@kaspth kaspth added the railties label Mar 28, 2015

@kaspth kaspth added this to the 5.0.0 milestone Mar 28, 2015

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Apr 1, 2015

Member

@kaspth Thank you for working on this 💛

Can we make it work without Minitest.autorun? Part of the idea of the original runner was to have a simple interface to run a bunch of tests:

Rails::TestRunner.run(["test/controllers", "test/mailers", "test/functional"])

When relying on Minitest.autorun the API feels unclear and relies on the require side-effects. There is a related issue #17708 and PR #19435.

Member

senny commented Apr 1, 2015

@kaspth Thank you for working on this 💛

Can we make it work without Minitest.autorun? Part of the idea of the original runner was to have a simple interface to run a bunch of tests:

Rails::TestRunner.run(["test/controllers", "test/mailers", "test/functional"])

When relying on Minitest.autorun the API feels unclear and relies on the require side-effects. There is a related issue #17708 and PR #19435.

Show outdated Hide outdated railties/lib/rails/test_unit/minitest_plugin.rb
opts.on("-n", "--name [FILENAME:LINE]",
"Run a single test by appending the line number to filename:\n\n" \
"bin/rails test test/models/user_test.rb:27") do |name|

This comment has been minimized.

@senny

senny Apr 1, 2015

Member

I'm a bit confused by this option. Isn't -n supposed to name the method to run? Also the example does not use the flag at all. This seems to be more an explanation of the [file or directory] argument at the top.

@senny

senny Apr 1, 2015

Member

I'm a bit confused by this option. Isn't -n supposed to name the method to run? Also the example does not use the flag at all. This seems to be more an explanation of the [file or directory] argument at the top.

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Apr 1, 2015

Member

I still think we should have a TestRunner class. maybe that class wont parse the options anymore. But it would be good to have the test runner logic in one class.

Member

arthurnn commented Apr 1, 2015

I still think we should have a TestRunner class. maybe that class wont parse the options anymore. But it would be good to have the test runner logic in one class.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 6, 2015

Member

Alright, updated here @senny @arthurnn

Member

kaspth commented Apr 6, 2015

Alright, updated here @senny @arthurnn

Show outdated Hide outdated railties/test/application/test_runner_test.rb
@@ -14,7 +15,7 @@ def teardown
teardown_app
end
def test_run_in_test_environment
def test_run_in_differing_environments

This comment has been minimized.

@senny

senny Apr 6, 2015

Member

@kaspth do you think it's worth keep the other test around to be sure it runs in test env by default?

@senny

senny Apr 6, 2015

Member

@kaspth do you think it's worth keep the other test around to be sure it runs in test env by default?

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

I must admit the setup line length got me to consolidate the tests. I do think you're right about keeping the old method, so we get a better test name, 👍

@kaspth

kaspth Apr 11, 2015

Member

I must admit the setup line length got me to consolidate the tests. I do think you're right about keeping the old method, so we get a better test name, 👍

Show outdated Hide outdated railties/test/application/test_runner_test.rb
end
end
def test_run_multiple_files_and_run_one_file_by_line_only_runs_filter

This comment has been minimized.

@senny

senny Apr 6, 2015

Member

Is this the behavior we want or a limitation by our current integration with minitest?

@senny

senny Apr 6, 2015

Member

Is this the behavior we want or a limitation by our current integration with minitest?

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

this is a limitation of Minitest atm, that only allows you to pass one filter.
I have a semi-patch for Minitest to solve that. Until we send the PR and gets merged, thats the behaviour we can have.

@arthurnn

arthurnn Apr 7, 2015

Member

this is a limitation of Minitest atm, that only allows you to pass one filter.
I have a semi-patch for Minitest to solve that. Until we send the PR and gets merged, thats the behaviour we can have.

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

Sorry, what I said is not true.
We can implement a filter that accept files and lines, or just files. see that Minitest uses the filter ===.
https://github.com/seattlerb/minitest/blob/master/lib/minitest.rb#L289

So our filter can hold a chain of files:line and we accept multiple results.

@arthurnn

arthurnn Apr 7, 2015

Member

Sorry, what I said is not true.
We can implement a filter that accept files and lines, or just files. see that Minitest uses the filter ===.
https://github.com/seattlerb/minitest/blob/master/lib/minitest.rb#L289

So our filter can hold a chain of files:line and we accept multiple results.

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

@arthurnn Right, I was thinking about adding a chain. Do we want to support passing several line filters?

@kaspth

kaspth Apr 11, 2015

Member

@arthurnn Right, I was thinking about adding a chain. Do we want to support passing several line filters?

This comment has been minimized.

@senny

senny Apr 11, 2015

Member

@kaspth I'm not sure we need to support that. What got my attention is that these tests specify behavior that is caused by the current implementation not because we designed it like that. I think the behavior is surprising, you can pass arguments that are not at all considered. It would be better to either warn the user when that happens or get it working.

@senny

senny Apr 11, 2015

Member

@kaspth I'm not sure we need to support that. What got my attention is that these tests specify behavior that is caused by the current implementation not because we designed it like that. I think the behavior is surprising, you can pass arguments that are not at all considered. It would be better to either warn the user when that happens or get it working.

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

It was actually easier to just implement a FilterChain to let us run more line filters and augment Minitest's -n option.

@kaspth

kaspth Apr 11, 2015

Member

It was actually easier to just implement a FilterChain to let us run more line filters and augment Minitest's -n option.

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

But I just realized that doesn't solve your original concern, @senny. I'll try working on it some more.

@kaspth

kaspth Apr 11, 2015

Member

But I just realized that doesn't solve your original concern, @senny. I'll try working on it some more.

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Apr 6, 2015

Member

@kaspth really liking this patch 💛 Can you post the help output when running with -h?

Member

senny commented Apr 6, 2015

@kaspth really liking this patch 💛 Can you post the help output when running with -h?

@zenspider

This comment has been minimized.

Show comment
Hide comment
@zenspider

zenspider Apr 7, 2015

Contributor

@senny it already does.

Contributor

zenspider commented Apr 7, 2015

@senny it already does.

Show outdated Hide outdated railties/lib/rails/test_unit/testing.rake
ARGV.shift if ARGV[0] == "test"
Rails::TestRunner.run

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

why the runner dont receive the args?
I remember I initially had the ARGV.shift too, but I think, it didnt work when you do: bundle exec rake test db:migrate , dont fully remember why.

@arthurnn

arthurnn Apr 7, 2015

Member

why the runner dont receive the args?
I remember I initially had the ARGV.shift too, but I think, it didnt work when you do: bundle exec rake test db:migrate , dont fully remember why.

This comment has been minimized.

@zenspider

zenspider Apr 7, 2015

Contributor

By the time a rake task is run, ARGV is out of the picture. That's how that task got run in the first place.

@zenspider

zenspider Apr 7, 2015

Contributor

By the time a rake task is run, ARGV is out of the picture. That's how that task got run in the first place.

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

@zenspider Right, so as long as rake is the funnel people will have to keep using TEST and TESTOPTS.

@kaspth

kaspth Apr 11, 2015

Member

@zenspider Right, so as long as rake is the funnel people will have to keep using TEST and TESTOPTS.

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

I guess we could modify a Rake::TestTask to run something like ruby -Ilib:test -r"rails/test_unit/minitest_plugin" whatever.rb

@kaspth

kaspth Apr 11, 2015

Member

I guess we could modify a Rake::TestTask to run something like ruby -Ilib:test -r"rails/test_unit/minitest_plugin" whatever.rb

@@ -284,6 +354,18 @@ def test_fixture
RUBY
end
def create_backtrace_test

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

the output of the tests that check backtrace look the same when backtrace is enable or disabled. That looks weird IMO. We should have different outputs I think.

@arthurnn

arthurnn Apr 7, 2015

Member

the output of the tests that check backtrace look the same when backtrace is enable or disabled. That looks weird IMO. We should have different outputs I think.

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

Right, I can add a failure message to the assert_kind_of and match on that.

@kaspth

kaspth Apr 11, 2015

Member

Right, I can add a failure message to the assert_kind_of and match on that.

Show outdated Hide outdated railties/test/application/test_runner_test.rb
end
end
end
private

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

should we also test the cases where tests fail, and the right output is there?

@arthurnn

arthurnn Apr 7, 2015

Member

should we also test the cases where tests fail, and the right output is there?

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

The way I see it, the happy path testing is to make sure we're correctly funneling to Minitest. Thus adding tests for more than our extensions would be testing Minitest.

@kaspth

kaspth Apr 11, 2015

Member

The way I see it, the happy path testing is to make sure we're correctly funneling to Minitest. Thus adding tests for more than our extensions would be testing Minitest.

Show outdated Hide outdated activesupport/lib/active_support/test_case.rb
@@ -45,6 +45,41 @@ def test_order
test_order
end
def run(reporter, options = {})
if filter = line_filter_for(options[:patterns] || [])

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

why do we need this method here, and not in the Runner?
If we had this method in the runner, we could called it from the rails_minitest_plugin.
And tests that only extend Minitest::Test would also work.

@arthurnn

arthurnn Apr 7, 2015

Member

why do we need this method here, and not in the Runner?
If we had this method in the runner, we could called it from the rails_minitest_plugin.
And tests that only extend Minitest::Test would also work.

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

We need to check which runnable implements a test - we don't have access to the runnable in rails_minitest_plugin.

Also adding it to Runner would make Active Support depend on Railties - this way stays clear of that.

@kaspth

kaspth Apr 11, 2015

Member

We need to check which runnable implements a test - we don't have access to the runnable in rails_minitest_plugin.

Also adding it to Runner would make Active Support depend on Railties - this way stays clear of that.

Show outdated Hide outdated railties/lib/rails/commands/test.rb
@@ -2,4 +2,4 @@
$: << File.expand_path("../../test", APP_PATH)
Rails::TestRunner.run(ARGV)
Minitest.autorun

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

why do we call the autorun here and not the Rails runner?

@arthurnn

arthurnn Apr 7, 2015

Member

why do we call the autorun here and not the Rails runner?

This comment has been minimized.

@senny

senny Apr 22, 2015

Member

regarding the naming discussion TestRequirer and TestRunner. I'd rather opt for something, which does not rely on Minitest.autorun.

@kaspth do you see any reasons not to go down the path of explicitly calling Minitest.run when we want it to run? Also clearing in between runs, making it possible to run different sets of tests in the same process?

@senny

senny Apr 22, 2015

Member

regarding the naming discussion TestRequirer and TestRunner. I'd rather opt for something, which does not rely on Minitest.autorun.

@kaspth do you see any reasons not to go down the path of explicitly calling Minitest.run when we want it to run? Also clearing in between runs, making it possible to run different sets of tests in the same process?

Show outdated Hide outdated railties/lib/rails/test_unit/minitest_plugin.rb
ENV["RAILS_ENV"] = options[:environment] || "test"
options[:patterns] = OptionParser.new(options[:args]).order!
Rails::TestRunner.require_files options[:patterns]

This comment has been minimized.

@arthurnn

arthurnn Apr 7, 2015

Member

I just realized we have this require call here, so do we need the TestRunner.run method at all?

@arthurnn

arthurnn Apr 7, 2015

Member

I just realized we have this require call here, so do we need the TestRunner.run method at all?

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Apr 7, 2015

Member

I like where this is going too.. Thanks @kaspth

Member

arthurnn commented Apr 7, 2015

I like where this is going too.. Thanks @kaspth

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Apr 7, 2015

Member

@zenspider I meant to post the output here to GitHub.

Member

senny commented Apr 7, 2015

@zenspider I meant to post the output here to GitHub.

@zenspider

This comment has been minimized.

Show comment
Hide comment
@zenspider

zenspider Apr 7, 2015

Contributor

"Runner" == manager class. That's what MT4 had and why I explicitly did a rewrite for MT5. Manager classes are bad design and should be avoided where possible. I explicitly designed test classes (runnables, really) to be able to run themselves on purpose. This allows for different types of runnables (eg Test vs Benchmark). Having a test class that knows how to run its own tests makes sense. It's what tests do: run, so why externalize the functionality of it?

Avoid a "Runner" type class. You'll be better off in the long run.

Contributor

zenspider commented Apr 7, 2015

"Runner" == manager class. That's what MT4 had and why I explicitly did a rewrite for MT5. Manager classes are bad design and should be avoided where possible. I explicitly designed test classes (runnables, really) to be able to run themselves on purpose. This allows for different types of runnables (eg Test vs Benchmark). Having a test class that knows how to run its own tests makes sense. It's what tests do: run, so why externalize the functionality of it?

Avoid a "Runner" type class. You'll be better off in the long run.

@zenspider

This comment has been minimized.

Show comment
Hide comment
@zenspider

zenspider Apr 7, 2015

Contributor

@kaspth Sorry, I missed your original question: you've removed 75 lines, of course I'm a fan.

Contributor

zenspider commented Apr 7, 2015

@kaspth Sorry, I missed your original question: you've removed 75 lines, of course I'm a fan.

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Apr 7, 2015

Member

@zenspider I can see your Runner-argument for Minitest itself but this runner does nothing else than requiring the files. Someone needs to require the tests before they can run themselves. I'd like to have an interface in Rails to run a set of files and folders. The rest boils down to wether Runner is a good name for that.

Member

senny commented Apr 7, 2015

@zenspider I can see your Runner-argument for Minitest itself but this runner does nothing else than requiring the files. Someone needs to require the tests before they can run themselves. I'd like to have an interface in Rails to run a set of files and folders. The rest boils down to wether Runner is a good name for that.

@arthurnn arthurnn self-assigned this Apr 8, 2015

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 11, 2015

Member

@zenspider Great to hear and, yes, I agree on removing the need for a runner and just use Minitest's autorun or run where appropriate.

@senny I'd be happy to go back to the TestRequirer name instead of TestRunner.

Member

kaspth commented Apr 11, 2015

@zenspider Great to hear and, yes, I agree on removing the need for a runner and just use Minitest's autorun or run where appropriate.

@senny I'd be happy to go back to the TestRequirer name instead of TestRunner.

Show outdated Hide outdated activesupport/lib/active_support/testing/filter_chain.rb
private
def derive_regexp(filter)
filter =~ %r%/(.*)/% ? Regexp.new($1) : filter

This comment has been minimized.

@kaspth

kaspth Apr 11, 2015

Member

If we didn't duplicate some logic from Minitest here, passing a -n /hello/ filter won't filter correctly.

@kaspth

kaspth Apr 11, 2015

Member

If we didn't duplicate some logic from Minitest here, passing a -n /hello/ filter won't filter correctly.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 11, 2015

Member

@zenspider Writing the FilterChain is a bit cumbersome because we kind of have to hack Minitest. Would you consider letting others extend Minitest's filter in the same way reporters can?

Member

kaspth commented Apr 11, 2015

@zenspider Writing the FilterChain is a bit cumbersome because we kind of have to hack Minitest. Would you consider letting others extend Minitest's filter in the same way reporters can?

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 12, 2015

Member

@senny I forgot to post the help output. Here it is:

minitest options:
    -h, --help                        Display this help.
    -s, --seed SEED                   Sets random seed
    -v, --verbose                     Verbose. Show progress processing files.
    -n, --name PATTERN                Filter run on /pattern/ or string.

Known extensions: rails
Usage: bin/rails test [options] [file or directory]
You can run a single test by appending the line number to filename:

bin/rails test test/models/user_test.rb:27
    -e, --environment [ENV]           Run tests in the ENV environment
    -b, --backtrace                   Show the complete backtrace

We should probably add more separators - currently it looks a bit cramped.

Member

kaspth commented Apr 12, 2015

@senny I forgot to post the help output. Here it is:

minitest options:
    -h, --help                        Display this help.
    -s, --seed SEED                   Sets random seed
    -v, --verbose                     Verbose. Show progress processing files.
    -n, --name PATTERN                Filter run on /pattern/ or string.

Known extensions: rails
Usage: bin/rails test [options] [file or directory]
You can run a single test by appending the line number to filename:

bin/rails test test/models/user_test.rb:27
    -e, --environment [ENV]           Run tests in the ENV environment
    -b, --backtrace                   Show the complete backtrace

We should probably add more separators - currently it looks a bit cramped.

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Apr 13, 2015

Member

@kaspth I agree, the output is pretty dense at the moment. Also, the usage (Usage: bin/rails test [options] [file or directory]) needs adjusting that it's possible to pass multiple files or directories:

Member

senny commented Apr 13, 2015

@kaspth I agree, the output is pretty dense at the moment. Also, the usage (Usage: bin/rails test [options] [file or directory]) needs adjusting that it's possible to pass multiple files or directories:

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 13, 2015

Member

@senny I've tweaked the output to this:

minitest options:
    -h, --help                       Display this help.
    -s, --seed SEED                  Sets random seed
    -v, --verbose                    Verbose. Show progress processing files.
    -n, --name PATTERN               Filter run on /pattern/ or string.

Known extensions: rails

Usage: bin/rails test [options] [files or directories]
You can run a single test in file by appending the line number:

    bin/rails test test/models/user_test.rb:27

Rails options:
    -e, --environment [ENV]          Run tests in the ENV environment
    -b, --backtrace                  Show the complete backtrace
Member

kaspth commented Apr 13, 2015

@senny I've tweaked the output to this:

minitest options:
    -h, --help                       Display this help.
    -s, --seed SEED                  Sets random seed
    -v, --verbose                    Verbose. Show progress processing files.
    -n, --name PATTERN               Filter run on /pattern/ or string.

Known extensions: rails

Usage: bin/rails test [options] [files or directories]
You can run a single test in file by appending the line number:

    bin/rails test test/models/user_test.rb:27

Rails options:
    -e, --environment [ENV]          Run tests in the ENV environment
    -b, --backtrace                  Show the complete backtrace
@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Apr 13, 2015

Member

@kaspth looking better already. What do you think about a section similar to the one about the line number but describing to run multiple directories. Something along the lines of:

Usage: bin/rails test [options] [files or directories]

You can run a single test in file by appending the line number:

    bin/rails test test/models/user_test.rb:27

You can run multiple files and directories at the same time by passing multiple arguments:

    bin/rails test test/models test/controllers test/integration/login_test.rb

Rails options:
    -e, --environment [ENV]          Run tests in the ENV environment
    -b, --backtrace                  Show the complete backtrace
Member

senny commented Apr 13, 2015

@kaspth looking better already. What do you think about a section similar to the one about the line number but describing to run multiple directories. Something along the lines of:

Usage: bin/rails test [options] [files or directories]

You can run a single test in file by appending the line number:

    bin/rails test test/models/user_test.rb:27

You can run multiple files and directories at the same time by passing multiple arguments:

    bin/rails test test/models test/controllers test/integration/login_test.rb

Rails options:
    -e, --environment [ENV]          Run tests in the ENV environment
    -b, --backtrace                  Show the complete backtrace
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 13, 2015

Member

@senny Yeah, that's great. I've added it.

Member

kaspth commented Apr 13, 2015

@senny Yeah, that's great. I've added it.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 19, 2015

Member

@senny @arthurnn Alright, I've added the ability to mix files and line filters as people please.

Something crazy like this should work:

test/controllers test/models/account_test.rb test/models/post_test.rb:4

Potentially we could output the failures on a single line, so you could copy that and run all the failing tests at once. But I don't know how useful that is.

I've also changed the code to not augment Minitest when there's no line filters.

However, when there's at least one line filter, we need to create filters for every file. If we didn't, running tests like:

test/models/account_test.rb test/models/post_test.rb:4

would only run the filter with the line on it. I feel like we should be able to avoid creating those extra filters, but I haven't been able to figure it out.

Member

kaspth commented Apr 19, 2015

@senny @arthurnn Alright, I've added the ability to mix files and line filters as people please.

Something crazy like this should work:

test/controllers test/models/account_test.rb test/models/post_test.rb:4

Potentially we could output the failures on a single line, so you could copy that and run all the failing tests at once. But I don't know how useful that is.

I've also changed the code to not augment Minitest when there's no line filters.

However, when there's at least one line filter, we need to create filters for every file. If we didn't, running tests like:

test/models/account_test.rb test/models/post_test.rb:4

would only run the filter with the line on it. I feel like we should be able to avoid creating those extra filters, but I haven't been able to figure it out.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Apr 19, 2015

Member

I really don't understand the two test failures in Railties: https://travis-ci.org/rails/rails/builds/59126781. When I inspect the files required in those tests, they require the correct scaffolded files. And they require test_helper which requires rails/test_help which then calls Minitest.autorun.

Could ARGV being long gone in the rake test task be a factor in this?

Member

kaspth commented Apr 19, 2015

I really don't understand the two test failures in Railties: https://travis-ci.org/rails/rails/builds/59126781. When I inspect the files required in those tests, they require the correct scaffolded files. And they require test_helper which requires rails/test_help which then calls Minitest.autorun.

Could ARGV being long gone in the rake test task be a factor in this?

Show outdated Hide outdated activesupport/lib/active_support/test_case.rb
@@ -45,6 +46,15 @@ def test_order
test_order
end
def run(reporter, options = {})
if options[:patterns] && options[:patterns].any? { |p| p.include?(':') }

This comment has been minimized.

@senny

senny Apr 22, 2015

Member

Regarding the comment about test-failures:

My guess is that options[:patterns] here is ["db:migrate", "test"]. This does include a : resulting in a CompositeFilter being added.

@senny

senny Apr 22, 2015

Member

Regarding the comment about test-failures:

My guess is that options[:patterns] here is ["db:migrate", "test"]. This does include a : resulting in a CompositeFilter being added.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth May 4, 2015

Member

@senny @arthurnn Alright, I think I got this now 😄

I've moved away from autorun and fixed the errors in rake_test.rb. What should we do about active_support/testing/autorun? Can we outright remove it or do we have to deprecate?

Member

kaspth commented May 4, 2015

@senny @arthurnn Alright, I think I got this now 😄

I've moved away from autorun and fixed the errors in rake_test.rb. What should we do about active_support/testing/autorun? Can we outright remove it or do we have to deprecate?

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth May 4, 2015

Member

Hm, now the generator tests are failing: https://travis-ci.org/rails/rails/jobs/61201646.

I'll look into them tomorrow (might be related to the lingering autorun @senny spotted).

Member

kaspth commented May 4, 2015

Hm, now the generator tests are failing: https://travis-ci.org/rails/rails/jobs/61201646.

I'll look into them tomorrow (might be related to the lingering autorun @senny spotted).

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny May 4, 2015

Member

@kaspth 👍 ping me and I'll take another look. Really happy to see this coming along. Great work 💛

Member

senny commented May 4, 2015

@kaspth 👍 ping me and I'll take another look. Really happy to see this coming along. Great work 💛

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth May 5, 2015

Member

@senny I've been trying to debug why the generator tests are failing, but so far I've gone nowhere.

It seems the tests aren't run at all.

The generator generates an engine with a Rakefile with a standard rake test task. Which leaves me befuddled as to why the tests aren't run. Do you have any ideas?

Member

kaspth commented May 5, 2015

@senny I've been trying to debug why the generator tests are failing, but so far I've gone nowhere.

It seems the tests aren't run at all.

The generator generates an engine with a Rakefile with a standard rake test task. Which leaves me befuddled as to why the tests aren't run. Do you have any ideas?

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny May 6, 2015

Member

@kaspth I can probably take a look later today.

Member

senny commented May 6, 2015

@kaspth I can probably take a look later today.

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny May 6, 2015

Member

@kaspth investigated a bit. The issue is the removed autorun line in test_help.rb. As these tests are run using Rake::TestTask's they expect that autorun will be required at some point.

Member

senny commented May 6, 2015

@kaspth investigated a bit. The issue is the removed autorun line in test_help.rb. As these tests are run using Rake::TestTask's they expect that autorun will be required at some point.

@zamith

This comment has been minimized.

Show comment
Hide comment
@zamith

zamith May 7, 2015

Contributor

Hey guys,

First of all let me say that it is great to finally have a runner bundled with rails that can compete with the one from RSpec.

Also, as the maintainer of m, I feel obligated to ask if there is something I can do to help out.

One thing I noticed is that this is very Minitest 5 specific, Rails 5 drops support for any previous version?

Thanks for the awesome job. 👍

Contributor

zamith commented May 7, 2015

Hey guys,

First of all let me say that it is great to finally have a runner bundled with rails that can compete with the one from RSpec.

Also, as the maintainer of m, I feel obligated to ask if there is something I can do to help out.

One thing I noticed is that this is very Minitest 5 specific, Rails 5 drops support for any previous version?

Thanks for the awesome job. 👍

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn May 25, 2015

Member

Ok, I did a few tests with this, and found a few problems:

  • we need to add require "active_support/core_ext/module/attribute_accessors" to minitest_plugin.rb . as we use mattr_accessor()
  • ruby -Itest test/models/post_test.rb is failing with ``test': test_the_truth is already defined in PostTest`
Member

arthurnn commented May 25, 2015

Ok, I did a few tests with this, and found a few problems:

  • we need to add require "active_support/core_ext/module/attribute_accessors" to minitest_plugin.rb . as we use mattr_accessor()
  • ruby -Itest test/models/post_test.rb is failing with ``test': test_the_truth is already defined in PostTest`
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth May 25, 2015

Member

@arthurnn Thanks, I've added the missing require 😄

However, I can't reproduce your test failure locally. Where are you running the command?
I cd'ed into one of the apps generated by test_runner_test.rb (added a simple Gemfile with my Rails clone, arel master and sqlite3), and ran ruby -Itest test/models/post_test.rb just fine.

I've added a test for this too: https://github.com/rails/rails/pull/19571/files#diff-f4d22edadb7ed6b4cc88e000d6c3ed69R254

Member

kaspth commented May 25, 2015

@arthurnn Thanks, I've added the missing require 😄

However, I can't reproduce your test failure locally. Where are you running the command?
I cd'ed into one of the apps generated by test_runner_test.rb (added a simple Gemfile with my Rails clone, arel master and sqlite3), and ran ruby -Itest test/models/post_test.rb just fine.

I've added a test for this too: https://github.com/rails/rails/pull/19571/files#diff-f4d22edadb7ed6b4cc88e000d6c3ed69R254

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn May 26, 2015

Member

@kaspth I pushed my test app https://github.com/arthurnn/myapp .
This is the full stacktrace of the error https://gist.github.com/a4189a0fb5bea964af56

Member

arthurnn commented May 26, 2015

@kaspth I pushed my test app https://github.com/arthurnn/myapp .
This is the full stacktrace of the error https://gist.github.com/a4189a0fb5bea964af56

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth May 30, 2015

Member

@arthurnn Thanks, I've solved it now! 😄

Basically when I set the options[:patterns] I expected opts.order! to return nil and not an empty array when there were no elements.
So plugin_rails_init would hit https://github.com/rails/rails/pull/19571/files#diff-ea6c92bf87b30213dbd5eafb8431976cR38, which erroneously would rerequire the files. I guess Rails' autoloading assumed this meant post_test.rb changed and began reloading it? Anyway it should work now 😉

Member

kaspth commented May 30, 2015

@arthurnn Thanks, I've solved it now! 😄

Basically when I set the options[:patterns] I expected opts.order! to return nil and not an empty array when there were no elements.
So plugin_rails_init would hit https://github.com/rails/rails/pull/19571/files#diff-ea6c92bf87b30213dbd5eafb8431976cR38, which erroneously would rerequire the files. I guess Rails' autoloading assumed this meant post_test.rb changed and began reloading it? Anyway it should work now 😉

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth May 31, 2015

Member

So my earlier solution broke a test running bin/rails test without arguments. Because options[:patterns] was now nil, it didn't require any files and no tests were run.

I've replaced this with a Minitest.run_with_autorun attribute. Basically when we require test_help.rb we can determine what we've been run with: https://github.com/rails/rails/pull/19571/files#diff-96b8e5694f7954d018d9958ef89f4049R12
If it wasn't the rails extension then it has to be autorun. Therefore plugin_rails_init won't attempt to require any files if we've been run with autorun. Because people should make sure all the files they need have been required if they aren't using the Rails test runner.

Member

kaspth commented May 31, 2015

So my earlier solution broke a test running bin/rails test without arguments. Because options[:patterns] was now nil, it didn't require any files and no tests were run.

I've replaced this with a Minitest.run_with_autorun attribute. Basically when we require test_help.rb we can determine what we've been run with: https://github.com/rails/rails/pull/19571/files#diff-96b8e5694f7954d018d9958ef89f4049R12
If it wasn't the rails extension then it has to be autorun. Therefore plugin_rails_init won't attempt to require any files if we've been run with autorun. Because people should make sure all the files they need have been required if they aren't using the Rails test runner.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth May 31, 2015

Member

@arthurnn And we're green! https://travis-ci.org/rails/rails/builds/64804478 - the sqlite failure is the one master is plagued with 😄

Member

kaspth commented May 31, 2015

@arthurnn And we're green! https://travis-ci.org/rails/rails/builds/64804478 - the sqlite failure is the one master is plagued with 😄

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny May 31, 2015

Member

@kaspth I try to give this PR some testing tomorrow. Excited to see it getting close to the finish line.

Member

senny commented May 31, 2015

@kaspth I try to give this PR some testing tomorrow. Excited to see it getting close to the finish line.

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Jun 1, 2015

Member

@kaspth created a new app from this branch using --dev. Executing the runner without any tests fails with the following error:

$ bin/rails t                                                                                                                                                                                                        ~/Projects/playground/ruby/rails/test_runner
/Users/senny/Projects/rails/railties/lib/rails/test_unit/minitest_plugin.rb:40:in `plugin_rails_init': undefined method `backtrace_cleaner' for Rails:Module (NoMethodError)
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:74:in `block in init_plugins'
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:72:in `each'
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:72:in `init_plugins'
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:123:in `run'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/test.rb:5:in `<top (required)>'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:128:in `require'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:128:in `require_command!'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:86:in `test'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
    from /Users/senny/Projects/rails/railties/lib/rails/commands.rb:18:in `<top (required)>'
    from bin/rails:4:in `require'
    from bin/rails:4:in `<main>'
Member

senny commented Jun 1, 2015

@kaspth created a new app from this branch using --dev. Executing the runner without any tests fails with the following error:

$ bin/rails t                                                                                                                                                                                                        ~/Projects/playground/ruby/rails/test_runner
/Users/senny/Projects/rails/railties/lib/rails/test_unit/minitest_plugin.rb:40:in `plugin_rails_init': undefined method `backtrace_cleaner' for Rails:Module (NoMethodError)
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:74:in `block in init_plugins'
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:72:in `each'
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:72:in `init_plugins'
    from /Users/senny/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/minitest-5.7.0/lib/minitest.rb:123:in `run'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/test.rb:5:in `<top (required)>'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:128:in `require'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:128:in `require_command!'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:86:in `test'
    from /Users/senny/Projects/rails/railties/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
    from /Users/senny/Projects/rails/railties/lib/rails/commands.rb:18:in `<top (required)>'
    from bin/rails:4:in `require'
    from bin/rails:4:in `<main>'
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 1, 2015

Member

@senny dang it, I'll take a look 😄

@zenspider I'm not sure I follow?

Member

kaspth commented Jun 1, 2015

@senny dang it, I'll take a look 😄

@zenspider I'm not sure I follow?

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Jun 1, 2015

Member

@kaspth most likely the plugin currently depends on test_help.rb being required. Without tests, that's not the case.

Member

senny commented Jun 1, 2015

@kaspth most likely the plugin currently depends on test_help.rb being required. Without tests, that's not the case.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 2, 2015

Member

@senny Right, I added a simple respond_to. I guess, it's not super clear why we have to check it. Let me know if you want me to change it 😄

Member

kaspth commented Jun 2, 2015

@senny Right, I added a simple respond_to. I guess, it's not super clear why we have to check it. Let me know if you want me to change it 😄

Show outdated Hide outdated railties/lib/rails/test_unit/minitest_plugin.rb
Rails::TestRequirer.require_files options[:patterns] unless run_with_autorun
unless options[:full_backtrace] || ENV["BACKTRACE"]
Minitest.backtrace_filter = Rails.backtrace_cleaner if Rails.respond_to?(:backtrace_cleaner)

This comment has been minimized.

@senny

senny Jun 2, 2015

Member

as you said the respond_to? is not very clear. Let's add a comment to describe why it's there.

@senny

senny Jun 2, 2015

Member

as you said the respond_to? is not very clear. Let's add a comment to describe why it's there.

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Jun 2, 2015

Member

@kaspth if you haven't already, can you add a test-case with a command that results in no runnables? Just to make sure we don't run into this problem again.

Member

senny commented Jun 2, 2015

@kaspth if you haven't already, can you add a test-case with a command that results in no runnables? Just to make sure we don't run into this problem again.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 2, 2015

Member

@senny Definitely. I'll add one now 👍

Member

kaspth commented Jun 2, 2015

@senny Definitely. I'll add one now 👍

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 2, 2015

Member

@senny I can't figure out how to write a failing test for this. Rails is loaded at this point so it'll respond to backtrace_cleaner just fine.

def test_run_app_without_tests
  Dir.chdir(app_path) do
    `rm -rf test`
    `bin/rails test`.tap do |output|
      assert_match '0 runs, 0 assertions', output
    end
  end
end

I guess the test name really should be something like: "run app without rails loaded".

Member

kaspth commented Jun 2, 2015

@senny I can't figure out how to write a failing test for this. Rails is loaded at this point so it'll respond to backtrace_cleaner just fine.

def test_run_app_without_tests
  Dir.chdir(app_path) do
    `rm -rf test`
    `bin/rails test`.tap do |output|
      assert_match '0 runs, 0 assertions', output
    end
  end
end

I guess the test name really should be something like: "run app without rails loaded".

@zenspider

This comment has been minimized.

Show comment
Hide comment
@zenspider

zenspider Jun 2, 2015

Contributor

@kaspth That's because it is hard to follow an idiot making an idiotic comment. :) I've deleted my comment since I entirely missed the point. I'm blaming a lack of caffeine.

Contributor

zenspider commented Jun 2, 2015

@kaspth That's because it is hard to follow an idiot making an idiotic comment. :) I've deleted my comment since I entirely missed the point. I'm blaming a lack of caffeine.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 3, 2015

Member

@zenspider haha, alright then 😄

Member

kaspth commented Jun 3, 2015

@zenspider haha, alright then 😄

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 3, 2015

Member

@senny I found out why the test doesn't reproduce the issue. Here's boot.rb from the generated app in the test:

require '/Users/kasperhansen/Documents/code/rails/load_paths'
require 'rails/all'

And a real app:

ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)

require 'bundler/setup' # Set up gems listed in the Gemfile.

Not sure if we should just override that file or if there's a better way to simulate no runnables where Rails isn't loaded.

Member

kaspth commented Jun 3, 2015

@senny I found out why the test doesn't reproduce the issue. Here's boot.rb from the generated app in the test:

require '/Users/kasperhansen/Documents/code/rails/load_paths'
require 'rails/all'

And a real app:

ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)

require 'bundler/setup' # Set up gems listed in the Gemfile.

Not sure if we should just override that file or if there's a better way to simulate no runnables where Rails isn't loaded.

@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Jun 4, 2015

Member

@kaspth let's start by overwriting the file. If someone has a more elegant idea we can change it later on.

/cc @rafaelfranca

Member

senny commented Jun 4, 2015

@kaspth let's start by overwriting the file. If someone has a more elegant idea we can change it later on.

/cc @rafaelfranca

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Jun 4, 2015

Member

Yeah, I think overwriting is good.

Member

rafaelfranca commented Jun 4, 2015

Yeah, I think overwriting is good.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 4, 2015

Member

@senny @rafaelfranca I've pushed the change overwriting the file.

Member

kaspth commented Jun 4, 2015

@senny @rafaelfranca I've pushed the change overwriting the file.

def test_env
puts Rails.env
class PostTest < ActiveSupport::TestCase
test 'declarative syntax works' do

This comment has been minimized.

@kaspth

kaspth Jun 4, 2015

Member

This test would always pass because we were using def test_the_truth, which didn't cause the bug @arthurnn experienced with the RuntimeError saying "already defined in PostTest."

I'm not sure how to make this clearer.

@kaspth

kaspth Jun 4, 2015

Member

This test would always pass because we were using def test_the_truth, which didn't cause the bug @arthurnn experienced with the RuntimeError saying "already defined in PostTest."

I'm not sure how to make this clearer.

end
end
RUBY
assert_match env, run_test_command("-e #{env} test/unit/env_test.rb")
Dir.chdir(app_path) do
`ruby -Itest test/models/post_test.rb`.tap do |output|

This comment has been minimized.

@kaspth

kaspth Jun 4, 2015

Member

Also if we ever regress here RuntimeError just goes straight through and is spit out in the log. I've tried catching it, but I guess the fact that we're in another process(?) throws us off. The tests will be marked as a failure, but we'll never hit the assertions below.

@kaspth

kaspth Jun 4, 2015

Member

Also if we ever regress here RuntimeError just goes straight through and is spit out in the log. I've tried catching it, but I guess the fact that we're in another process(?) throws us off. The tests will be marked as a failure, but we'll never hit the assertions below.

Improve test runner's Minitest integration.
This also adds free mix and matching of directories, files and lines filters.
Like so:

bin/rails test models/post_test.rb test/integration models/person_test.rb:26

You can also mix in a traditional Minitest filter:

bin/rails test test/integration -n /check_it_out/
@senny

This comment has been minimized.

Show comment
Hide comment
@senny

senny Jun 8, 2015

Member

Just gave the latest version of this PR a spin and everything worked. (Even did some spring testing). Help text is descriptive and the new integration is superb. Thank you @kaspth for your work! 💛 💛 💛

Member

senny commented Jun 8, 2015

Just gave the latest version of this PR a spin and everything worked. (Even did some spring testing). Help text is descriptive and the new integration is superb. Thank you @kaspth for your work! 💛 💛 💛

@senny senny merged commit b6fc8e2 into rails:master Jun 8, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

senny added a commit that referenced this pull request Jun 8, 2015

Merge pull request #19571 from kaspth/improve-runner-integration
Improve Test Runner's Minitest integration.

@kaspth kaspth deleted the kaspth:improve-runner-integration branch Jun 8, 2015

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Jun 8, 2015

Member

Woot! So glad to have this merged ❤️💛💜💚💙

Member

kaspth commented Jun 8, 2015

Woot! So glad to have this merged ❤️💛💜💚💙

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Jun 8, 2015

Member

NICE JOB!!!!

Member

arthurnn commented Jun 8, 2015

NICE JOB!!!!

@deivid-rodriguez

This comment has been minimized.

Show comment
Hide comment
@deivid-rodriguez

deivid-rodriguez Jun 8, 2015

Contributor

@kaspth Thanks a lot for this, it's a great improvement! 😄

Contributor

deivid-rodriguez commented Jun 8, 2015

@kaspth Thanks a lot for this, it's a great improvement! 😄

senny added a commit to senny/rails that referenced this pull request Jun 8, 2015

use our runner (`bin/test`) for framework components.
This adds a script `bin/test` to most Rails framework components. The
script uses the rails minitest plugin to augment the runner.
See rails#19571 for details about the
plugin.

I did not yet add `bin/test` for activerecord, activejob and railties.
These components rely on specific setup performed in the rake-tasks.

senny added a commit to senny/rails that referenced this pull request Jun 11, 2015

use our runner (`bin/test`) for framework components.
This adds a script `bin/test` to most Rails framework components. The
script uses the rails minitest plugin to augment the runner.
See rails#19571 for details about the
plugin.

I did not yet add `bin/test` for activerecord, activejob and railties.
These components rely on specific setup performed in the rake-tasks.

@rafaelfranca rafaelfranca modified the milestones: 5.0.0 [temp], 5.0.0 Dec 30, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment