bin/rails test runner (rerun snippets, run tests by line, option documentation)

[senny]

Successor of #18596.

This is the first stab at implementing a test runner for Rails. Moving forward we want to achieve the following goals with our runner:

• Have an executable with proper arguments and documentation
• Provide features not part of minitest. (running by line, rerun snippets, color?)
• Get rid of some hacks we had to pull when using rake as a test runner (mainly around Rails.env setup)

Follow-up work (can happen after this PR has been merged):

• revisit how plugins can hook into the testing process (rake test:prepare db:test:prepare)
• expand the runner (color?, fail fast?)
• ...

Next Steps

Get feedback and merge this foundation into rails/rails. Then everyone gets the chance to help us improve the runner.

Thank you @arthurnn for helping me out ❤️

This work was inspired by maxitest.

[senny]

/cc @rafaelfranca @carlosantoniodasilva

[arthurnn]

After this is merged, a failed test would be shown like:

[arthurnn@ralph myapp]$ bin/rails test /Users/arthurnn/dev/railstest/myapp/test/models/comment_test.rb:8
Run options: --seed 28435

# Running:

F

Finished in 0.007840s, 127.5510 runs/s, 255.1020 assertions/s.

  1) Failure:
CommentTest#test_failling_one [/Users/arthurnn/dev/railstest/myapp/test/models/comment_test.rb:10]:
Failed refutation, no message given

1 runs, 2 assertions, 1 failures, 0 errors, 0 skips

Failed tests:

bin/rails test /Users/arthurnn/dev/railstest/myapp/test/models/comment_test.rb:8

Also things like this would work:

bin/rails test
bin/rails test test/models
bin/rails test test/models/comment_test.rb

And all the rake equivalents:

bin/rake test
bin/rake test test/models
bin/rake test test/models/comment_test.rb

cc @dhh / @tenderlove

[rafaelfranca]

❤️ 💚 💙 💛 💜

[kaspth]

Should we make a convention out of a test pattern, say test/**/*_test.rb, and support syntax such as:

bin/rails test models
bin/rails test models/comment  

which, respectively, would be the same as:

bin/rails test test/models
bin/rails test test/models/comment_test.rb

[senny]

@kaspth That is what we had with rake test. In my opinion this is not necessary. When using the rspec runner I was always fine having to write the directory. I also find it much easier to discover than more convention case.

[kaspth]

@senny I'm having trouble following you about the discoverability part, can you elaborate?

[senny]

I myself find it easy to understand what the possibilities are when the first argument is called file or directory. The convention pattern approach relies on documentation for people to find it. When working on this I looked closer at the bin/rake test task and there was tons of stuff I didn't know it could do. Like
bin/rake test app/user for example.

I'm not really opposed to adding it but I have the feeling that doesn't add much to the table.

[senny]

Also, the directories are autocompletable while the patterns are not.

[kaspth]

@senny The shipping test task does a lot of stuff that's hard to figure out. I'm 👍 on reigning it in.

[dhh]

👍 to only using paths.

On Mar 5, 2015, at 08:26, Yves Senn notifications@github.com wrote:

Also, the directories are autocompletable while the patterns are not.

—
Reply to this email directly or view it on GitHub.

[tenderlove]

What is this plugin trying to do? I don't understand this code.

[arthurnn]

This plugin is here for two things:

• Add the TestUnitReporter , which outputs:

Failed tests:

bin/rails test /Users/arthurnn/dev/railstest/myapp/test/models/comment_test.rb:8

• Add a filter option to minitest. So we can find the method that we want to run if the runner had a line argument specified, and only run that individual test.

[tenderlove]

I see. Since we control the superclass (everything inherits from AS::TestCase) we can implement the runnables method, so maybe we don't need these globals.

[arthurnn]

I am not 100% if I follow.
We still need to pass to Minitest runner which method we need to run, so we need the plugin, and need someway to know if we are running with the rails_runner or not, thats why $rails_test_runner

[dhh]

Does this report failures as it's running, or only at the end?

[arthurnn]

Does this report failures as it's running, or only at the end?

Only at the end for now. Now that we have our reporter and a rails minitest plugin we will definitely do that too.

[dhh]

👍

On Mar 7, 2015, at 15:36, Arthur Nogueira Neves notifications@github.com wrote:

Does this report failures as it's running, or only at the end?

Only at the end for now. Now that we have our reporter and a rails minitest plugin we will definitely do that too.

—
Reply to this email directly or view it on GitHub.

[rafaelfranca]

Seems good to me but I don't know anything about the implementation. Maybe it is good to ask @zenspider's opinion about the implementation before merging.

[zenspider]

I've just taken a quick glance at this and I guess I don't get it. I can see that you're using the plugin system to put in a custom reporter (almost the same as the reporters provided in seattlerb/minitest-sprint btw) and that's perfect. But, I don't understand why Rails::TestRunner exists as a whole. Can you explain why it exists? It looks like it is a reinvented wheel.

Also, it looks like run_test_file tests the test mechanism by shelling out. That's heavy. Is there some reason for that?

[arthurnn]

@zenspider the Rails::TestRunner exists for a few reasons:

• encapsulate the options we can receive on the runner i.e. rails test file.rb, file.rb is a parameter.
• wrapper to find_method, so we can find the test method if you pass a line number to the runner.
• Setup Rails configs, for instance ENV["RAILS_ENV"]

Pretty much, its main responsibility, is to encapsulate the test options, so we can use it on the minitest plugin, but also on the rails command https://github.com/rails/rails/pull/19216/files#diff-d372de5c2ec1bba7b73aa34bb2d2a798R5 https://github.com/rails/rails/pull/19216/files#diff-47b9fc20bec6924b9946969a002d33c4R9

[tenderlove]

We can do the file name and line number thing without the global. Here is a patch against master that will do it. Adding a runner file seems fine. Maybe we could get access to the option parser on minitest so we don't have to duplicate the settings.

[arthurnn]

@tenderlove A few concerns of you gist:

• This will only work if you pass the exactly line number that the tests starts at.
• You will need to pass the file:line on the filter option of minitest(-n), which is probably not what we want.

I agree that the global is not ideal, I am sure after we merge this people will come-up with a better solution to fix this problem. But to be honest I think this is kinda a implementation detail.

@senny thoughts?

[tenderlove]

This will only work if you pass the exactly line number that the tests starts at.

Yes, I did this as a proof of concept to show that the implementation in this PR may not be the ideal place. I'm trying to give a working demo that doesn't require so much code.

[senny]

While the implementation is not perfect, I'm merging this to create a foundation we can collaborate on. Feel free to send pull requests improving the runner 💛

[arthurnn]

❤️ ❤️ ❤️ ❤️ ❤️

[zenspider]

This still seems like a metric fuckton of code for something that really only needs to hook into the plugin system that minitest already provides. Having an entire runner, adding ~400 lines but only removing ~60? Seems entirely unnecessary. This can be done in 20-30 lines using facilities that minitest already provides.

• encapsulate the options we can receive on the runner i.e. rails test file.rb, file.rb is a parameter.

Minitest already has a system for plugins to extend option processing.

• wrapper to find_method, so we can find the test method if you pass a line number to the runner.

Doesn't need a wrapper. You just need to process and manipulate options

• Setup Rails configs, for instance ENV["RAILS_ENV"]

That certainly doesn't need a runner. Again, the plugin system handles this entirely.

[arthurnn]

thanks @zenspider . I will give it a shot at your suggestions, and cc you in the PR. ❤️

[senny]

@zenspider I agree that we can make the implementation more focus. One thing I really want to have with the new runner is to advertise it's functionality. Similar to the rspec command I'd like to print a "Usage" for every option we have to make the features more discoverable.

[zenspider]

That's handled too.

[senny]

@zenspider thanks for the heads up. We'll look into a better integration with minitest to reduce the code on our end.

[matthuhiggins]

It is worth mentioning that this integrates spring with single test runs.

[BenMorganIO]

👍

[sunnyrjuneja]

Hi everybody, Is this in the rails 4.2.1 release? I just updated rails and it didn't show up in help.

λ →  bin/rails -v
Rails 4.2.1
λ →  bin/rails -h
Usage: rails COMMAND [ARGS]

The most common rails commands are:
 generate    Generate new code (short-cut alias: "g")
 console     Start the Rails console (short-cut alias: "c")
 server      Start the Rails server (short-cut alias: "s")
 dbconsole   Start a console for the database specified in config/database.yml
             (short-cut alias: "db")
 new         Create a new Rails application. "rails new my_app" creates a
             new application called MyApp in "./my_app"

In addition to those, there are:
 destroy      Undo code generated with "generate" (short-cut alias: "d")
 plugin new   Generates skeleton for developing a Rails plugin
 runner       Run a piece of code in the application environment (short-cut alias: "r")

All commands can be run with -h (or --help) for more information.

[arthurnn]

@whatasunnyday no, this is on rails 5 only.
I am extracting this to a gem, to make it work on rails 4.1 and 4.2 !

[sunnyrjuneja]

@arthurnn I just realized that it said landed on master, not 4.2.1. I'm dumb! Sorry for the useless comment. Anyways, thanks for the great work ❤️ ❤️ ❤️.

[BenMorganIO]

@whatasunnyday I was hoping for it to be in 4.2.1 as well lol

[emilebosch]

Hey guys, is this gonna be implementing fail fast, and rerun failed tests? This would be awesome.