Introduce Minitest integration #330

Closed
wants to merge 1 commit into
from

Projects

None yet

6 participants

@kbrock
Contributor
kbrock commented May 14, 2015

Hi

This is working for me, but looking for some input.

  1. Suggestions on how to handle lib and test requires? I though the mutant -I test -I lib would have loaded them into the LOAD_PATH, but they were not set. Should I load these from config? Is that available?
  2. Suggestions on loading in the tests? (this seems to be the way rake testtask does it)
  3. What belongs in run vs setup?
  4. How much output do you want from minitest while running tests?
  5. Do you want to start imposing "describe" type tests?
  6. Do you want to support "minitest spec tests too?"

cleanup:

  1. need to squash
  2. need to remove rakefile stuff
  3. Want to include attribution, but this code has reworked so many times by different people that I'm not sure which git commits use as my base. And a majority of the lines have been changed. You have a preference for a bunch of commits vs clean git history?

Thanks

@kbrock kbrock changed the title from Minitest integ to Introduce Minitest integration May 14, 2015
@kbrock kbrock changed the title from Introduce Minitest integration to [WIP] Introduce Minitest integration May 14, 2015
@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+ end
+end
+
+module Mutant
+ class Integration
+ # Minitest integration
+ class Minitest < self
+ # represents the data associated with a test
+ class TestCase
+ attr_accessor :class_name
+ attr_accessor :test_method
+
+ def initialize(class_name, test_method)
+ @class_name = class_name
+ @test_method = test_method
+ end
@mbj
mbj May 14, 2015 Owner

Mutant depends on concord, which allows to combine this:

        attr_accessor :class_name
        attr_accessor :test_method

        def initialize(class_name, test_method)
          @class_name = class_name
          @test_method = test_method
        end

Into:

include Concord::Public.new(:class_name, :test_method)

Which is shorter to write/read and requires less unit testing as concord is mutation covered already and the injected semantics are trustwrothy because "trust your dependencies".

@mbj
mbj May 14, 2015 Owner

#class_name, and #test_method do not have public call sides, so you can use include Concord.new(:class_name, :test_method) to not create the public interface.

@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+
+module Mutant
+ class Integration
+ # Minitest integration
+ class Minitest < self
+ # represents the data associated with a test
+ class TestCase
+ attr_accessor :class_name
+ attr_accessor :test_method
+
+ def initialize(class_name, test_method)
+ @class_name = class_name
+ @test_method = test_method
+ end
+
+ def clazz
@mbj
mbj May 14, 2015 Owner

I think its the ruby convention to name methods returning instances of Class #klass.

@mbj
mbj May 14, 2015 Owner

Also there is no public call side for this method, we want to make it private for that reason.

@mbj mbj and 1 other commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+ class TestCase
+ attr_accessor :class_name
+ attr_accessor :test_method
+
+ def initialize(class_name, test_method)
+ @class_name = class_name
+ @test_method = test_method
+ end
+
+ def clazz
+ Object.const_get(class_name)
+ end
+
+ def method_name
+ test_method[5..-1]
+ end
@mbj
mbj May 14, 2015 Owner

I think this method should be memoized via Adamantium to create a truly idempotent getter #method_name.

@kbrock
kbrock May 15, 2015 Contributor

done

@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+ def clazz
+ Object.const_get(class_name)
+ end
+
+ def method_name
+ test_method[5..-1]
+ end
+
+ def runs_successfully?(reporter)
+ ::Minitest::Runnable.run_one_method(clazz, test_method, reporter)
+ reporter.passes?
+ end
+ end
+
+ ALL = Mutant::Expression.parse('*')
+ EXPRESSION_DELIMITER = ' '.freeze
@mbj
mbj May 14, 2015 Owner

These constants are duplicated in the rspec integration, lets move them to the base class Mutant::Integration.

@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+ end
+
+ def setup
+ $LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
+ $LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
+ Dir.glob('./test/**/*_test.rb').each { |f| require f }
+ Dir.glob('./test/**/test_*.rb').each { |f| require f }
+ end
+
+ # summary reporter detects failures
+ # progress reporter prints dots
+ # could include option verbose if desired
+ def make_reporter(output)
+ options = { io: output }
+ reporter = ::Minitest::CompositeReporter.new
+ reporter << ::Minitest::SummaryReporter.new(options[:io], options)
@mbj
mbj May 14, 2015 Owner

Instead to access the hash key :io, lets just pass in the lvar output as the first argument.

@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+
+ def setup
+ $LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
+ $LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
+ Dir.glob('./test/**/*_test.rb').each { |f| require f }
+ Dir.glob('./test/**/test_*.rb').each { |f| require f }
+ end
+
+ # summary reporter detects failures
+ # progress reporter prints dots
+ # could include option verbose if desired
+ def make_reporter(output)
+ options = { io: output }
+ reporter = ::Minitest::CompositeReporter.new
+ reporter << ::Minitest::SummaryReporter.new(options[:io], options)
+ reporter << ::Minitest::ProgressReporter.new(options[:io], options)
@mbj mbj and 1 other commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+ def initialize
+ @output = StringIO.new
+ @reporter = make_reporter(@output)
+ end
+
+ def setup
+ $LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
+ $LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
+ Dir.glob('./test/**/*_test.rb').each { |f| require f }
+ Dir.glob('./test/**/test_*.rb').each { |f| require f }
+ end
+
+ # summary reporter detects failures
+ # progress reporter prints dots
+ # could include option verbose if desired
+ def make_reporter(output)
@mbj
mbj May 14, 2015 Owner

This method has no public call side, so lets make it private.

@kbrock
kbrock May 15, 2015 Contributor

done

@mbj mbj and 1 other commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+ # summary reporter detects failures
+ # progress reporter prints dots
+ # could include option verbose if desired
+ def make_reporter(output)
+ options = { io: output }
+ reporter = ::Minitest::CompositeReporter.new
+ reporter << ::Minitest::SummaryReporter.new(options[:io], options)
+ reporter << ::Minitest::ProgressReporter.new(options[:io], options)
+ reporter
+ end
+
+ def call(tests)
+ examples = tests.map(&all_tests_index.method(:fetch)).to_set
+ start = Time.now
+
+ examples.detect { |test| !test.runs_successfully?(@reporter) }
@mbj
mbj May 14, 2015 Owner

The return value of Enumerable#detect is unused here. We should use Enumerable#each to reduce the amount of unused semantics being executed.

@kbrock
kbrock May 15, 2015 Contributor

detect bails on the first error.

I used this to make rubocop happy.

I'll put in other changes and see how it goes.

@mbj
mbj May 15, 2015 Owner

Ahh I see. This is more side effect full than I thought.

@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+ end
+
+ ALL = Mutant::Expression.parse('*')
+ EXPRESSION_DELIMITER = ' '.freeze
+
+ register 'minitest'
+
+ def initialize
+ @output = StringIO.new
+ @reporter = make_reporter(@output)
+ end
+
+ def setup
+ $LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
+ $LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
+ Dir.glob('./test/**/*_test.rb').each { |f| require f }
@mbj
mbj May 14, 2015 Owner

Instead of the explicit block being passed to Enumerable#each I think we should go for the shorter version each(&method(:require)).

@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+
+ ALL = Mutant::Expression.parse('*')
+ EXPRESSION_DELIMITER = ' '.freeze
+
+ register 'minitest'
+
+ def initialize
+ @output = StringIO.new
+ @reporter = make_reporter(@output)
+ end
+
+ def setup
+ $LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
+ $LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
+ Dir.glob('./test/**/*_test.rb').each { |f| require f }
+ Dir.glob('./test/**/test_*.rb').each { |f| require f }
@mbj mbj commented on an outdated diff May 14, 2015
lib/mutant/integration/minitest.rb
+module Mutant
+ class Integration
+ # Minitest integration
+ class Minitest < self
+ # represents the data associated with a test
+ class TestCase
+ attr_accessor :class_name
+ attr_accessor :test_method
+
+ def initialize(class_name, test_method)
+ @class_name = class_name
+ @test_method = test_method
+ end
+
+ def clazz
+ Object.const_get(class_name)
@mbj
mbj May 14, 2015 Owner

Are you sure this works on nested classes? I think you better want to use Mutant.constant_lookup which handles nested classes correctly.

@mbj
Owner
mbj commented May 14, 2015

@kbrock Cool. I left some quick code only comments, my OSS time is very limited will look into this ASAP!

@kbrock
Contributor
kbrock commented May 15, 2015

Darn, introduced a failure somewhere.

There are a few methods that are common that I could move up. not sure if it is good:

  1. all_tests
  2. all_tests_index
  3. parse_example
    or maybe: test_from_description("minitest", index, full_description)
@dkubb
Collaborator
dkubb commented May 17, 2015

@kbrock One thing I often do to isolate which commit introduced an error is to push each commit individually using something like https://github.com/dkubb/git-tools/blob/master/git-push-each

When each commit is pushed individually, CircleCI will run the tests on each one. The first one to fail introduced the error so you can at least narrow down where a fix should be applied.

If you comment out the "^$remote/$branch" part on line 13 you should be able to re-push each commit in this branch.

@dkubb
Collaborator
dkubb commented May 17, 2015

@mbj why is travis-ci being used for tests? From what I saw in other branches it was failing yet you still merged. What's the point in having it around if it's not a blocker for merging?

@dkubb
Collaborator
dkubb commented May 17, 2015

@kbrock also if you'd like a hand with this branch, feel free to add me as a committer. I could fix the merge conflict and push each commit if you want.

@dkubb dkubb and 1 other commented on an outdated diff May 17, 2015
lib/mutant/integration/minitest.rb
+ # Minitest integration
+ class Minitest < self
+ # represents the data associated with a test
+ class TestCase
+ include Concord.new(:class_name, :test_method)
+ include Adamantium
+
+ attr_reader :class_name
+
+ def run_passes?(reporter)
+ ::Minitest::Runnable.run_one_method(klass, test_method, reporter)
+ reporter.passed?
+ end
+
+ def method_name
+ test_method[5..-1]
@dkubb
dkubb May 17, 2015 Collaborator

Would test_method.drop(5) work just as well here?

@kbrock
kbrock May 18, 2015 Contributor

I like the look of that, but I tried "abcde".drop(2) in ruby 2.1 and it failed.
Is that an active support thing?

@dkubb
dkubb May 18, 2015 Collaborator

What is test_method ? If it's a String then your way is probably better since String is not Enumerable. I was thinking it was an Array or something, in which case #drop is better.

Aside: I'm not really sure why they decided String shouldn't be Enumerable. I find myself wanting to treat it as a list of "things" and apply Enumerable methods on a semi-regular basis.

@dkubb dkubb and 2 others commented on an outdated diff May 17, 2015
lib/mutant/integration/minitest.rb
+
+ def klass
+ Mutant.constant_lookup(class_name)
+ end
+ memoize :klass
+ end
+
+ register 'minitest'
+
+ def initialize
+ @output = StringIO.new
+ @reporter = make_reporter(@output)
+ end
+
+ def setup
+ $LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
@dkubb
dkubb May 17, 2015 Collaborator

What about:

$LOAD_PATH |= %w[test lib]
@kbrock
kbrock May 18, 2015 Contributor

thanks. I really don't like messing with the load path here.

Actually, I don't like anything that I did in setup.

Also not sure when setup should be called

@kbrock
kbrock May 18, 2015 Contributor

$LOAD_PATH is readonly - won't work.

I could do:

%w[test lib].each { |dir| $LOAD_PATH << dir unless $LOAD_PATH.include?(dir) }
@dkubb
dkubb May 18, 2015 Collaborator

I guess that's slightly better since it removes duplication.

Is there any harm in doing $LOAD_PATH.concat(%w[test lib])? If those directories do happen to exist in the load path, appending them at the end isn't going to make a difference.

@mbj
mbj May 18, 2015 Owner

Also not sure when setup should be called

Mutant needs precise control on when it "infects" the VM under its control with 3rd party code. For that reason there is the distinction between #initialize and #setup.

This distinction might be more relevant for rspec. I'll look into it once I have the time to test this branch.

@dkubb dkubb and 1 other commented on an outdated diff May 17, 2015
lib/mutant/integration/minitest.rb
+ @reporter = make_reporter(@output)
+ end
+
+ def setup
+ $LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
+ $LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
+ Dir.glob('./test/**/*_test.rb').each(&method(:require))
+ Dir.glob('./test/**/test_*.rb').each(&method(:require))
+ self
+ end
+
+ def call(tests)
+ examples = tests.map(&all_tests_index.method(:fetch)).to_set
+ start = Time.now
+
+ examples.each { |test| break unless test.run_passes?(@reporter) }
@dkubb
dkubb May 17, 2015 Collaborator

It's debateable, but I wonder if this would be simpler:

examples.all? { |test| test.run_passes?(@reporter) }

Enumerable#all? still short circuits if one of the tests returns false.

@mbj
mbj May 17, 2015 Owner

I think thats better, as it is overall less AST.

@mbj
Owner
mbj commented May 17, 2015

@mbj why is travis-ci being used for tests? From what I saw in other branches it was failing yet you still merged. What's the point in having it around if it's not a blocker for merging?

I only use circle these days for mutant. As travis is so slow. Also it is a way more indeterministic.

Before each release I typically make travis pass for all matrix elements, so yeah it has some value. I wounder if I can reduce travis usage only for master builds.

@dkubb
Collaborator
dkubb commented May 17, 2015

Before each release I typically make travis pass for all matrix element

@mbj my guess is that CircleCI runs a different set of checks than TravisCI. If they are put in sync I would guess most of the time if one passes the other will pass too (... eventually, in the case of TravisCI).

I wounder if I can reduce travis usage only for master builds.

I'd support just having it run on master only. I know it can be done since we previously restricted DM to run on commits to release-* branches. The full matrix test only makes sense on releasable code, and CircleCI does a better job quickly testing feature branches.

OT but does anyone know if TravisCI is overprovisioned or something? It doesn't look very good for their service when people testing it out have to wait so much longer for results compared to CircleCI. I would much prefer if they were closer in performance, because I think the competition is beneficial to everyone, but at the moment it's no contest.

@mbj
Owner
mbj commented May 17, 2015

I'd support just having it run on master only. I know it can be done since we previously restricted DM to run on commits to release-* branches. The full matrix test only makes sense on releasable code, and CircleCI does a better job quickly testing feature branches.

I actually considered to use a payed plan for my most popular OSS projects. To get full speedy builds with a matrix. There are Circle/Travis competitors that might have this feature (matrix + fast) already.

@kbrock
Contributor
kbrock commented May 18, 2015

@dkubb looks like I was not calling setup in the tests.

@mbj I'm confused about the contract with Integration.setup(name). I expect that code to call setup on the class the first time it is loaded. Thoughts?

@kbrock
Contributor
kbrock commented May 18, 2015

squashed.

Ideas why I need to add test and lib to $LOAD_PATH?
Shouldn't calling mutant with -I test -I list do taht for me?

@mbj
Owner
mbj commented May 18, 2015

@mbj I'm confused about the contract with Integration.setup(name). I expect that code to call setup on the class the first time it is loaded. Thoughts?

Its Integration#setup, not .setup, instance vs class method.

I replied here: #330 (comment)

@mbj
Owner
mbj commented May 18, 2015

Ideas why I need to add test and lib to $LOAD_PATH?
Shouldn't calling mutant with -I test -I list do that for me?

No. -I and friends are meant to setup the library/app under test, not the test for that library. The distinction is particularly important for rspec, as we have to do the subject matching before rspec "infects" the VM with its global state (on Integration#setup), slowing down subject matching significantly due the performance nature of Class#name on anonymous classes.

@dkubb
Collaborator
dkubb commented May 19, 2015

I was able to get the metrics part of the build to pass after applying this patch to the branch:

diff --git a/config/flay.yml b/config/flay.yml
index 9674218..558e402 100644
--- a/config/flay.yml
+++ b/config/flay.yml
@@ -1,3 +1,3 @@
 ---
 threshold: 18
-total_score: 1305
+total_score: 1278
diff --git a/lib/mutant/integration/minitest.rb b/lib/mutant/integration/minitest.rb
index 07db3bf..c8342ff 100644
--- a/lib/mutant/integration/minitest.rb
+++ b/lib/mutant/integration/minitest.rb
@@ -46,7 +46,7 @@ module Mutant
       end

       def setup
-        $LOAD_PATH.concat(%w(test lib))
+        $LOAD_PATH.concat(%w[test lib])
         Dir.glob('./test/**/*_test.rb').each(&method(:require))
         Dir.glob('./test/**/test_*.rb').each(&method(:require))
         self
@@ -102,7 +102,6 @@ module Mutant
         end
       end

-
       # summary reporter detects failures
       # progress reporter prints dots
       # could include option verbose if desired

I'm waiting for the self-mutating part to pass, so I don't know if the whole build is successful, but this should move things a bit forward.

@kbrock
Contributor
kbrock commented May 19, 2015

@dkubb thank you

I pulled out the code that made this more similar to rspec. no need.

@kbrock
Contributor
kbrock commented May 21, 2015

Looks like I lost the full_description method.
Added back in and lets see how that works

@kbrock kbrock changed the title from [WIP] Introduce Minitest integration to Introduce Minitest integration May 21, 2015
@dkubb
Collaborator
dkubb commented May 21, 2015

@mbj FYI it looks like this PR is passing now. The code looks good enough to merge in it's current state.

@kbrock have you tested this against any minitest projects to see how it works? I am curious if it's helped identify any bugs in the test suite.

@backus
Collaborator
backus commented May 22, 2015

@kbrock have you tested this against any minitest projects to see how it works?

@dkubb @kbrock If you want to test it on a live project I just moved tests (only a few dozen) for blockscore-ruby (branch 4.1.0) to minitest and I'm trying to make this branch work with it. I seem to be getting a lot of false positives, but I might be using it wrong. I'm calling it like this:

bundle exec mutant -I lib -I test --require test_helper 'BlockScore.api_key' --use minitest

and it seems to just report almost all permutations are evil. When I make many of the changes and run the tests manually it doesn't seem to be true. Am I doing something wrong or could this be an issue with the minitest setup?

@mbj
Owner
mbj commented May 22, 2015

Am I doing something wrong or could this be an issue with the minitest setup?

Are you getting neutral / noop errors reported? If so the build-in self test for integration correctness (mintest, constraints on your project etc) fails. That could be a signal for remaining problems with the minitest integration.

I still did not had a chance to do a full pass on this PR, sorry. Busy times.

@backus
Collaborator
backus commented May 22, 2015

I believe so, but I'm a bit concerned that I'm executing something wrong. I'll give an example:

I have this test file:

require File.expand_path(File.join(__FILE__, '../test_helper'))

class CandidateResourceTest < Minitest::Test
  include ResourceTest

  def test_history
    candidate = TestClient.create_candidate
    response = candidate.history
    assert_equal Array, response.class
  end

  def test_hits
    candidate = TestClient.create_candidate
    response = candidate.hits
    assert_equal Array, response.class
  end

  def test_update
    candidate = TestClient.create_candidate
    candidate.name_first = 'Chris'
    assert_equal true, candidate.save
    assert_equal candidate.name_first, 'Chris'
  end

  def test_delete
    candidate = TestClient.create_candidate
    candidate.delete
    assert_equal candidate.deleted, true
  end
end

Now if I run this

bundle exec mutant -I lib -I test --require candidate_test 'BlockScore::Candidate#history' --use minitest

With this test directory structure

test
├── candidate_test.rb
├── company_test.rb
├── factories.rb
├── person_test.rb
├── question_set_test.rb
└── test_helper.rb

I get this 'evil' message:

evil:BlockScore::Candidate#history:/Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:15:d6599
@@ -1,4 +1,4 @@
 def history
-  resource_member("history")
+  resource_member(self)
 end
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions=[<Mutant::Expression: BlockScore::Candidate#history>] subject_ignores=[] subject_selects=[]>
Integration:     minitest
Expect Coverage: 100.00%
Jobs:            8
Includes:        ["lib", "test"]
Requires:        ["candidate_test"]
Subjects:        1
Mutations:       9
Kills:           1
Alive:           8
Runtime:         0.25s
Killtime:        0.00s
Overhead:        29641.44%
Coverage:        11.11%
Expected:        100.00%

and when I modify that file accordingly:

$ cat lib/blockscore/candidate.rb
require 'blockscore/actions/create'
require 'blockscore/actions/retrieve'
require 'blockscore/actions/update'
require 'blockscore/actions/delete'
require 'blockscore/actions/all'

module BlockScore
  class Candidate < BlockScore::Base
    include BlockScore::Actions::Create
    include BlockScore::Actions::Retrieve
    include BlockScore::Actions::Update
    include BlockScore::Actions::Delete
    include BlockScore::Actions::All

    def history
      # resource_member('history')
      resource_member(self)
    end

    def hits
      resource_member('hits')
    end

    private

    def resource_member(member)
      self.class.get "#{self.class.endpoint}#{id}/#{member}", {}
    end
  end
end

and run rake test:

$ rake test

# ...

................E........

Finished in 0.626178s, 39.9247 runs/s, 39.9247 assertions/s.

  1) Error:
CandidateResourceTest#test_history:
URI::InvalidURIError: bad URI(is not URI?): https://api.blockscore.com/candidates/1a95be85702ddb90f7fcc1dc/#<BlockScore::Candidate:0x007fd0bc301df0>
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/rfc3986_parser.rb:66:in `split'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/rfc3986_parser.rb:72:in `parse'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/common.rb:226:in `parse'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/common.rb:713:in `URI'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty/request.rb:47:in `path='
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty/request.rb:34:in `initialize'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:521:in `new'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:521:in `perform_request'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:459:in `get'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:559:in `get'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:53:in `execute_request'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:43:in `request'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:13:in `get'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:27:in `resource_member'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:17:in `history'
    /Users/johnbackus/Development/blockscore-ruby/test/candidate_test.rb:8:in `test_history'

25 runs, 25 assertions, 0 failures, 1 errors, 0 skips
rake aborted!
Command failed with status (1): [ruby -I"lib:lib:test"  "/Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/rake/rake_test_loader.rb" "test/*_test.rb" ]
/Users/johnbackus/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `eval'
/Users/johnbackus/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)

it fails.

So is this a noop like you said or am I doing something wrong?

@kbrock
Contributor
kbrock commented May 22, 2015

@bakus agreed - It works in the mini test app, but it is not really helping me with trollop either.

I'll look into this.

@gregnavis

@kbrock great work! I'm excited to see this integration.

I experimented with mutant in a Rails app that is tested with MiniTest. I'm not able to make it work (this may be because of my incorrect use of it though). I run into errors like:

/Users/grn/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/bundler/gems/mutant-cfb976e1412f/lib/mutant/expression.rb:135:in `parse': Expression: "AdminControllerTest#an_error_is_displayed_when_the_e-mail_in_a_password_reset_is_not_found" is not valid (Mutant::Expression::InvalidExpressionError)

It seems that removing the hyphen from the test name resolves the problem. I also can see messages of the form:

Class#name from: NewRelic::Agent::Samplers::VMSampler returned :vm. Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil

This is probably unrelated to the integration itself and should be reported as a separate issue. @mbj could you confirm this, please?

I also experimented with MiniTest (without Rails) and when I define my tests with #test (i.e. test 'this should do something' instead of def test_this_should_do_something) I get errors like:

test 'positive? works for positive values' do
end

# results in /private/tmp/mutant/test/arithmetic_test.rb:10:in `test': wrong number of arguments (1 for 2) (ArgumentError)

test '#positive? works for positive values' do
end
# results in /private/tmp/mutant/test/arithmetic_test.rb:10:in `test': unknown command '#' (ArgumentError)

The example code is available as a gist - https://gist.github.com/grn/f4bef8dbc1b3d20550ff.

@mbj
Owner
mbj commented May 22, 2015

Class#name from: NewRelic::Agent::Samplers::VMSampler returned :vm. Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil

This is from the new relic agent that violates the basic ruby invariant. Mutant uses this invariant to detect subjects, and warns when its violated. So those projects (there had been more like this) can get notified to fix their behavior. Mutant ignores such classes or modules from subject matching.

@mbj
Owner
mbj commented May 22, 2015

I hope to find time over the weekend to finally check this PR out and help to fix the last remaining problems.

@mbj
Owner
mbj commented May 22, 2015

/Users/grn/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/bundler/gems/mutant-cfb976e1412f/lib/mutant/expression.rb:135:inparse': Expression: "AdminControllerTest#an_error_is_displayed_when_the_e-mail_in_a_password_reset_is_not_found" is not valid (Mutant::Expression::InvalidExpressionError)
`

The identifier an_error_is_displayed_when_the_e-mail_in_a_password_reset_is_not_found is NOT a valid literal ruby method name. Mutants expression parser detects this and will not match it. The - between e-mail violates. You probably want to fix that, and turn it into and underscore _.

So yes this is a problem in your test suite. User error.

@gregnavis

I'm not sure about what to think about that. @mbj take a look at the code that triggers this error:

test 'an error is displayed when the e-mail in a password reset is not found' do
# ...
end

Note that I do not use def but #test. Under the hood this is converted to a method name by replacing whitespace with underscores and a test method is defined with #define_method.

Note also that #define_method is much more liberal in allowed method names:

irb(main):001:0> define_method("this!should not-work@but#does") { puts "SUCCESS!" }
=> #<Proc:0x007f996b0d5b18@(irb):1 (lambda)>
irb(main):002:0> send("this!should not-work@but#does")
SUCCESS!
=> nil

I'm not sure how important it's from the perspective of mutant. @mbj what do you think?

@kbrock
Contributor
kbrock commented May 22, 2015

@grn Are there any best practices around MiniTest?

rspec has a better specs.

describe Class1 do
  describe "#positive?" do
    it { is_expected.to be_positive }
    it { expect(subject.positive?).to be_true
  end
end

All tests in the describe are focused on Class1#positive?. Use a # for instance methods. Use a describe when focusing on methods. Use a context when focusing on state - even though those are aliases.

When mutating #positive?, run those, and only those tests.
This avoids possible test coverage that was dumb luck / cross test contamination.

Are there similar best practices for MiniTest?
When reading the spec part of MiniTest they don't seem to push towards a standard.

So can we expect all tests to start with '#positive?' if that is present? Or they could use the describe block...

test '#positive? works for positive values'
@kbrock
Contributor
kbrock commented May 22, 2015

@mbj We may need to introduce tests for each of the MiniTest formats.

Do you have a suggestion for removing the hardcoded directory name (i.e.: "test") from the tests?
Maybe requiring -I test for MiniTest is our best route.

@mbj
Owner
mbj commented May 22, 2015

The minitest integration could try not to extract the tested method, so reducing the selection granularity, to only identify the class the test belongs to.

@gregnavis

@kbrock I'm not aware of anything like better specs (I cannot guarantee it doesn't exist though). Because of that I think you should expect much less consistency in MiniTest tests. I guess that the tests in the wild don't start with a method name and a behaviour description (e.g. '#positive? works for positive values'). Here's a list of things that I think may be important for the integration:

  1. two ways to define a test (def test_xyz and test 'xyz')
  2. two conventions for test classes - TestArithmetic (the MiniTest-convention) and ArithmeticTest (the Rails convention); the files are named test_arithmetic.rb and arithmetic_test.rb respectively
  3. in Rails, the test cases inherit from ActiveSupport::TestCase or ActiveController::TestCase; the inheritance hierarchy is ActiveController::TestCase < ActiveSupport::TestCase < Minitest::Test
  4. there's MiniTest::Spec which resembles RSpec but with some differences; I can't give you any details about this as I don't use it

I don't think we need to address all the points above. In my opinion it's better to get started and support the most basic functionality (e. g. supporting both def test_xyz and test 'xyz').

I can see two solutions to the problem of selecting appropriate test cases:

  1. requiring the names of test methods to have an appropriate form (e.g. ##{method_name} #{behaviour})
  2. adding some kind of annotations to the tests, perhaps in comments
@mbj
Owner
mbj commented May 25, 2015

Do you have a suggestion for removing the hardcoded directory name (i.e.: "test") from the tests?
Maybe requiring -I test for MiniTest is our best route.

@kbrock As long we have tight control when the lib/app under tests gets loaded and when the tests get loaded / executed I'm fine with that.

To be explicit, we need the following to be separated:

  • Loading the application / lib
  • Loading the test framework and the tests
  • Executing a subset of the tests.

Separated in sense of: Mutant can trigger these stages explicitly.

@mbj mbj referenced this pull request in saturnflyer/surrounded May 31, 2015
@saturnflyer saturnflyer adding mutant eaca142
@davydovanton davydovanton referenced this pull request in mperham/sidekiq Jun 22, 2015
Closed

Try out mutant #2403

@backus
Collaborator
backus commented Jun 29, 2015

Did this PR die?

@kbrock
Contributor
kbrock commented Jun 29, 2015

@backus thanks. ping taken

@dkubb
Collaborator
dkubb commented Jun 29, 2015

@kbrock FYI my offer to work with you on this PR is still open :) feel free to add me as a committer to your branch. I could do a few simple things like rebase this on top of master and fix conflicts as well as help test this out.

@mbj
Owner
mbj commented Jul 6, 2015

@kbrock Any chance you can give me and @dkubb commit access to your repo fork, so we can begin to fix work on this?

@kbrock
Contributor
kbrock commented Jul 6, 2015

@mbj @dkubb done

@mbj
Owner
mbj commented Jul 6, 2015

@kbrock cool. Thx. First we'll have to rebase this as the integration API changed slightly. Are you okay with suashing your commits? I'll keep your authorship intact, unless we significantly rewrite things.

@kbrock
Contributor
kbrock commented Jul 6, 2015

@mbj I will squash right now. thnx

@mbj
Owner
mbj commented Jul 6, 2015

@mbj I will squash right now. thnx

Cool. Thx. BTW this is the commit with the breaking API change, you have to accept config an instance of Mutant::Config on Integration#new: d03e7fb

@kbrock
Contributor
kbrock commented Jul 6, 2015

@mbj yea - I remember wanting access to that - good call. I don't have time today to look into this. let me know how far you get.

@mbj
Owner
mbj commented Jul 6, 2015

@mbj yea - I remember wanting access to that - good call. I don't have time today to look into this. let me know how far you get.

My policy is to always wait passing down objects till there is a use case, later in the commit history there is the introduction of a configurable Mutant::Expression::Parser object. In d647563 this happened.

It also allows to have extension specific expressions.

@mbj
Owner
mbj commented Jul 7, 2015

I just rebased the minitest branch by @kbrock on master and fixed the API usage inconsistencies.

To get a better oversight on minitest spec styles (to find a way to support them all), could the minitest users listening to this please name projects that qualify for:

  • Small
  • Very high test coverage (ideally 100% mutation coverage)
  • Uses an ideomatic minitest style consistently

Ideally we have coverage for all major minitest styles. To turn these into corpus tests for mutant.

@mbj
Owner
mbj commented Jul 7, 2015

@mperham Would the Sidekiq project (or a subset of its namespace) qualify for the requirements above?

@mperham
mperham commented Jul 7, 2015

@mbj No idea if Sidekiq qualifies. I have good test coverage but not 100%. You are welcome to use it if useful.

@mbj
Owner
mbj commented Jul 7, 2015

@mbj No idea if Sidekiq qualifies. I have good test coverage but not 100%. You are welcome to use it if useful.

Does it use a consistent minitest style? That would qualify as representative for one of the major ways to use minitest, convention wise I mean.

@backus
Collaborator
backus commented Jul 7, 2015

To get a better oversight on minitest spec styles (to find a way to suppor them all), could the minitest users listening to this please name projects that qualify for:

  • Small
  • Very high test coverage (ideally 100% mutation coverage)
  • Uses an ideomatic minitest style consistently
  • Ideally we have coverage for all major minitest styles.

@zenspider might have some good input here since he is the largest contributor listed on github for minitest, ruby2ruby, and heckle.

@mbj
Owner
mbj commented Jul 7, 2015

You tell me. https://github.com/mperham/sidekiq/tree/master/test

I've no idea, as I never did much with minitest. Thats my problem here.

I do not know what to expect. From reading peoples responses there seem to be at least two styles:

  • Test prefixed classes with test_ prefixed methods
  • Something else with an rspec-a-like syntax, that might generate the above internally, but potentially generates method names that can only be meta-programmed but not be literally present in a ruby source file.

My problem is: Is the listing from above complete, or not?

@mperham
mperham commented Jul 7, 2015

Afaik, your list is complete.

Traditional:
https://github.com/mperham/sidekiq/blob/master/test/test_web_helpers.rb
Spec: https://github.com/mperham/sidekiq/blob/master/test/test_manager.rb

Sidekiq overwhelmingly uses spec but there's one or two uses of traditional.

On Tue, Jul 7, 2015 at 3:24 PM, Markus Schirp notifications@github.com
wrote:

You tell me. https://github.com/mperham/sidekiq/tree/master/test

I've no idea, as I never did much with minitest. Thats my problem here.

I do not know what to expect. From reading peoples responses there seem
to be at least two styles:

  • Test suffixed classes with test_ prefixed methods
  • Something else with an rspec-a-like syntax, that might generate the
    above internally, but potentially generates method names that can only be
    meta-programmed but not be literally present in a ruby source file.

My problem is: Is the listing from above complete, or not?


Reply to this email directly or view it on GitHub
#330 (comment).

@mbj
Owner
mbj commented Jul 7, 2015

Afaik, your list is complete.

Thx that helps a lot. Can I assume there is an 1:1 mapping between Test prefixed class/module with one in the tested code?

So Foo is in the traditional style always tested via TestFoo ?

@mperham
mperham commented Jul 7, 2015

Yeah, I'd think so. Note I use TestManager, not TestSidekiqManager, to test Sidekiq::Manager so it's not directly mappable.

@mbj
Owner
mbj commented Jul 7, 2015

Better question: How could mutant infer that this test:

https://github.com/mperham/sidekiq/blob/master/test/test_manager.rb

Is meaning to specify behavior of this class:

https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/manager.rb

I actually could trace normal test execution to determine this. But I do not like this for various reasons as it has a fundamental downside I typically refer to as "implicit coverage". I plan to support such "implicit" coverage as a side effect of another planned feature.

But ideally we find a way to explicitly specify this relationship. Does minitest in "spec" style have support for metadata. So could I write a test like this:

class TestManager < Sidekiq::Test
  describe 'manager', specifies: Sidekiq::Manager do
    ...

?

@mperham
mperham commented Jul 7, 2015

I don't think you can. I can see how you could rely on rspec's describe(class) for that but I don't know if minitest has the equivalent.

@mperham
mperham commented Jul 7, 2015

I'd be happy to mark up the Sidekiq test suite to support Mutant if you gave minitest users an annotation to use, e.g.

class TestManager < Minitest::Test
  exercises Sidekiq::Manager
@mbj
Owner
mbj commented Jul 7, 2015

I don't think you can. I can see how you could rely on rspec's describe(class) for that but I don't know if minitest has the equivalent.

And thats the key point. This explicit mapping is easy for rspec. And it has a big leverage for execution speed and coverage correctness.

Minitest integration always failed in the past for the lack of this specific mapping metadata. Unless I get the time to do the tracing based selection I see no way how we could do a good rspec equivalent mutant integration with minitest - unless we add this metadata to minitest.

And I'm not an minitest user both OSS and commercial - so there was not enough incentive to actually do one of the proposed solutions. Just running all tests per mutation is not an option. Both because implicit coverage and runtime.

@mbj
Owner
mbj commented Jul 7, 2015

I'd be happy to mark up the Sidekiq test suite to support Mutant if you gave minitest users an annotation to use, e.g.

Yeah, these are project specific integrations. I did plenty of them already. But not the generic one. Project specific is easy and can be done "tomorrow" ;)

@mbj
Owner
mbj commented Jul 7, 2015

@kbrock, @mperham Lets do the following: We simply assume all minitest mutant users will do this exercises TheSubjecOfTest. If they are not willing to annotate: Mutant will not work for them.

That will at least get us started and unblocked.

@mbj
Owner
mbj commented Jul 7, 2015

And instead for having to do a constant reference in this exercises thing, lets allow to pass N mutant expressions.

So you can actually declare a test to match * (any subject under mutation), if you really want this painful way :P

@mbj
Owner
mbj commented Jul 8, 2015

I pushed my latest version that should pass the specs and supports an extension to minitest specifying the subject of coverage explicitly.

This does the job for the test app mutants tests run against:

class TestAppTest < Minitest::Test
  def self.cover(expression)
    @expression = expression
  end

  def self.cover_expression
    unless @expression
      fail "Cover expression for #{self} is not specified"
    end

    @expression
  end
end

class TestApp::LiteralTest < TestAppTest
  cover 'TestApp::Literal*'

  def test_command
    object = ::TestApp::Literal.new
    subject = object.command('x')

    assert_equal object, subject
  end

  def test_string
    object = ::TestApp::Literal.new
    subject = object.string

    assert_equal 'string', subject
  end
end

I'll try to get a subset of sidekiq passing against this branch on CI later this week.

@mbj
Owner
mbj commented Sep 22, 2015

Closing this PR in favor of #445.

@mbj mbj closed this Sep 22, 2015
@kbrock
Contributor
kbrock commented Sep 22, 2015

👍 thanks

@kbrock kbrock deleted the kbrock:minitest branch Sep 22, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment