New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Minitest integration #330

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
6 participants
@kbrock
Contributor

kbrock commented May 14, 2015

Hi

This is working for me, but looking for some input.

  1. Suggestions on how to handle lib and test requires? I though the mutant -I test -I lib would have loaded them into the LOAD_PATH, but they were not set. Should I load these from config? Is that available?
  2. Suggestions on loading in the tests? (this seems to be the way rake testtask does it)
  3. What belongs in run vs setup?
  4. How much output do you want from minitest while running tests?
  5. Do you want to start imposing "describe" type tests?
  6. Do you want to support "minitest spec tests too?"

cleanup:

  1. need to squash
  2. need to remove rakefile stuff
  3. Want to include attribution, but this code has reworked so many times by different people that I'm not sure which git commits use as my base. And a majority of the lines have been changed. You have a preference for a bunch of commits vs clean git history?

Thanks

@kbrock kbrock changed the title from Minitest integ to Introduce Minitest integration May 14, 2015

@kbrock kbrock changed the title from Introduce Minitest integration to [WIP] Introduce Minitest integration May 14, 2015

Show outdated Hide outdated lib/mutant/integration/minitest.rb
def initialize(class_name, test_method)
@class_name = class_name
@test_method = test_method
end

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

Mutant depends on concord, which allows to combine this:

        attr_accessor :class_name
        attr_accessor :test_method

        def initialize(class_name, test_method)
          @class_name = class_name
          @test_method = test_method
        end

Into:

include Concord::Public.new(:class_name, :test_method)

Which is shorter to write/read and requires less unit testing as concord is mutation covered already and the injected semantics are trustwrothy because "trust your dependencies".

@mbj

mbj May 14, 2015

Owner

Mutant depends on concord, which allows to combine this:

        attr_accessor :class_name
        attr_accessor :test_method

        def initialize(class_name, test_method)
          @class_name = class_name
          @test_method = test_method
        end

Into:

include Concord::Public.new(:class_name, :test_method)

Which is shorter to write/read and requires less unit testing as concord is mutation covered already and the injected semantics are trustwrothy because "trust your dependencies".

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

#class_name, and #test_method do not have public call sides, so you can use include Concord.new(:class_name, :test_method) to not create the public interface.

@mbj

mbj May 14, 2015

Owner

#class_name, and #test_method do not have public call sides, so you can use include Concord.new(:class_name, :test_method) to not create the public interface.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
@test_method = test_method
end
def clazz

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

I think its the ruby convention to name methods returning instances of Class #klass.

@mbj

mbj May 14, 2015

Owner

I think its the ruby convention to name methods returning instances of Class #klass.

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

Also there is no public call side for this method, we want to make it private for that reason.

@mbj

mbj May 14, 2015

Owner

Also there is no public call side for this method, we want to make it private for that reason.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
def method_name
test_method[5..-1]
end

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

I think this method should be memoized via Adamantium to create a truly idempotent getter #method_name.

@mbj

mbj May 14, 2015

Owner

I think this method should be memoized via Adamantium to create a truly idempotent getter #method_name.

This comment has been minimized.

@kbrock

kbrock May 15, 2015

Contributor

done

@kbrock

kbrock May 15, 2015

Contributor

done

Show outdated Hide outdated lib/mutant/integration/minitest.rb
end
ALL = Mutant::Expression.parse('*')
EXPRESSION_DELIMITER = ' '.freeze

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

These constants are duplicated in the rspec integration, lets move them to the base class Mutant::Integration.

@mbj

mbj May 14, 2015

Owner

These constants are duplicated in the rspec integration, lets move them to the base class Mutant::Integration.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
def make_reporter(output)
options = { io: output }
reporter = ::Minitest::CompositeReporter.new
reporter << ::Minitest::SummaryReporter.new(options[:io], options)

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

Instead to access the hash key :io, lets just pass in the lvar output as the first argument.

@mbj

mbj May 14, 2015

Owner

Instead to access the hash key :io, lets just pass in the lvar output as the first argument.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
options = { io: output }
reporter = ::Minitest::CompositeReporter.new
reporter << ::Minitest::SummaryReporter.new(options[:io], options)
reporter << ::Minitest::ProgressReporter.new(options[:io], options)

This comment has been minimized.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
# summary reporter detects failures
# progress reporter prints dots
# could include option verbose if desired
def make_reporter(output)

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

This method has no public call side, so lets make it private.

@mbj

mbj May 14, 2015

Owner

This method has no public call side, so lets make it private.

This comment has been minimized.

@kbrock

kbrock May 15, 2015

Contributor

done

@kbrock

kbrock May 15, 2015

Contributor

done

Show outdated Hide outdated lib/mutant/integration/minitest.rb
examples = tests.map(&all_tests_index.method(:fetch)).to_set
start = Time.now
examples.detect { |test| !test.runs_successfully?(@reporter) }

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

The return value of Enumerable#detect is unused here. We should use Enumerable#each to reduce the amount of unused semantics being executed.

@mbj

mbj May 14, 2015

Owner

The return value of Enumerable#detect is unused here. We should use Enumerable#each to reduce the amount of unused semantics being executed.

This comment has been minimized.

@kbrock

kbrock May 15, 2015

Contributor

detect bails on the first error.

I used this to make rubocop happy.

I'll put in other changes and see how it goes.

@kbrock

kbrock May 15, 2015

Contributor

detect bails on the first error.

I used this to make rubocop happy.

I'll put in other changes and see how it goes.

This comment has been minimized.

@mbj

mbj May 15, 2015

Owner

Ahh I see. This is more side effect full than I thought.

@mbj

mbj May 15, 2015

Owner

Ahh I see. This is more side effect full than I thought.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
def setup
$LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
$LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
Dir.glob('./test/**/*_test.rb').each { |f| require f }

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

Instead of the explicit block being passed to Enumerable#each I think we should go for the shorter version each(&method(:require)).

@mbj

mbj May 14, 2015

Owner

Instead of the explicit block being passed to Enumerable#each I think we should go for the shorter version each(&method(:require)).

Show outdated Hide outdated lib/mutant/integration/minitest.rb
$LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')
$LOAD_PATH << 'lib' unless $LOAD_PATH.include?('lib')
Dir.glob('./test/**/*_test.rb').each { |f| require f }
Dir.glob('./test/**/test_*.rb').each { |f| require f }

This comment has been minimized.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
end
def clazz
Object.const_get(class_name)

This comment has been minimized.

@mbj

mbj May 14, 2015

Owner

Are you sure this works on nested classes? I think you better want to use Mutant.constant_lookup which handles nested classes correctly.

@mbj

mbj May 14, 2015

Owner

Are you sure this works on nested classes? I think you better want to use Mutant.constant_lookup which handles nested classes correctly.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 14, 2015

Owner

@kbrock Cool. I left some quick code only comments, my OSS time is very limited will look into this ASAP!

Owner

mbj commented May 14, 2015

@kbrock Cool. I left some quick code only comments, my OSS time is very limited will look into this ASAP!

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock May 15, 2015

Contributor

Darn, introduced a failure somewhere.

There are a few methods that are common that I could move up. not sure if it is good:

  1. all_tests
  2. all_tests_index
  3. parse_example
    or maybe: test_from_description("minitest", index, full_description)
Contributor

kbrock commented May 15, 2015

Darn, introduced a failure somewhere.

There are a few methods that are common that I could move up. not sure if it is good:

  1. all_tests
  2. all_tests_index
  3. parse_example
    or maybe: test_from_description("minitest", index, full_description)
@dkubb

This comment has been minimized.

Show comment
Hide comment
@dkubb

dkubb May 17, 2015

Collaborator

@kbrock One thing I often do to isolate which commit introduced an error is to push each commit individually using something like https://github.com/dkubb/git-tools/blob/master/git-push-each

When each commit is pushed individually, CircleCI will run the tests on each one. The first one to fail introduced the error so you can at least narrow down where a fix should be applied.

If you comment out the "^$remote/$branch" part on line 13 you should be able to re-push each commit in this branch.

Collaborator

dkubb commented May 17, 2015

@kbrock One thing I often do to isolate which commit introduced an error is to push each commit individually using something like https://github.com/dkubb/git-tools/blob/master/git-push-each

When each commit is pushed individually, CircleCI will run the tests on each one. The first one to fail introduced the error so you can at least narrow down where a fix should be applied.

If you comment out the "^$remote/$branch" part on line 13 you should be able to re-push each commit in this branch.

@dkubb

This comment has been minimized.

Show comment
Hide comment
@dkubb

dkubb May 17, 2015

Collaborator

@mbj why is travis-ci being used for tests? From what I saw in other branches it was failing yet you still merged. What's the point in having it around if it's not a blocker for merging?

Collaborator

dkubb commented May 17, 2015

@mbj why is travis-ci being used for tests? From what I saw in other branches it was failing yet you still merged. What's the point in having it around if it's not a blocker for merging?

@dkubb

This comment has been minimized.

Show comment
Hide comment
@dkubb

dkubb May 17, 2015

Collaborator

@kbrock also if you'd like a hand with this branch, feel free to add me as a committer. I could fix the merge conflict and push each commit if you want.

Collaborator

dkubb commented May 17, 2015

@kbrock also if you'd like a hand with this branch, feel free to add me as a committer. I could fix the merge conflict and push each commit if you want.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
end
def method_name
test_method[5..-1]

This comment has been minimized.

@dkubb

dkubb May 17, 2015

Collaborator

Would test_method.drop(5) work just as well here?

@dkubb

dkubb May 17, 2015

Collaborator

Would test_method.drop(5) work just as well here?

This comment has been minimized.

@kbrock

kbrock May 18, 2015

Contributor

I like the look of that, but I tried "abcde".drop(2) in ruby 2.1 and it failed.
Is that an active support thing?

@kbrock

kbrock May 18, 2015

Contributor

I like the look of that, but I tried "abcde".drop(2) in ruby 2.1 and it failed.
Is that an active support thing?

This comment has been minimized.

@dkubb

dkubb May 18, 2015

Collaborator

What is test_method ? If it's a String then your way is probably better since String is not Enumerable. I was thinking it was an Array or something, in which case #drop is better.

Aside: I'm not really sure why they decided String shouldn't be Enumerable. I find myself wanting to treat it as a list of "things" and apply Enumerable methods on a semi-regular basis.

@dkubb

dkubb May 18, 2015

Collaborator

What is test_method ? If it's a String then your way is probably better since String is not Enumerable. I was thinking it was an Array or something, in which case #drop is better.

Aside: I'm not really sure why they decided String shouldn't be Enumerable. I find myself wanting to treat it as a list of "things" and apply Enumerable methods on a semi-regular basis.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
end
def setup
$LOAD_PATH << 'test' unless $LOAD_PATH.include?('test')

This comment has been minimized.

@dkubb

dkubb May 17, 2015

Collaborator

What about:

$LOAD_PATH |= %w[test lib]
@dkubb

dkubb May 17, 2015

Collaborator

What about:

$LOAD_PATH |= %w[test lib]

This comment has been minimized.

@kbrock

kbrock May 18, 2015

Contributor

thanks. I really don't like messing with the load path here.

Actually, I don't like anything that I did in setup.

Also not sure when setup should be called

@kbrock

kbrock May 18, 2015

Contributor

thanks. I really don't like messing with the load path here.

Actually, I don't like anything that I did in setup.

Also not sure when setup should be called

This comment has been minimized.

@kbrock

kbrock May 18, 2015

Contributor

$LOAD_PATH is readonly - won't work.

I could do:

%w[test lib].each { |dir| $LOAD_PATH << dir unless $LOAD_PATH.include?(dir) }
@kbrock

kbrock May 18, 2015

Contributor

$LOAD_PATH is readonly - won't work.

I could do:

%w[test lib].each { |dir| $LOAD_PATH << dir unless $LOAD_PATH.include?(dir) }

This comment has been minimized.

@dkubb

dkubb May 18, 2015

Collaborator

I guess that's slightly better since it removes duplication.

Is there any harm in doing $LOAD_PATH.concat(%w[test lib])? If those directories do happen to exist in the load path, appending them at the end isn't going to make a difference.

@dkubb

dkubb May 18, 2015

Collaborator

I guess that's slightly better since it removes duplication.

Is there any harm in doing $LOAD_PATH.concat(%w[test lib])? If those directories do happen to exist in the load path, appending them at the end isn't going to make a difference.

This comment has been minimized.

@mbj

mbj May 18, 2015

Owner

Also not sure when setup should be called

Mutant needs precise control on when it "infects" the VM under its control with 3rd party code. For that reason there is the distinction between #initialize and #setup.

This distinction might be more relevant for rspec. I'll look into it once I have the time to test this branch.

@mbj

mbj May 18, 2015

Owner

Also not sure when setup should be called

Mutant needs precise control on when it "infects" the VM under its control with 3rd party code. For that reason there is the distinction between #initialize and #setup.

This distinction might be more relevant for rspec. I'll look into it once I have the time to test this branch.

Show outdated Hide outdated lib/mutant/integration/minitest.rb
examples = tests.map(&all_tests_index.method(:fetch)).to_set
start = Time.now
examples.each { |test| break unless test.run_passes?(@reporter) }

This comment has been minimized.

@dkubb

dkubb May 17, 2015

Collaborator

It's debateable, but I wonder if this would be simpler:

examples.all? { |test| test.run_passes?(@reporter) }

Enumerable#all? still short circuits if one of the tests returns false.

@dkubb

dkubb May 17, 2015

Collaborator

It's debateable, but I wonder if this would be simpler:

examples.all? { |test| test.run_passes?(@reporter) }

Enumerable#all? still short circuits if one of the tests returns false.

This comment has been minimized.

@mbj

mbj May 17, 2015

Owner

I think thats better, as it is overall less AST.

@mbj

mbj May 17, 2015

Owner

I think thats better, as it is overall less AST.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 17, 2015

Owner

@mbj why is travis-ci being used for tests? From what I saw in other branches it was failing yet you still merged. What's the point in having it around if it's not a blocker for merging?

I only use circle these days for mutant. As travis is so slow. Also it is a way more indeterministic.

Before each release I typically make travis pass for all matrix elements, so yeah it has some value. I wounder if I can reduce travis usage only for master builds.

Owner

mbj commented May 17, 2015

@mbj why is travis-ci being used for tests? From what I saw in other branches it was failing yet you still merged. What's the point in having it around if it's not a blocker for merging?

I only use circle these days for mutant. As travis is so slow. Also it is a way more indeterministic.

Before each release I typically make travis pass for all matrix elements, so yeah it has some value. I wounder if I can reduce travis usage only for master builds.

@dkubb

This comment has been minimized.

Show comment
Hide comment
@dkubb

dkubb May 17, 2015

Collaborator

Before each release I typically make travis pass for all matrix element

@mbj my guess is that CircleCI runs a different set of checks than TravisCI. If they are put in sync I would guess most of the time if one passes the other will pass too (... eventually, in the case of TravisCI).

I wounder if I can reduce travis usage only for master builds.

I'd support just having it run on master only. I know it can be done since we previously restricted DM to run on commits to release-* branches. The full matrix test only makes sense on releasable code, and CircleCI does a better job quickly testing feature branches.

OT but does anyone know if TravisCI is overprovisioned or something? It doesn't look very good for their service when people testing it out have to wait so much longer for results compared to CircleCI. I would much prefer if they were closer in performance, because I think the competition is beneficial to everyone, but at the moment it's no contest.

Collaborator

dkubb commented May 17, 2015

Before each release I typically make travis pass for all matrix element

@mbj my guess is that CircleCI runs a different set of checks than TravisCI. If they are put in sync I would guess most of the time if one passes the other will pass too (... eventually, in the case of TravisCI).

I wounder if I can reduce travis usage only for master builds.

I'd support just having it run on master only. I know it can be done since we previously restricted DM to run on commits to release-* branches. The full matrix test only makes sense on releasable code, and CircleCI does a better job quickly testing feature branches.

OT but does anyone know if TravisCI is overprovisioned or something? It doesn't look very good for their service when people testing it out have to wait so much longer for results compared to CircleCI. I would much prefer if they were closer in performance, because I think the competition is beneficial to everyone, but at the moment it's no contest.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 17, 2015

Owner

I'd support just having it run on master only. I know it can be done since we previously restricted DM to run on commits to release-* branches. The full matrix test only makes sense on releasable code, and CircleCI does a better job quickly testing feature branches.

I actually considered to use a payed plan for my most popular OSS projects. To get full speedy builds with a matrix. There are Circle/Travis competitors that might have this feature (matrix + fast) already.

Owner

mbj commented May 17, 2015

I'd support just having it run on master only. I know it can be done since we previously restricted DM to run on commits to release-* branches. The full matrix test only makes sense on releasable code, and CircleCI does a better job quickly testing feature branches.

I actually considered to use a payed plan for my most popular OSS projects. To get full speedy builds with a matrix. There are Circle/Travis competitors that might have this feature (matrix + fast) already.

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock May 18, 2015

Contributor

@dkubb looks like I was not calling setup in the tests.

@mbj I'm confused about the contract with Integration.setup(name). I expect that code to call setup on the class the first time it is loaded. Thoughts?

Contributor

kbrock commented May 18, 2015

@dkubb looks like I was not calling setup in the tests.

@mbj I'm confused about the contract with Integration.setup(name). I expect that code to call setup on the class the first time it is loaded. Thoughts?

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock May 18, 2015

Contributor

squashed.

Ideas why I need to add test and lib to $LOAD_PATH?
Shouldn't calling mutant with -I test -I list do taht for me?

Contributor

kbrock commented May 18, 2015

squashed.

Ideas why I need to add test and lib to $LOAD_PATH?
Shouldn't calling mutant with -I test -I list do taht for me?

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 18, 2015

Owner

@mbj I'm confused about the contract with Integration.setup(name). I expect that code to call setup on the class the first time it is loaded. Thoughts?

Its Integration#setup, not .setup, instance vs class method.

I replied here: #330 (comment)

Owner

mbj commented May 18, 2015

@mbj I'm confused about the contract with Integration.setup(name). I expect that code to call setup on the class the first time it is loaded. Thoughts?

Its Integration#setup, not .setup, instance vs class method.

I replied here: #330 (comment)

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 18, 2015

Owner

Ideas why I need to add test and lib to $LOAD_PATH?
Shouldn't calling mutant with -I test -I list do that for me?

No. -I and friends are meant to setup the library/app under test, not the test for that library. The distinction is particularly important for rspec, as we have to do the subject matching before rspec "infects" the VM with its global state (on Integration#setup), slowing down subject matching significantly due the performance nature of Class#name on anonymous classes.

Owner

mbj commented May 18, 2015

Ideas why I need to add test and lib to $LOAD_PATH?
Shouldn't calling mutant with -I test -I list do that for me?

No. -I and friends are meant to setup the library/app under test, not the test for that library. The distinction is particularly important for rspec, as we have to do the subject matching before rspec "infects" the VM with its global state (on Integration#setup), slowing down subject matching significantly due the performance nature of Class#name on anonymous classes.

@dkubb

This comment has been minimized.

Show comment
Hide comment
@dkubb

dkubb May 19, 2015

Collaborator

I was able to get the metrics part of the build to pass after applying this patch to the branch:

diff --git a/config/flay.yml b/config/flay.yml
index 9674218..558e402 100644
--- a/config/flay.yml
+++ b/config/flay.yml
@@ -1,3 +1,3 @@
 ---
 threshold: 18
-total_score: 1305
+total_score: 1278
diff --git a/lib/mutant/integration/minitest.rb b/lib/mutant/integration/minitest.rb
index 07db3bf..c8342ff 100644
--- a/lib/mutant/integration/minitest.rb
+++ b/lib/mutant/integration/minitest.rb
@@ -46,7 +46,7 @@ module Mutant
       end

       def setup
-        $LOAD_PATH.concat(%w(test lib))
+        $LOAD_PATH.concat(%w[test lib])
         Dir.glob('./test/**/*_test.rb').each(&method(:require))
         Dir.glob('./test/**/test_*.rb').each(&method(:require))
         self
@@ -102,7 +102,6 @@ module Mutant
         end
       end

-
       # summary reporter detects failures
       # progress reporter prints dots
       # could include option verbose if desired

I'm waiting for the self-mutating part to pass, so I don't know if the whole build is successful, but this should move things a bit forward.

Collaborator

dkubb commented May 19, 2015

I was able to get the metrics part of the build to pass after applying this patch to the branch:

diff --git a/config/flay.yml b/config/flay.yml
index 9674218..558e402 100644
--- a/config/flay.yml
+++ b/config/flay.yml
@@ -1,3 +1,3 @@
 ---
 threshold: 18
-total_score: 1305
+total_score: 1278
diff --git a/lib/mutant/integration/minitest.rb b/lib/mutant/integration/minitest.rb
index 07db3bf..c8342ff 100644
--- a/lib/mutant/integration/minitest.rb
+++ b/lib/mutant/integration/minitest.rb
@@ -46,7 +46,7 @@ module Mutant
       end

       def setup
-        $LOAD_PATH.concat(%w(test lib))
+        $LOAD_PATH.concat(%w[test lib])
         Dir.glob('./test/**/*_test.rb').each(&method(:require))
         Dir.glob('./test/**/test_*.rb').each(&method(:require))
         self
@@ -102,7 +102,6 @@ module Mutant
         end
       end

-
       # summary reporter detects failures
       # progress reporter prints dots
       # could include option verbose if desired

I'm waiting for the self-mutating part to pass, so I don't know if the whole build is successful, but this should move things a bit forward.

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock May 19, 2015

Contributor

@dkubb thank you

I pulled out the code that made this more similar to rspec. no need.

Contributor

kbrock commented May 19, 2015

@dkubb thank you

I pulled out the code that made this more similar to rspec. no need.

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock May 21, 2015

Contributor

Looks like I lost the full_description method.
Added back in and lets see how that works

Contributor

kbrock commented May 21, 2015

Looks like I lost the full_description method.
Added back in and lets see how that works

@kbrock kbrock changed the title from [WIP] Introduce Minitest integration to Introduce Minitest integration May 21, 2015

@dkubb

This comment has been minimized.

Show comment
Hide comment
@dkubb

dkubb May 21, 2015

Collaborator

@mbj FYI it looks like this PR is passing now. The code looks good enough to merge in it's current state.

@kbrock have you tested this against any minitest projects to see how it works? I am curious if it's helped identify any bugs in the test suite.

Collaborator

dkubb commented May 21, 2015

@mbj FYI it looks like this PR is passing now. The code looks good enough to merge in it's current state.

@kbrock have you tested this against any minitest projects to see how it works? I am curious if it's helped identify any bugs in the test suite.

@backus

This comment has been minimized.

Show comment
Hide comment
@backus

backus May 22, 2015

Contributor

@kbrock have you tested this against any minitest projects to see how it works?

@dkubb @kbrock If you want to test it on a live project I just moved tests (only a few dozen) for blockscore-ruby (branch 4.1.0) to minitest and I'm trying to make this branch work with it. I seem to be getting a lot of false positives, but I might be using it wrong. I'm calling it like this:

bundle exec mutant -I lib -I test --require test_helper 'BlockScore.api_key' --use minitest

and it seems to just report almost all permutations are evil. When I make many of the changes and run the tests manually it doesn't seem to be true. Am I doing something wrong or could this be an issue with the minitest setup?

Contributor

backus commented May 22, 2015

@kbrock have you tested this against any minitest projects to see how it works?

@dkubb @kbrock If you want to test it on a live project I just moved tests (only a few dozen) for blockscore-ruby (branch 4.1.0) to minitest and I'm trying to make this branch work with it. I seem to be getting a lot of false positives, but I might be using it wrong. I'm calling it like this:

bundle exec mutant -I lib -I test --require test_helper 'BlockScore.api_key' --use minitest

and it seems to just report almost all permutations are evil. When I make many of the changes and run the tests manually it doesn't seem to be true. Am I doing something wrong or could this be an issue with the minitest setup?

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 22, 2015

Owner

Am I doing something wrong or could this be an issue with the minitest setup?

Are you getting neutral / noop errors reported? If so the build-in self test for integration correctness (mintest, constraints on your project etc) fails. That could be a signal for remaining problems with the minitest integration.

I still did not had a chance to do a full pass on this PR, sorry. Busy times.

Owner

mbj commented May 22, 2015

Am I doing something wrong or could this be an issue with the minitest setup?

Are you getting neutral / noop errors reported? If so the build-in self test for integration correctness (mintest, constraints on your project etc) fails. That could be a signal for remaining problems with the minitest integration.

I still did not had a chance to do a full pass on this PR, sorry. Busy times.

@backus

This comment has been minimized.

Show comment
Hide comment
@backus

backus May 22, 2015

Contributor

I believe so, but I'm a bit concerned that I'm executing something wrong. I'll give an example:

I have this test file:

require File.expand_path(File.join(__FILE__, '../test_helper'))

class CandidateResourceTest < Minitest::Test
  include ResourceTest

  def test_history
    candidate = TestClient.create_candidate
    response = candidate.history
    assert_equal Array, response.class
  end

  def test_hits
    candidate = TestClient.create_candidate
    response = candidate.hits
    assert_equal Array, response.class
  end

  def test_update
    candidate = TestClient.create_candidate
    candidate.name_first = 'Chris'
    assert_equal true, candidate.save
    assert_equal candidate.name_first, 'Chris'
  end

  def test_delete
    candidate = TestClient.create_candidate
    candidate.delete
    assert_equal candidate.deleted, true
  end
end

Now if I run this

bundle exec mutant -I lib -I test --require candidate_test 'BlockScore::Candidate#history' --use minitest

With this test directory structure

test
├── candidate_test.rb
├── company_test.rb
├── factories.rb
├── person_test.rb
├── question_set_test.rb
└── test_helper.rb

I get this 'evil' message:

evil:BlockScore::Candidate#history:/Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:15:d6599
@@ -1,4 +1,4 @@
 def history
-  resource_member("history")
+  resource_member(self)
 end
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions=[<Mutant::Expression: BlockScore::Candidate#history>] subject_ignores=[] subject_selects=[]>
Integration:     minitest
Expect Coverage: 100.00%
Jobs:            8
Includes:        ["lib", "test"]
Requires:        ["candidate_test"]
Subjects:        1
Mutations:       9
Kills:           1
Alive:           8
Runtime:         0.25s
Killtime:        0.00s
Overhead:        29641.44%
Coverage:        11.11%
Expected:        100.00%

and when I modify that file accordingly:

$ cat lib/blockscore/candidate.rb
require 'blockscore/actions/create'
require 'blockscore/actions/retrieve'
require 'blockscore/actions/update'
require 'blockscore/actions/delete'
require 'blockscore/actions/all'

module BlockScore
  class Candidate < BlockScore::Base
    include BlockScore::Actions::Create
    include BlockScore::Actions::Retrieve
    include BlockScore::Actions::Update
    include BlockScore::Actions::Delete
    include BlockScore::Actions::All

    def history
      # resource_member('history')
      resource_member(self)
    end

    def hits
      resource_member('hits')
    end

    private

    def resource_member(member)
      self.class.get "#{self.class.endpoint}#{id}/#{member}", {}
    end
  end
end

and run rake test:

$ rake test

# ...

................E........

Finished in 0.626178s, 39.9247 runs/s, 39.9247 assertions/s.

  1) Error:
CandidateResourceTest#test_history:
URI::InvalidURIError: bad URI(is not URI?): https://api.blockscore.com/candidates/1a95be85702ddb90f7fcc1dc/#<BlockScore::Candidate:0x007fd0bc301df0>
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/rfc3986_parser.rb:66:in `split'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/rfc3986_parser.rb:72:in `parse'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/common.rb:226:in `parse'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/common.rb:713:in `URI'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty/request.rb:47:in `path='
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty/request.rb:34:in `initialize'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:521:in `new'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:521:in `perform_request'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:459:in `get'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:559:in `get'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:53:in `execute_request'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:43:in `request'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:13:in `get'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:27:in `resource_member'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:17:in `history'
    /Users/johnbackus/Development/blockscore-ruby/test/candidate_test.rb:8:in `test_history'

25 runs, 25 assertions, 0 failures, 1 errors, 0 skips
rake aborted!
Command failed with status (1): [ruby -I"lib:lib:test"  "/Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/rake/rake_test_loader.rb" "test/*_test.rb" ]
/Users/johnbackus/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `eval'
/Users/johnbackus/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)

it fails.

So is this a noop like you said or am I doing something wrong?

Contributor

backus commented May 22, 2015

I believe so, but I'm a bit concerned that I'm executing something wrong. I'll give an example:

I have this test file:

require File.expand_path(File.join(__FILE__, '../test_helper'))

class CandidateResourceTest < Minitest::Test
  include ResourceTest

  def test_history
    candidate = TestClient.create_candidate
    response = candidate.history
    assert_equal Array, response.class
  end

  def test_hits
    candidate = TestClient.create_candidate
    response = candidate.hits
    assert_equal Array, response.class
  end

  def test_update
    candidate = TestClient.create_candidate
    candidate.name_first = 'Chris'
    assert_equal true, candidate.save
    assert_equal candidate.name_first, 'Chris'
  end

  def test_delete
    candidate = TestClient.create_candidate
    candidate.delete
    assert_equal candidate.deleted, true
  end
end

Now if I run this

bundle exec mutant -I lib -I test --require candidate_test 'BlockScore::Candidate#history' --use minitest

With this test directory structure

test
├── candidate_test.rb
├── company_test.rb
├── factories.rb
├── person_test.rb
├── question_set_test.rb
└── test_helper.rb

I get this 'evil' message:

evil:BlockScore::Candidate#history:/Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:15:d6599
@@ -1,4 +1,4 @@
 def history
-  resource_member("history")
+  resource_member(self)
 end
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions=[<Mutant::Expression: BlockScore::Candidate#history>] subject_ignores=[] subject_selects=[]>
Integration:     minitest
Expect Coverage: 100.00%
Jobs:            8
Includes:        ["lib", "test"]
Requires:        ["candidate_test"]
Subjects:        1
Mutations:       9
Kills:           1
Alive:           8
Runtime:         0.25s
Killtime:        0.00s
Overhead:        29641.44%
Coverage:        11.11%
Expected:        100.00%

and when I modify that file accordingly:

$ cat lib/blockscore/candidate.rb
require 'blockscore/actions/create'
require 'blockscore/actions/retrieve'
require 'blockscore/actions/update'
require 'blockscore/actions/delete'
require 'blockscore/actions/all'

module BlockScore
  class Candidate < BlockScore::Base
    include BlockScore::Actions::Create
    include BlockScore::Actions::Retrieve
    include BlockScore::Actions::Update
    include BlockScore::Actions::Delete
    include BlockScore::Actions::All

    def history
      # resource_member('history')
      resource_member(self)
    end

    def hits
      resource_member('hits')
    end

    private

    def resource_member(member)
      self.class.get "#{self.class.endpoint}#{id}/#{member}", {}
    end
  end
end

and run rake test:

$ rake test

# ...

................E........

Finished in 0.626178s, 39.9247 runs/s, 39.9247 assertions/s.

  1) Error:
CandidateResourceTest#test_history:
URI::InvalidURIError: bad URI(is not URI?): https://api.blockscore.com/candidates/1a95be85702ddb90f7fcc1dc/#<BlockScore::Candidate:0x007fd0bc301df0>
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/rfc3986_parser.rb:66:in `split'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/rfc3986_parser.rb:72:in `parse'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/common.rb:226:in `parse'
    /Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/uri/common.rb:713:in `URI'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty/request.rb:47:in `path='
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty/request.rb:34:in `initialize'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:521:in `new'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:521:in `perform_request'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:459:in `get'
    /Users/johnbackus/.rvm/gems/ruby-2.2.2/gems/httparty-0.13.4/lib/httparty.rb:559:in `get'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:53:in `execute_request'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:43:in `request'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/connection.rb:13:in `get'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:27:in `resource_member'
    /Users/johnbackus/Development/blockscore-ruby/lib/blockscore/candidate.rb:17:in `history'
    /Users/johnbackus/Development/blockscore-ruby/test/candidate_test.rb:8:in `test_history'

25 runs, 25 assertions, 0 failures, 1 errors, 0 skips
rake aborted!
Command failed with status (1): [ruby -I"lib:lib:test"  "/Users/johnbackus/.rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/rake/rake_test_loader.rb" "test/*_test.rb" ]
/Users/johnbackus/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `eval'
/Users/johnbackus/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => test
(See full trace by running task with --trace)

it fails.

So is this a noop like you said or am I doing something wrong?

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock May 22, 2015

Contributor

@Bakus agreed - It works in the mini test app, but it is not really helping me with trollop either.

I'll look into this.

Contributor

kbrock commented May 22, 2015

@Bakus agreed - It works in the mini test app, but it is not really helping me with trollop either.

I'll look into this.

@gregnavis

This comment has been minimized.

Show comment
Hide comment
@gregnavis

gregnavis May 22, 2015

@kbrock great work! I'm excited to see this integration.

I experimented with mutant in a Rails app that is tested with MiniTest. I'm not able to make it work (this may be because of my incorrect use of it though). I run into errors like:

/Users/grn/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/bundler/gems/mutant-cfb976e1412f/lib/mutant/expression.rb:135:in `parse': Expression: "AdminControllerTest#an_error_is_displayed_when_the_e-mail_in_a_password_reset_is_not_found" is not valid (Mutant::Expression::InvalidExpressionError)

It seems that removing the hyphen from the test name resolves the problem. I also can see messages of the form:

Class#name from: NewRelic::Agent::Samplers::VMSampler returned :vm. Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil

This is probably unrelated to the integration itself and should be reported as a separate issue. @mbj could you confirm this, please?

I also experimented with MiniTest (without Rails) and when I define my tests with #test (i.e. test 'this should do something' instead of def test_this_should_do_something) I get errors like:

test 'positive? works for positive values' do
end

# results in /private/tmp/mutant/test/arithmetic_test.rb:10:in `test': wrong number of arguments (1 for 2) (ArgumentError)

test '#positive? works for positive values' do
end
# results in /private/tmp/mutant/test/arithmetic_test.rb:10:in `test': unknown command '#' (ArgumentError)

The example code is available as a gist - https://gist.github.com/grn/f4bef8dbc1b3d20550ff.

gregnavis commented May 22, 2015

@kbrock great work! I'm excited to see this integration.

I experimented with mutant in a Rails app that is tested with MiniTest. I'm not able to make it work (this may be because of my incorrect use of it though). I run into errors like:

/Users/grn/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/bundler/gems/mutant-cfb976e1412f/lib/mutant/expression.rb:135:in `parse': Expression: "AdminControllerTest#an_error_is_displayed_when_the_e-mail_in_a_password_reset_is_not_found" is not valid (Mutant::Expression::InvalidExpressionError)

It seems that removing the hyphen from the test name resolves the problem. I also can see messages of the form:

Class#name from: NewRelic::Agent::Samplers::VMSampler returned :vm. Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil

This is probably unrelated to the integration itself and should be reported as a separate issue. @mbj could you confirm this, please?

I also experimented with MiniTest (without Rails) and when I define my tests with #test (i.e. test 'this should do something' instead of def test_this_should_do_something) I get errors like:

test 'positive? works for positive values' do
end

# results in /private/tmp/mutant/test/arithmetic_test.rb:10:in `test': wrong number of arguments (1 for 2) (ArgumentError)

test '#positive? works for positive values' do
end
# results in /private/tmp/mutant/test/arithmetic_test.rb:10:in `test': unknown command '#' (ArgumentError)

The example code is available as a gist - https://gist.github.com/grn/f4bef8dbc1b3d20550ff.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 22, 2015

Owner

Class#name from: NewRelic::Agent::Samplers::VMSampler returned :vm. Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil

This is from the new relic agent that violates the basic ruby invariant. Mutant uses this invariant to detect subjects, and warns when its violated. So those projects (there had been more like this) can get notified to fix their behavior. Mutant ignores such classes or modules from subject matching.

Owner

mbj commented May 22, 2015

Class#name from: NewRelic::Agent::Samplers::VMSampler returned :vm. Fix your lib to follow normal ruby semantics!
{Module,Class}#name should return resolvable constant name as String or nil

This is from the new relic agent that violates the basic ruby invariant. Mutant uses this invariant to detect subjects, and warns when its violated. So those projects (there had been more like this) can get notified to fix their behavior. Mutant ignores such classes or modules from subject matching.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 22, 2015

Owner

I hope to find time over the weekend to finally check this PR out and help to fix the last remaining problems.

Owner

mbj commented May 22, 2015

I hope to find time over the weekend to finally check this PR out and help to fix the last remaining problems.

@gregnavis

This comment has been minimized.

Show comment
Hide comment
@gregnavis

gregnavis May 25, 2015

@kbrock I'm not aware of anything like better specs (I cannot guarantee it doesn't exist though). Because of that I think you should expect much less consistency in MiniTest tests. I guess that the tests in the wild don't start with a method name and a behaviour description (e.g. '#positive? works for positive values'). Here's a list of things that I think may be important for the integration:

  1. two ways to define a test (def test_xyz and test 'xyz')
  2. two conventions for test classes - TestArithmetic (the MiniTest-convention) and ArithmeticTest (the Rails convention); the files are named test_arithmetic.rb and arithmetic_test.rb respectively
  3. in Rails, the test cases inherit from ActiveSupport::TestCase or ActiveController::TestCase; the inheritance hierarchy is ActiveController::TestCase < ActiveSupport::TestCase < Minitest::Test
  4. there's MiniTest::Spec which resembles RSpec but with some differences; I can't give you any details about this as I don't use it

I don't think we need to address all the points above. In my opinion it's better to get started and support the most basic functionality (e. g. supporting both def test_xyz and test 'xyz').

I can see two solutions to the problem of selecting appropriate test cases:

  1. requiring the names of test methods to have an appropriate form (e.g. ##{method_name} #{behaviour})
  2. adding some kind of annotations to the tests, perhaps in comments

gregnavis commented May 25, 2015

@kbrock I'm not aware of anything like better specs (I cannot guarantee it doesn't exist though). Because of that I think you should expect much less consistency in MiniTest tests. I guess that the tests in the wild don't start with a method name and a behaviour description (e.g. '#positive? works for positive values'). Here's a list of things that I think may be important for the integration:

  1. two ways to define a test (def test_xyz and test 'xyz')
  2. two conventions for test classes - TestArithmetic (the MiniTest-convention) and ArithmeticTest (the Rails convention); the files are named test_arithmetic.rb and arithmetic_test.rb respectively
  3. in Rails, the test cases inherit from ActiveSupport::TestCase or ActiveController::TestCase; the inheritance hierarchy is ActiveController::TestCase < ActiveSupport::TestCase < Minitest::Test
  4. there's MiniTest::Spec which resembles RSpec but with some differences; I can't give you any details about this as I don't use it

I don't think we need to address all the points above. In my opinion it's better to get started and support the most basic functionality (e. g. supporting both def test_xyz and test 'xyz').

I can see two solutions to the problem of selecting appropriate test cases:

  1. requiring the names of test methods to have an appropriate form (e.g. ##{method_name} #{behaviour})
  2. adding some kind of annotations to the tests, perhaps in comments
@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj May 25, 2015

Owner

Do you have a suggestion for removing the hardcoded directory name (i.e.: "test") from the tests?
Maybe requiring -I test for MiniTest is our best route.

@kbrock As long we have tight control when the lib/app under tests gets loaded and when the tests get loaded / executed I'm fine with that.

To be explicit, we need the following to be separated:

  • Loading the application / lib
  • Loading the test framework and the tests
  • Executing a subset of the tests.

Separated in sense of: Mutant can trigger these stages explicitly.

Owner

mbj commented May 25, 2015

Do you have a suggestion for removing the hardcoded directory name (i.e.: "test") from the tests?
Maybe requiring -I test for MiniTest is our best route.

@kbrock As long we have tight control when the lib/app under tests gets loaded and when the tests get loaded / executed I'm fine with that.

To be explicit, we need the following to be separated:

  • Loading the application / lib
  • Loading the test framework and the tests
  • Executing a subset of the tests.

Separated in sense of: Mutant can trigger these stages explicitly.

mbj referenced this pull request in saturnflyer/surrounded May 31, 2015

@davydovanton davydovanton referenced this pull request Jun 22, 2015

Closed

Try out mutant #2403

@backus

This comment has been minimized.

Show comment
Hide comment
@backus

backus Jun 29, 2015

Contributor

Did this PR die?

Contributor

backus commented Jun 29, 2015

Did this PR die?

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock Jun 29, 2015

Contributor

@backus thanks. ping taken

Contributor

kbrock commented Jun 29, 2015

@backus thanks. ping taken

@dkubb

This comment has been minimized.

Show comment
Hide comment
@dkubb

dkubb Jun 29, 2015

Collaborator

@kbrock FYI my offer to work with you on this PR is still open :) feel free to add me as a committer to your branch. I could do a few simple things like rebase this on top of master and fix conflicts as well as help test this out.

Collaborator

dkubb commented Jun 29, 2015

@kbrock FYI my offer to work with you on this PR is still open :) feel free to add me as a committer to your branch. I could do a few simple things like rebase this on top of master and fix conflicts as well as help test this out.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 6, 2015

Owner

@kbrock Any chance you can give me and @dkubb commit access to your repo fork, so we can begin to fix work on this?

Owner

mbj commented Jul 6, 2015

@kbrock Any chance you can give me and @dkubb commit access to your repo fork, so we can begin to fix work on this?

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock Jul 6, 2015

Contributor

@mbj @dkubb done

Contributor

kbrock commented Jul 6, 2015

@mbj @dkubb done

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 6, 2015

Owner

@kbrock cool. Thx. First we'll have to rebase this as the integration API changed slightly. Are you okay with suashing your commits? I'll keep your authorship intact, unless we significantly rewrite things.

Owner

mbj commented Jul 6, 2015

@kbrock cool. Thx. First we'll have to rebase this as the integration API changed slightly. Are you okay with suashing your commits? I'll keep your authorship intact, unless we significantly rewrite things.

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock Jul 6, 2015

Contributor

@mbj I will squash right now. thnx

Contributor

kbrock commented Jul 6, 2015

@mbj I will squash right now. thnx

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 6, 2015

Owner

@mbj I will squash right now. thnx

Cool. Thx. BTW this is the commit with the breaking API change, you have to accept config an instance of Mutant::Config on Integration#new: d03e7fb

Owner

mbj commented Jul 6, 2015

@mbj I will squash right now. thnx

Cool. Thx. BTW this is the commit with the breaking API change, you have to accept config an instance of Mutant::Config on Integration#new: d03e7fb

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock Jul 6, 2015

Contributor

@mbj yea - I remember wanting access to that - good call. I don't have time today to look into this. let me know how far you get.

Contributor

kbrock commented Jul 6, 2015

@mbj yea - I remember wanting access to that - good call. I don't have time today to look into this. let me know how far you get.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 6, 2015

Owner

@mbj yea - I remember wanting access to that - good call. I don't have time today to look into this. let me know how far you get.

My policy is to always wait passing down objects till there is a use case, later in the commit history there is the introduction of a configurable Mutant::Expression::Parser object. In d647563 this happened.

It also allows to have extension specific expressions.

Owner

mbj commented Jul 6, 2015

@mbj yea - I remember wanting access to that - good call. I don't have time today to look into this. let me know how far you get.

My policy is to always wait passing down objects till there is a use case, later in the commit history there is the introduction of a configurable Mutant::Expression::Parser object. In d647563 this happened.

It also allows to have extension specific expressions.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

I just rebased the minitest branch by @kbrock on master and fixed the API usage inconsistencies.

To get a better oversight on minitest spec styles (to find a way to support them all), could the minitest users listening to this please name projects that qualify for:

  • Small
  • Very high test coverage (ideally 100% mutation coverage)
  • Uses an ideomatic minitest style consistently

Ideally we have coverage for all major minitest styles. To turn these into corpus tests for mutant.

Owner

mbj commented Jul 7, 2015

I just rebased the minitest branch by @kbrock on master and fixed the API usage inconsistencies.

To get a better oversight on minitest spec styles (to find a way to support them all), could the minitest users listening to this please name projects that qualify for:

  • Small
  • Very high test coverage (ideally 100% mutation coverage)
  • Uses an ideomatic minitest style consistently

Ideally we have coverage for all major minitest styles. To turn these into corpus tests for mutant.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

@mperham Would the Sidekiq project (or a subset of its namespace) qualify for the requirements above?

Owner

mbj commented Jul 7, 2015

@mperham Would the Sidekiq project (or a subset of its namespace) qualify for the requirements above?

@mperham

This comment has been minimized.

Show comment
Hide comment
@mperham

mperham Jul 7, 2015

@mbj No idea if Sidekiq qualifies. I have good test coverage but not 100%. You are welcome to use it if useful.

mperham commented Jul 7, 2015

@mbj No idea if Sidekiq qualifies. I have good test coverage but not 100%. You are welcome to use it if useful.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

@mbj No idea if Sidekiq qualifies. I have good test coverage but not 100%. You are welcome to use it if useful.

Does it use a consistent minitest style? That would qualify as representative for one of the major ways to use minitest, convention wise I mean.

Owner

mbj commented Jul 7, 2015

@mbj No idea if Sidekiq qualifies. I have good test coverage but not 100%. You are welcome to use it if useful.

Does it use a consistent minitest style? That would qualify as representative for one of the major ways to use minitest, convention wise I mean.

@mperham

This comment has been minimized.

Show comment
Hide comment
@backus

This comment has been minimized.

Show comment
Hide comment
@backus

backus Jul 7, 2015

Contributor

To get a better oversight on minitest spec styles (to find a way to suppor them all), could the minitest users listening to this please name projects that qualify for:

  • Small
  • Very high test coverage (ideally 100% mutation coverage)
  • Uses an ideomatic minitest style consistently
  • Ideally we have coverage for all major minitest styles.

@zenspider might have some good input here since he is the largest contributor listed on github for minitest, ruby2ruby, and heckle.

Contributor

backus commented Jul 7, 2015

To get a better oversight on minitest spec styles (to find a way to suppor them all), could the minitest users listening to this please name projects that qualify for:

  • Small
  • Very high test coverage (ideally 100% mutation coverage)
  • Uses an ideomatic minitest style consistently
  • Ideally we have coverage for all major minitest styles.

@zenspider might have some good input here since he is the largest contributor listed on github for minitest, ruby2ruby, and heckle.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

You tell me. https://github.com/mperham/sidekiq/tree/master/test

I've no idea, as I never did much with minitest. Thats my problem here.

I do not know what to expect. From reading peoples responses there seem to be at least two styles:

  • Test prefixed classes with test_ prefixed methods
  • Something else with an rspec-a-like syntax, that might generate the above internally, but potentially generates method names that can only be meta-programmed but not be literally present in a ruby source file.

My problem is: Is the listing from above complete, or not?

Owner

mbj commented Jul 7, 2015

You tell me. https://github.com/mperham/sidekiq/tree/master/test

I've no idea, as I never did much with minitest. Thats my problem here.

I do not know what to expect. From reading peoples responses there seem to be at least two styles:

  • Test prefixed classes with test_ prefixed methods
  • Something else with an rspec-a-like syntax, that might generate the above internally, but potentially generates method names that can only be meta-programmed but not be literally present in a ruby source file.

My problem is: Is the listing from above complete, or not?

@mperham

This comment has been minimized.

Show comment
Hide comment
@mperham

mperham Jul 7, 2015

Afaik, your list is complete.

Traditional:
https://github.com/mperham/sidekiq/blob/master/test/test_web_helpers.rb
Spec: https://github.com/mperham/sidekiq/blob/master/test/test_manager.rb

Sidekiq overwhelmingly uses spec but there's one or two uses of traditional.

On Tue, Jul 7, 2015 at 3:24 PM, Markus Schirp notifications@github.com
wrote:

You tell me. https://github.com/mperham/sidekiq/tree/master/test

I've no idea, as I never did much with minitest. Thats my problem here.

I do not know what to expect. From reading peoples responses there seem
to be at least two styles:

  • Test suffixed classes with test_ prefixed methods
  • Something else with an rspec-a-like syntax, that might generate the
    above internally, but potentially generates method names that can only be
    meta-programmed but not be literally present in a ruby source file.

My problem is: Is the listing from above complete, or not?


Reply to this email directly or view it on GitHub
#330 (comment).

mperham commented Jul 7, 2015

Afaik, your list is complete.

Traditional:
https://github.com/mperham/sidekiq/blob/master/test/test_web_helpers.rb
Spec: https://github.com/mperham/sidekiq/blob/master/test/test_manager.rb

Sidekiq overwhelmingly uses spec but there's one or two uses of traditional.

On Tue, Jul 7, 2015 at 3:24 PM, Markus Schirp notifications@github.com
wrote:

You tell me. https://github.com/mperham/sidekiq/tree/master/test

I've no idea, as I never did much with minitest. Thats my problem here.

I do not know what to expect. From reading peoples responses there seem
to be at least two styles:

  • Test suffixed classes with test_ prefixed methods
  • Something else with an rspec-a-like syntax, that might generate the
    above internally, but potentially generates method names that can only be
    meta-programmed but not be literally present in a ruby source file.

My problem is: Is the listing from above complete, or not?


Reply to this email directly or view it on GitHub
#330 (comment).

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

Afaik, your list is complete.

Thx that helps a lot. Can I assume there is an 1:1 mapping between Test prefixed class/module with one in the tested code?

So Foo is in the traditional style always tested via TestFoo ?

Owner

mbj commented Jul 7, 2015

Afaik, your list is complete.

Thx that helps a lot. Can I assume there is an 1:1 mapping between Test prefixed class/module with one in the tested code?

So Foo is in the traditional style always tested via TestFoo ?

@mperham

This comment has been minimized.

Show comment
Hide comment
@mperham

mperham Jul 7, 2015

Yeah, I'd think so. Note I use TestManager, not TestSidekiqManager, to test Sidekiq::Manager so it's not directly mappable.

mperham commented Jul 7, 2015

Yeah, I'd think so. Note I use TestManager, not TestSidekiqManager, to test Sidekiq::Manager so it's not directly mappable.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

Better question: How could mutant infer that this test:

https://github.com/mperham/sidekiq/blob/master/test/test_manager.rb

Is meaning to specify behavior of this class:

https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/manager.rb

I actually could trace normal test execution to determine this. But I do not like this for various reasons as it has a fundamental downside I typically refer to as "implicit coverage". I plan to support such "implicit" coverage as a side effect of another planned feature.

But ideally we find a way to explicitly specify this relationship. Does minitest in "spec" style have support for metadata. So could I write a test like this:

class TestManager < Sidekiq::Test
  describe 'manager', specifies: Sidekiq::Manager do
    ...

?

Owner

mbj commented Jul 7, 2015

Better question: How could mutant infer that this test:

https://github.com/mperham/sidekiq/blob/master/test/test_manager.rb

Is meaning to specify behavior of this class:

https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/manager.rb

I actually could trace normal test execution to determine this. But I do not like this for various reasons as it has a fundamental downside I typically refer to as "implicit coverage". I plan to support such "implicit" coverage as a side effect of another planned feature.

But ideally we find a way to explicitly specify this relationship. Does minitest in "spec" style have support for metadata. So could I write a test like this:

class TestManager < Sidekiq::Test
  describe 'manager', specifies: Sidekiq::Manager do
    ...

?

@mperham

This comment has been minimized.

Show comment
Hide comment
@mperham

mperham Jul 7, 2015

I don't think you can. I can see how you could rely on rspec's describe(class) for that but I don't know if minitest has the equivalent.

mperham commented Jul 7, 2015

I don't think you can. I can see how you could rely on rspec's describe(class) for that but I don't know if minitest has the equivalent.

@mperham

This comment has been minimized.

Show comment
Hide comment
@mperham

mperham Jul 7, 2015

I'd be happy to mark up the Sidekiq test suite to support Mutant if you gave minitest users an annotation to use, e.g.

class TestManager < Minitest::Test
  exercises Sidekiq::Manager

mperham commented Jul 7, 2015

I'd be happy to mark up the Sidekiq test suite to support Mutant if you gave minitest users an annotation to use, e.g.

class TestManager < Minitest::Test
  exercises Sidekiq::Manager
@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

I don't think you can. I can see how you could rely on rspec's describe(class) for that but I don't know if minitest has the equivalent.

And thats the key point. This explicit mapping is easy for rspec. And it has a big leverage for execution speed and coverage correctness.

Minitest integration always failed in the past for the lack of this specific mapping metadata. Unless I get the time to do the tracing based selection I see no way how we could do a good rspec equivalent mutant integration with minitest - unless we add this metadata to minitest.

And I'm not an minitest user both OSS and commercial - so there was not enough incentive to actually do one of the proposed solutions. Just running all tests per mutation is not an option. Both because implicit coverage and runtime.

Owner

mbj commented Jul 7, 2015

I don't think you can. I can see how you could rely on rspec's describe(class) for that but I don't know if minitest has the equivalent.

And thats the key point. This explicit mapping is easy for rspec. And it has a big leverage for execution speed and coverage correctness.

Minitest integration always failed in the past for the lack of this specific mapping metadata. Unless I get the time to do the tracing based selection I see no way how we could do a good rspec equivalent mutant integration with minitest - unless we add this metadata to minitest.

And I'm not an minitest user both OSS and commercial - so there was not enough incentive to actually do one of the proposed solutions. Just running all tests per mutation is not an option. Both because implicit coverage and runtime.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

I'd be happy to mark up the Sidekiq test suite to support Mutant if you gave minitest users an annotation to use, e.g.

Yeah, these are project specific integrations. I did plenty of them already. But not the generic one. Project specific is easy and can be done "tomorrow" ;)

Owner

mbj commented Jul 7, 2015

I'd be happy to mark up the Sidekiq test suite to support Mutant if you gave minitest users an annotation to use, e.g.

Yeah, these are project specific integrations. I did plenty of them already. But not the generic one. Project specific is easy and can be done "tomorrow" ;)

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

@kbrock, @mperham Lets do the following: We simply assume all minitest mutant users will do this exercises TheSubjecOfTest. If they are not willing to annotate: Mutant will not work for them.

That will at least get us started and unblocked.

Owner

mbj commented Jul 7, 2015

@kbrock, @mperham Lets do the following: We simply assume all minitest mutant users will do this exercises TheSubjecOfTest. If they are not willing to annotate: Mutant will not work for them.

That will at least get us started and unblocked.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 7, 2015

Owner

And instead for having to do a constant reference in this exercises thing, lets allow to pass N mutant expressions.

So you can actually declare a test to match * (any subject under mutation), if you really want this painful way :P

Owner

mbj commented Jul 7, 2015

And instead for having to do a constant reference in this exercises thing, lets allow to pass N mutant expressions.

So you can actually declare a test to match * (any subject under mutation), if you really want this painful way :P

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Jul 8, 2015

Owner

I pushed my latest version that should pass the specs and supports an extension to minitest specifying the subject of coverage explicitly.

This does the job for the test app mutants tests run against:

class TestAppTest < Minitest::Test
  def self.cover(expression)
    @expression = expression
  end

  def self.cover_expression
    unless @expression
      fail "Cover expression for #{self} is not specified"
    end

    @expression
  end
end

class TestApp::LiteralTest < TestAppTest
  cover 'TestApp::Literal*'

  def test_command
    object = ::TestApp::Literal.new
    subject = object.command('x')

    assert_equal object, subject
  end

  def test_string
    object = ::TestApp::Literal.new
    subject = object.string

    assert_equal 'string', subject
  end
end

I'll try to get a subset of sidekiq passing against this branch on CI later this week.

Owner

mbj commented Jul 8, 2015

I pushed my latest version that should pass the specs and supports an extension to minitest specifying the subject of coverage explicitly.

This does the job for the test app mutants tests run against:

class TestAppTest < Minitest::Test
  def self.cover(expression)
    @expression = expression
  end

  def self.cover_expression
    unless @expression
      fail "Cover expression for #{self} is not specified"
    end

    @expression
  end
end

class TestApp::LiteralTest < TestAppTest
  cover 'TestApp::Literal*'

  def test_command
    object = ::TestApp::Literal.new
    subject = object.command('x')

    assert_equal object, subject
  end

  def test_string
    object = ::TestApp::Literal.new
    subject = object.string

    assert_equal 'string', subject
  end
end

I'll try to get a subset of sidekiq passing against this branch on CI later this week.

@mbj

This comment has been minimized.

Show comment
Hide comment
@mbj

mbj Sep 22, 2015

Owner

Closing this PR in favor of #445.

Owner

mbj commented Sep 22, 2015

Closing this PR in favor of #445.

@mbj mbj closed this Sep 22, 2015

@kbrock

This comment has been minimized.

Show comment
Hide comment
@kbrock

kbrock Sep 22, 2015

Contributor

👍 thanks

Contributor

kbrock commented Sep 22, 2015

👍 thanks

@kbrock kbrock deleted the kbrock:minitest branch Sep 22, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment