Add test minitest integration #445

Open
wants to merge 2 commits into
from

Projects

None yet
@mbj
Owner
mbj commented Sep 22, 2015

[fix #92]

References: #330.

@mbj mbj referenced this pull request Sep 22, 2015
Closed

Introduce Minitest integration #330

@kbrock
Contributor
kbrock commented Sep 22, 2015

could you add a check list or some ideas what may need to be added here?

@mbj
Owner
mbj commented Sep 22, 2015

could you add a check list or some ideas what may need to be added here?

Will do. Right now it needs someone who tests it on a non trivial project, that has real unit tests. That will likely be myself. And I'm blocked by time right now.

@mperham
mperham commented Sep 22, 2015

I'll play with it today on sidekiq's suite.

@mbj
Owner
mbj commented Sep 22, 2015

I'll play with it today on sidekiq's suite.

I tried with sidekiq for maybe 10min back one month or so. But could not isolate a test / subject set that is a "real unit" test for testing the rspec integration.

By that I mean that most of the tests (and maybe even the global setup, I do not recall precicely) require redis / and or cause side effects.

You should start with -j1 to limit concurrency. I had some local commits that made the sidekiq test suite more accessible for mutant, but I lost them in the chaos that is my current live (sick for 4+ weeks).

Conceptually I think that sidekiq might not be the best project to verify the mutant-minitest integration. Since its domain is mostly managing nondeterministic things (threads, network, ...).

Not that I think sidekiq and mutant "cannot" play well together, but I think its at too big step. It would be easier to start with a smaller lib that does not have IO etc to validate the integration is correct, before using the integration on something bigger and running into false positives that burn time.

@dogweather

How would I try this out?

@mbj
Owner
mbj commented Nov 3, 2015

How would I try this out?

No time at all sorry. Someone with a less "global state hitting" minitest project should try.

@dogweather

I mean just, how would I invoke this from the command line? minitest in place of rspec?

@mbj
Owner
mbj commented Nov 3, 2015

I mean just, how would I invoke this from the command line? minitest in place of rspec?

--use minitest yes. And use this branch.

@dogweather

I gave it a try on a Rails project, and it executed part-way but then had an error:

RAILS_ENV=test bundle exec mutant -r ./config/environment -I test --use minitest Agency

Output:

Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [Agency]>
Integration:     Mutant::Integration::Minitest
Expect Coverage: 100.00%
Jobs:            8
Includes:        ["test"]
Requires:        ["./config/environment"]
Subjects:        14
Mutations:       729
Kills:           0
Alive:           0
Runtime:         0.00s
Killtime:        0.00s
Overhead:        Inf%
Coverage:        100.00%
Expected:        100.00%
Active subjects: 0
/Users/robb/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/bundler/gems/mutant-36269e744bcd/lib/mutant/integration/minitest.rb:60:in `expression_syntax': undefined method `cover_expression' for BusinessesControllerTest:Class (NoMethodError)
    from /Users/robb/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/bundler/gems/mutant-36269e744bcd/lib/mutant/integration/minitest.rb:138:in `construct_test'
    from /Users/robb/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/bundler/gems/mutant-36269e744bcd/lib/mutant/integration/minitest.rb:123:in `block in all_tests_index'
    from /Users/robb/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/bundler/gems/mutant-36269e744bcd/lib/mutant/integration/minitest.rb:122:in `each'
    from /Users/robb/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/bundler/gems/mutant-36269e744bcd/lib/mutant/integration/minitest.rb:122:in `each_with_object'
    from /Users/robb/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/bundler/gems/mutant-36269e744bcd/lib/mutant/integration/minitest.rb:122:in `all_tests_index'
    from /Users/robb/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/memoizable-0.4.2/lib/memoizable/method_builder.rb:117:in `call'

(etc.)

@mbj
Owner
mbj commented Nov 3, 2015

Likely this is triggered from missing autoload triggers. Typical rails symptom. Try to explicitly require the file that defines that method. Or ideally test on a non rails project.

@dogweather

It seems that Mutant expects the missing method #cover_expression. (?) I can't find any references to that online. What should be providing that?

        # Cover expression syntaxes
        #
        # @return [Array<String>]
        #
        # @api private
        def expression_syntax
          klass.cover_expression
        end

https://github.com/mbj/mutant/blob/feature/minitest-integration/lib/mutant/integration/minitest.rb#L54-L61

@mbj
Owner
mbj commented Nov 3, 2015

@dogweather Ahh, I remember see the discussion in #330.

@mbj mbj added the enhancement label Nov 16, 2015
@mbj
Owner
mbj commented Dec 20, 2015

Depends on #508 from software rusting.

@mbj mbj removed the in-progress label Dec 20, 2015
@mbj
Owner
mbj commented Dec 20, 2015

@dkubb Ready for review. Includes #508 so should not be merged before.

@dkubb dkubb and 1 other commented on an outdated diff Dec 21, 2015
lib/mutant/integration/minitest.rb
+ end
+
+ # Call test integration
+ #
+ # @param [Array<Tests>] tests
+ #
+ # @return [Result::Test]
+ #
+ # rubocop:disable MethodLength
+ def call(tests)
+ test_cases = tests.map(&all_tests_index.method(:fetch)).to_set
+
+ output = StringIO.new
+ reporter = ::Minitest::SummaryReporter.new(output)
+
+ start = Time.now
@dkubb
dkubb Dec 21, 2015 Collaborator

I would align the assignment operators here.

@backus
backus Dec 21, 2015 Contributor

Instead of recording Time.now and doing Time.now - start later could you use Benchmark.realtime?

@dkubb
dkubb Dec 28, 2015 Collaborator

@backus I agree, that is a much better idea.

Another nice side benefit is that you eliminate some extra noise from StringIO#rewind and StringIO#read since you could wrap the passed = test_cases.all? { |test| test.call(reporter) } line only.

@dkubb dkubb and 1 other commented on an outdated diff Dec 21, 2015
lib/mutant/integration/minitest.rb
+ #
+ # @return [Array<TestCase>]
+ def all_test_cases
+ ::Minitest::Runnable.runnables.flat_map(&method(:test_case))
+ end
+
+ # Turn a minitest runnable into its test cases
+ #
+ # Intentional utility method.
+ #
+ # @param [Object] runnable
+ #
+ # @return [Array<TestCase>]
+ def test_case(runnable)
+ runnable.runnable_methods.each_with_object([]) do |method_name, test_cases|
+ next unless method_name.start_with?(TEST_METHOD_PREFIX)
@dkubb
dkubb Dec 21, 2015 Collaborator

I wonder if you couldn't use Enumerable#select to match runnable methods with a given prefix, allowing you to just use map instead of Enumerable#each_with_object ?

@backus
backus Dec 22, 2015 Contributor

Adding onto @dkubb's comment this seems to be the only TEST_METHOD_PREFIX is used so that could be changed to TEST_METHOD_PATTERN = /\Atest_/.freeze and instead of Enumerable#select you could just do a grep(TEST_METHOD_PATTERN).map

@dkubb
dkubb Dec 28, 2015 Collaborator

Also you wouldn't even need to use map since grep with a block acts the same as map. This method could be rewritten as:

TEST_METHOD_PREFIX = /\Atest_/.freeze

# ...

def test_case(runnable)
  runnable.runnable_methods.grep(TEST_METHOD_PREFIX) do |method_name|
    TestCase.new(runnable, method_name)
  end
end

Or if you wanted to get fancy with Method#curry:

TEST_METHOD_PREFIX = /\Atest_/.freeze

# ...

def test_case(runnable)
  test_case = TestCase.method(:new).curry(2)[runnable]
  runnable.runnable_methods.grep(TEST_METHOD_PREFIX, &test_case)
end
@dkubb
dkubb Dec 28, 2015 Collaborator

@backus oh, nice find! I'd guess we can just get by with using Enumerable#map here instead.

@dkubb dkubb commented on an outdated diff Dec 21, 2015
mutant-minitest.gemspec
@@ -0,0 +1,24 @@
+# encoding: utf-8
+#
+require File.expand_path('../lib/mutant/version', __FILE__)
+
+Gem::Specification.new do |gem|
+ gem.name = 'mutant-minitest'
+ gem.version = Mutant::VERSION.dup
+ gem.authors = ['Markus Schirp']
+ gem.email = ['mbj@schirp-dso.com']
@dkubb
dkubb Dec 21, 2015 Collaborator

You could use %w[...] here.

@dkubb dkubb and 1 other commented on an outdated diff Dec 21, 2015
mutant-minitest.gemspec
+ gem.name = 'mutant-minitest'
+ gem.version = Mutant::VERSION.dup
+ gem.authors = ['Markus Schirp']
+ gem.email = ['mbj@schirp-dso.com']
+ gem.description = 'Minitest integration for mutant'
+ gem.summary = gem.description
+ gem.homepage = 'https://github.com/mbj/mutant'
+ gem.license = 'MIT'
+
+ gem.require_paths = %w[lib]
+ gem.files = `git ls-files -- lib/mutant/integration/minitest.rb`.split("\n")
+ gem.test_files = `git ls-files -- spec/integration/mutant/minitest.rb`.split("\n")
+ gem.extra_rdoc_files = %w[TODO LICENSE]
+
+ gem.add_runtime_dependency('mutant', "~> #{gem.version}")
+ gem.add_runtime_dependency('minitest', '>= 5.3.0')
@dkubb
dkubb Dec 21, 2015 Collaborator

Can you use ~> here, or ~> and >=? I generally would not recommend using >= by itself; especially when you don't control the upstream gem.

@mbj
mbj Dec 21, 2015 Owner

Good catch.

@dkubb dkubb and 1 other commented on an outdated diff Dec 21, 2015
mutant-minitest.gemspec
+ gem.authors = ['Markus Schirp']
+ gem.email = ['mbj@schirp-dso.com']
+ gem.description = 'Minitest integration for mutant'
+ gem.summary = gem.description
+ gem.homepage = 'https://github.com/mbj/mutant'
+ gem.license = 'MIT'
+
+ gem.require_paths = %w[lib]
+ gem.files = `git ls-files -- lib/mutant/integration/minitest.rb`.split("\n")
+ gem.test_files = `git ls-files -- spec/integration/mutant/minitest.rb`.split("\n")
+ gem.extra_rdoc_files = %w[TODO LICENSE]
+
+ gem.add_runtime_dependency('mutant', "~> #{gem.version}")
+ gem.add_runtime_dependency('minitest', '>= 5.3.0')
+
+ gem.add_development_dependency('bundler', '~> 1.3', '>= 1.3.5')
@dkubb
dkubb Dec 21, 2015 Collaborator

I believe bundler's version has changed a bit since this was written.

@mbj
mbj Dec 21, 2015 Owner

I think the bundler version should not be specified. I'm not aware about any bug this would prevent. I'll drop it here.

@dkubb dkubb commented on an outdated diff Dec 21, 2015
test_app/test/test_helper.rb
@@ -0,0 +1,23 @@
+$LOAD_PATH << File.join(File.dirname(__FILE__), '../lib')
+
+require 'test_app'
+require 'minitest/autorun'
+
+# require spec support files and shared behavior
+Dir[File.expand_path('../{support,shared}/**/*.rb', __FILE__)].each do |file|
@dkubb
dkubb Dec 21, 2015 Collaborator

I would sort the files to make sure they are always required in the same order.

@backus backus commented on the diff Dec 21, 2015
lib/mutant/integration/minitest.rb
+require 'minitest'
+
+# monkey patch for minitest
+module Minitest
+ # Prevent autorun from running tests when the VM closes
+ #
+ # Mutant needs control about the exit status of the VM and
+ # the moment of test execution
+ #
+ # @api private
+ #
+ # @return [nil]
+ def self.autorun
+ end
+
+end # Minitest
@backus
backus Dec 21, 2015 Contributor

Could rewrite this monkey patch on one line with something like

Minitest.define_singleton_method(:autorun) { }
@mbj
mbj Dec 21, 2015 Owner

Interesting idea to shorten it like this. I wounder how we can apply the axioms to deterministically prefer one form over the other. //cc @dkubb.

@dkubb
dkubb Dec 28, 2015 Collaborator

I'm kind of partial to the def form myself. I generally prefer to use built-in language syntax over meta-programming methods, assuming the end result is equal.

@mbj
mbj Dec 29, 2015 Owner

built-in language syntax over meta-programming methods, assuming the end result is equal.

Yes. And the rule of least power supports this as the metaprogramming closure has a much wider scope. Using the primary syntax has "less power" (scope wise) so should be preferred here.

@backus
backus Dec 29, 2015 Contributor

If you want to stick with the primary syntax here you could still cut two lines by instead writing

def Minitest.autorun
end

I know this syntax is usually frowned upon but it might make sense in a context like this. It also has the benefit then of not pinning YARD documentation to the toplevel Minitest namespace.

@mbj
mbj Jan 2, 2016 Owner

Reopening the scope Minitest is not needed technically, but maybe for consistency?

@backus backus and 1 other commented on an outdated diff Dec 21, 2015
lib/mutant/integration/minitest.rb
+ # the moment of test execution
+ #
+ # @api private
+ #
+ # @return [nil]
+ def self.autorun
+ end
+
+end # Minitest
+
+module Mutant
+ class Integration
+ # Minitest integration
+ class Minitest < self
+ TEST_METHOD_PREFIX = 'test_'.freeze
+ TEST_FILE_PATTERN = './test/**/{test_*,*_test}.rb'.freeze
@backus
backus Dec 21, 2015 Contributor

I pulled down the 100 most popular gems depending on minitest to see if the convention was this clear cut. A few differences I noticed:

  • The json gem which is now part of ruby's standard lib uses a "tests" directory which does not match this glob
  • The excon gem uses a "tests" directory then I think their test files have a _tests.rb suffix

Still looking for other deviations right now so I'll probably update this comment.

My conclusion here isn't necessarily that the glob should change though. Instead, I think this indicates that it might be important for to have a --test-pattern flag in the future which makes this configurable.

@mbj
mbj Dec 21, 2015 Owner

Thx for your investigation.

Yeah. These integration specific flags need to happen. Where I think that we even more discovered need for a persistent config file.

@backus
backus Dec 21, 2015 Contributor

No problem. I stopped looking for deviations since it sounds like you agree this needs to be configurable. As one last check I grepped out the gem.test_files setting in each gemspec. Here are the unique entries (minus a few weirdos):

Dir.glob("spec/**/*")
Dir["spec/**/*.rb"]
Dir["test/spec_*.rb"]
Dir["test/test_*.rb"]
`git ls-files -- Appraisals {spec}/*`.split("\n")
`git ls-files -- spec/*`.split("\n")
`git ls-files -- test/*`.split("\n")
`git ls-files -- test`.split($/)
`git ls-files -- {spec}/*`.split("\n")
`git ls-files -- {test,spec,features}/*`.split("\n")
`git ls-files -- {test}/*`.split("\n")
`git ls-files spec/{unit,integration}`.split($/)
gem.files.grep(%r{^(test|spec|features)/.*\.rb})
gem.files.grep(%r{^(test|spec|features)/})
gem.files.grep(%r{^(test|spec|features)/}).grep(%r{/test_[^/]+\.rb$})
gem.files.grep(%r{^spec/})
s.files.select { |path| path =~ /^[spec|tests]\/.*_[spec|tests]\.rb/ }
s.files.select {|path| path =~ /^test\/.*_test.rb/}
spec.files.grep(%r{^(test|spec|features)/})
spec.files.grep(%r{^(test|spec|features|acceptance)/})
spec.files.grep(%r{^(test|spec|features|examples)/})
spec.files.grep(%r{^test/})
@kbrock
Contributor
kbrock commented Dec 21, 2015

Heh. would be cool if rake just took the test file names out of the gemspec.
Meta data to rule them all.

@backus Thanks for all the research

@backus backus and 1 other commented on an outdated diff Dec 21, 2015
lib/mutant/integration/minitest.rb
+ TEST_METHOD_PREFIX = 'test_'.freeze
+ TEST_FILE_PATTERN = './test/**/{test_*,*_test}.rb'.freeze
+
+ register 'minitest'
+
+ # Compose a runnable with test method
+ #
+ # This looks actually like a missing object on minitest implementation.
+ class TestCase
+ include Adamantium, Concord.new(:klass, :test_method)
+
+ # Identification string
+ #
+ # @return [String]
+ def identification
+ "minitest:#{klass}##{test_method}"
@backus
backus Dec 21, 2015 Contributor

Maybe the format here could be extracted to a constant?

@mbj
mbj Dec 29, 2015 Owner

Yes. The format might even be sharable with the rspec implementation.

@mbj
Owner
mbj commented Dec 21, 2015

Heh. would be cool if rake just took the test file names out of the gemspec.
Meta data to rule them all.

Thats also a good idea. But likely I'll not have this in the first release. Also a repo can contain multiple .gemspecs and you might want to select only one. But in general this is a good approach to reduce doubt and to use a single source of truth.

@kbrock
Contributor
kbrock commented Dec 21, 2015

@mbj this is a feature for the testing frameworks themselves. Maybe built into their default rake failes

@backus backus commented on an outdated diff Dec 22, 2015
test_app/Gemfile.minitest-stdlib
@@ -0,0 +1,6 @@
+source 'https://rubygems.org'
+
+gem 'minitest', '~> 5.5.0'
@backus
backus Dec 22, 2015 Contributor

Should this version constraint match the mutant-minitest gemspec? It is '~> 5.5.0' here will be '~> 5.3.0 in the gemspec judging by this thread https://github.com/mbj/mutant/pull/445/files#r48111671

@backus
Contributor
backus commented Dec 22, 2015

The different test_app Gemfiles make me think that looking into using Appraisal might make sense.

@backus
Contributor
backus commented Dec 22, 2015

I noticed there aren't any unit tests for the minitest integration. This surprised me until I realized that running rake metrics:mutant was not yielding any mutations for the minitest integration. This makes sense since mutant integrations are conditionally required based on what you specify in --use and therefore mutant isn't requiring any of the minitest specific code in CI. I had to modify the command to

$ bundle exec mutant --ignore-subject 'Mutant::Meta*' --include lib --require mutant --require mutant/integration/minitest --use rspec --zombie -- 'Mutant::Integration::Minitest*'

and temporarily add minitest to the Gemfile in order to find the mutation coverage relevant to this PR.

@mbj am I missing something or are you still planning on writing a lot of tests for this PR?

@backus
Contributor
backus commented Dec 22, 2015

@mbj I'm done doing a pass over this PR

@mbj
Owner
mbj commented Dec 23, 2015

I noticed there aren't any unit tests for the minitest integration.

Yeah, these have to be present. Curious why --since did not trigger, maybe the integration integration test is eligible for mutation testing so mutant was satisfied.

@backus
Contributor
backus commented Dec 24, 2015

Curious why --since did not trigger

because require 'mutant/integration/minitest' is never invoked

maybe the integration integration test is eligible for mutation testing so mutant was satisfied.

No the integration test specifies mutant: false

@mbj
Owner
mbj commented Dec 29, 2015

The different test_app Gemfiles make me think that looking into using Appraisal might make sense.

The Gemfile does not contain information about the test files, only the .gemspec does.

@dkubb dkubb commented on an outdated diff Dec 31, 2015
test_app/test/test_helper.rb
+
+require 'test_app'
+require 'minitest/autorun'
+
+# require spec support files and shared behavior
+Dir[File.expand_path('../{support,shared}/**/*.rb', __FILE__)].each do |file|
+ require file
+end
+
+class TestAppTest < Minitest::Test
+ def self.cover(expression)
+ @expression = expression
+ end
+
+ def self.cover_expression
+ unless @expression
@dkubb
dkubb Dec 31, 2015 Collaborator

What about using @expression or fail ... instead? The semantics should be the same, but it's less code.

@dkubb
Collaborator
dkubb commented Dec 31, 2015

@mbj I rebased this on top of master (#508 was merged into master btw).

I can do another pass once the flagged issues are addressed.

@mbj
Owner
mbj commented Dec 31, 2015

@mbj I rebased this on top of master (#508 was merged into master btw).

thx. I'll have to do some followup work. Writing mutation covered unit specs mostly. Will ping you once done.

@mbj mbj was assigned by dkubb Dec 31, 2015
@mbj mbj closed this Jan 9, 2016
@mbj
Owner
mbj commented Jan 9, 2016

Github is drunk. It does not catch up with the commit I force pushed here.

@mbj mbj reopened this Jan 9, 2016
@mbj
Owner
mbj commented Jan 9, 2016

Okay, after deleting the branch and re-pushing it GH now allowed me to repoen the PR.

@gavinlaking

Not sure how useful this feedback is, but this does not seem to work with Minitest::Spec (which I use for Vedeu (https://github.com/gavinlaking/vedeu):

This is output from my mutant spike branch. (https://github.com/gavinlaking/vedeu/tree/spike/mutant)

Source/vedeu:spike/mutant$ rake mutant
Gemfile Found, running with bundle exec
bundle exec mutant -r ./lib/vedeu -I lib -I test --require test_helper --use minitest 'Vedeu'
/home/gavinlaking/Source/vedeu/test/test_helper.rb:89:in `singleton class': undefined method `describe' for class `#<Class:Minitest::Spec>' (NameError)
    from /home/gavinlaking/Source/vedeu/test/test_helper.rb:87:in `<class:Spec>'
    from /home/gavinlaking/Source/vedeu/test/test_helper.rb:80:in `<module:MiniTest>'
    from /home/gavinlaking/Source/vedeu/test/test_helper.rb:78:in `<top (required)>'
    ...

If I can provide further details to help, please give me a shout.\

Gav

@mbj
Owner
mbj commented Feb 28, 2016

Not sure how useful this feedback is, but this does not seem to work with Minitest::Spec (which I use for Vedeu (https://github.com/gavinlaking/vedeu):

Thx for the report, this is a known limitation. Easy to resolve through. I hope I get more OSS time to focus on such integrations. Thx for the report.

@mvz
Contributor
mvz commented May 23, 2016

I would love to use mutant on some of my projects that use minitest/spec. Is there any way I can help this move forward?

@mbj
Owner
mbj commented May 23, 2016

I would love to use mutant on some of my projects that use minitest/spec. Is there any way I can help this move forward?

Test this branch. Or inject cash so I can justify to do it on my time.

@kbrock
Contributor
kbrock commented May 23, 2016

is this close enough to just merge and say it is experimental?

@mvz
Contributor
mvz commented May 24, 2016

I'm testing this branch now. Some preliminary results (not all related to this branch I think):

  • I need to --include test on the command line, the equivalent of which does not seem to be needed for rspec.
  • I need to define cover and cover_expression. Shouldn't these be defined in mutant-minitest?
  • It seems mutant is using more tests than I expect based on the cover information I provide. I'm still investigating what's going on here.
@mbj
Owner
mbj commented May 24, 2016

It seems mutant is using more tests than I expect based on the cover information I provide. I'm still investigating what's going on here.

You are expected to define cover_expression per test. You can go arbitrarily fine or coarse grained as you want.

I need to --include test on the command line, the equivalent of which does not seem to be needed for rspec.

I think this is fair as mutant could not know which directory to use off hand?

I need to define cover and cover_expression. Shouldn't these be defined in mutant-minitest?

Mutant minitest could not infer which tests covers what subject, because minitest "stock" does not carry that metadata, unlike rspec.

@mbj
Owner
mbj commented May 30, 2016 edited

Summary: I'd like to merge this, but not release as a gem until I get a good external verification. This way its keept up-to date with advances in master.

To merge this I need:

  1. Someone with a real minitest project testing it
  2. A small 100% mutation tested minitest project that we can include in the corpus. 1) and 2) can be the same.

I'm open to do the work by myself, but I need a minitest project maintainer who is willing to accept and keep 100% mutation coverage. I'd do the work for covering the project. Or ideally I help (screenshare, hangout, email, github etc).

That project should ideally be:

  • Small
  • Stable
  • Does not need side effectful IO in tests (DB, remote APIs etc).

If someone is willing to step up. Shout at me.

@arthurnn

@mbj I work at GitHub, and i have been following this project for a while. We use Minitest, but our code base is pretty big so I don't know how this would work there. Anyways, I will give it a shoot this week and let you know.
Thanks

@mbj
Owner
mbj commented May 30, 2016

@mbj I work at GitHub, and i have been following this project for a while. We use Minitest, but our code base is pretty big so I don't know how this would work there. Anyways, I will give it a shoot this week and let you know.

Can you identify a small subset that has the required properties? Its generally possible to shard mutant execution to archive decent execution time even on big projects, also the --since flag can subset to only touched code. Both are valid strategies to get bigger code bases covered.

To note: Mutant (in master) is 100% self covering, while certainly not the size of github, its a proof of concept it "can" be done.

@mbj
Owner
mbj commented May 30, 2016

@arthurnn Does github have a public minitested library with mostly in-memory tests? That might be the ideal integration candidate.

@mbj
Owner
mbj commented May 30, 2016

@arthurnn OT: We have a "public" (As in anyone who wants gets an invite) Slack channel useful for more bandwith discussion, feel free to let me know if you want an invite for more hands-on mutant support.

@chastell
Contributor
chastell commented May 31, 2016 edited

Would Lovely Rufus and/or signore work? Both ~200 lines of code + ~500 lines of tests, Minitest-tested, RuboCop+Reek-clean (with some whitelisting) and I’d love to have them mutation-covered.

@mbj
Owner
mbj commented May 31, 2016

@chastell Both look okay. Can you confirm none of them has mandatory IO in the unit tests?

@chastell
Contributor
chastell commented Jun 1, 2016

Gaah, signore does do IO, sorry. Lovely Rufus doesn’t (well, one test reads a fixture, but doesn’t write anything).

@mbj
Owner
mbj commented Jun 1, 2016

one test reads a fixture, but doesn’t write anything).

Thats fair, its "intend to be idempotent" so fine for parallelized mutation testing.

@mvz
Contributor
mvz commented Jun 17, 2016

I have added mutant to my minitest-tested project gir_ffi here: https://github.com/mvz/gir_ffi/pull/71/files

@chastell you're welcome to use the code there to add mutant to lovely_rufus.

@mbj
Owner
mbj commented Jun 17, 2016

@mvz Thx for that. I'm time limited by an upcoming biological fork (AKA awaiting next baby, due any moment). Once forked I'll be able to pick up work on mutant. Also I'll reach out to @arthurnn in detail in the hope to get some very representative project using mutant.

@mvz
Contributor
mvz commented Jun 17, 2016

@mbj congratulations!

@mvz
Contributor
mvz commented Jul 26, 2016

The --since option doesn't seem to work with this integration.

@mbj
Owner
mbj commented Jul 26, 2016

The --since option doesn't seem to work with this integration.

--since should only affect the subject selection, and not intersect at all with the integration, as the integration simply gets feed less mutations as less mutations are generated from a more restricted subject set.

On what way it does not work?

@mbj
Owner
mbj commented Jul 26, 2016

@mbj congratulations!

Thx, we are still recovering efficient day by day organisation, so my OSS time is still negative.

@mvz
Contributor
mvz commented Jul 26, 2016

On what way it does not work?

The selection seems to always be empty. I'll see if I can find out more details of what goes wrong exactly. Good to know that it shouldn't depend on the integration.

@mbj
Owner
mbj commented Jul 26, 2016

The selection seems to always be empty. I'll see if I can find out more details of what goes wrong exactly. Good to know that it shouldn't depend on the integration.

Can you test if git diff $your_argument_given_to_since shows a diff that includes ruby code mutant would normally select on a full run without --since?

@mvz
Contributor
mvz commented Jul 27, 2016

Can you test if git diff $your_argument_given_to_since shows a diff that includes ruby code mutant would normally select on a full run without --since?

Oh, that is a good check. The main problem seems to be that '*' doesn't work as a match expression. It works fine using, e.g., 'GirFFI*'. Conclusion: --since works as it should.

@mbj
Owner
mbj commented Jul 27, 2016

Oh, that is a good check. The main problem seems to be that '' doesn't work as a match expression. It works fine using, e.g., 'GirFFI'. Conclusion: --since works as it should.

* is dangerous as it would also select stuff that is not in your SUT, in theory it should work also. If it does not select any available subject, please open an issue. I do not recommend using * with --since as it causes a lot of work internally on matching.

Ideally go for a shared toplevel namespace for your entire app and select this via YourApp*.

@mvz
Contributor
mvz commented Sep 18, 2016

@chastell are you still planning to add mutant to lovely_rufus? If so, perhaps I can lend a hand.

@chastell
Contributor

@mvz Argh, thanks for the patience – yes, but I’m not sure when, as I just got back from vacation and will be super busy in the next three weeks, then hope to finally have a bit of time for open source. :( Any help more than appreciated, of course!

@ch1c0t
ch1c0t commented Oct 26, 2016

This pull request introduces a special syntax to choose expressions:

cover 'TestApp::Literal*'

I wonder, have you considered running all the tests every time? It would encourage users to create smaller, better decomposed gems(to avoid waiting for a very long time, we will have to think better about avoiding high coupling).

@mbj
Owner
mbj commented Oct 26, 2016

I wonder, have you considered running all the tests every time? It would encourage users to create smaller, better decomposed gems(to avoid waiting for a very long time, we will have to think better about avoiding high coupling).

Does not scale for real world projects. If you want this behavior just use a top level cover '*'.

@mbj
Owner
mbj commented Oct 26, 2016

Also there is the problem of implicit coverage, one I've to blog about ;) Feel free to meet me in Slack (see README for invite link) where I can explain it.

@kbrock
Contributor
kbrock commented Dec 23, 2016

Is it possible to get this out there but just a beta / 0.0.1 version state?
So it will be out there albeit a little buggy?

There have been a number of changes a year ago, but after that, nothing really new is being introduced. So it feels like this is striving for perfection.

(granted, some would say that using mutant testing is striving for perfection...)

@mbj
Owner
mbj commented Dec 23, 2016

Is it possible to get this out there but just a beta / 0.0.1 version state?
So it will be out there albeit a little buggy?

There have been a number of changes a year ago, but after that, nothing really new is being introduced. So it feels like this is striving for perfection.

(granted, some would say that using mutant testing is striving for perfection...)

I've got negative time for OSS these days, very busy. I'd be fine if someone fixes it and releases as ${handle}-mutant-minitest, I'd reference it from the README.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment