Output for noop?

[mockdeep]

I'm getting a noop failure, but when I run the spec directly it passes. Is there a way I can get the actual failure message to debug it?

[mockdeep]

Looks like it may have to do with eager loading. When I run it like this it works:

RAILS_ENV=test mutant -r ./config/environment --rspec User#admin?

[dkubb]

Maybe it's more to do with how rails lazy loads classes.

In my mutant task I would eager load the classes, and then filter out a few I don't care about: https://gist.github.com/dkubb/6ae4bacedb5b7c11615a

In this specific project I'm moving towards namespacing all my ruby code, so I may just change this to only match code in my namespace and filter out almost everything besides stuff like ApplicationController.

[mockdeep]

Ah, you changed your script. I was having some problems with it before in terms of the way it loaded classes in test mode. ActiveSupport::Dependencies.autoloaded_constants was an empty array.

The funny thing is, mutant's greatest value for me tends to be with poorly structured code. Making sure something is really well tested before refactoring is a huge comfort. But it also means that I have to deal with a messier integration.

At any rate, it would be great if we could surface the exception from noop failures so that it's more clear what is going wrong. Seems like it ought to fail fast with a noop failure, too, since running against other mutations is kind of meaningless in that case.

[dkubb]

@mockdeep yes, I ran into some issues after upgrading to Rails 4. Whether the issues were always there, or they were caused by the upgrade isn't clear to me. Either way, I'm happier with this code than the previous code, but I'll be happier once I can get all my code namespaced and then use something very close to the devtools rake task.

I've found with mutant (and before it heckle) that it helped train me into what kinds of things I need to test in the code to cover more of the possible states it can be in. Also how to reduce the number of possible states without reducing functionality. I found it forced me to use smaller methods that have single responsibilities as compared to large methods that does a bunch of things before returning.

At any rate, it would be great if we could surface the exception from noop failures so that it's more clear what is going wrong.

I just want to be clear what you mean here. Could you give me an example?

[mockdeep]

I've actually been having these problems with Rails 3.2. Haven't worked with Rails 4, yet. Heckle helped me a lot in terms of figuring out what to test and feeling comfortable refactoring. The irony is that when you have pretty well structured code, mutation testing doesn't hold as much of an advantage.

I was getting the following output when I ran mutant and there wasn't anything to tell my why it happened. I think with noops, it should show the error from rspec. When I ran the spec directly it passed.

$ mutant -r ./config/environment --rspec User#admin?
Mutant configuration:
Matcher:        #<Mutant::Matcher::Method::Instance cache=#<Mutant::Cache> scope=User(id: integer, first_name: string, last_name: string, email: string, position: string, school_name: string, password_digest: string, created_at: datetime, updated_at: datetime, send_example: integer, edit_example: integer, status: string, type: string, deleted_at: datetime, phone: string, twitter: string) method=#<UnboundMethod: User#admin?>>
Subject Filter: Mutant::Predicate::CONTRADICTION
Strategy:       #<Mutant::Strategy::Rspec level=0>
You are using WebMock 1.13.0. VCR is known to work with WebMock >= 1.8.0, < 1.12. It may not work with this version.
User#admin?:/home/fletch/Dropbox/projects/synchroform/app/models/user.rb:52
F...............
(15/16)  93% - 52.35s
User#admin?:/home/fletch/Dropbox/projects/synchroform/app/models/user.rb:52
noop:User#admin?:/home/fletch/Dropbox/projects/synchroform/app/models/user.rb:52:84cd3
Parsed subject AST:
(def :admin?
  (args)
  (send
    (array
      (str "admin1@gmail.com")
      (str "admin2@gmail.com")) :include?
    (send nil :email)))
Unparsed source:
def admin?
  ["admin1@gmail.com", "admin2@gmail.com"].include?(email)
end
(15/16)  93% - 52.35s
Subjects:  1
Mutations: 16
Kills:     15
Runtime:   57.86s
Killtime:  52.35s
Overhead:  9.52%
Coverage:  93.75%
Alive:     1
Nodes handled by generic mutator (type:occurrences):

[dkubb]

@mockdeep note how the unparsed source includes "admin1@gmail.com" twice? The parsed ast looks good, it looks like the unparsing stage failed. If one of your test checks for the "admin2@gmail.com" address that could be the source of failure.

[mockdeep]

That was a typo. I changed the emails. Fixed now. Regardless, the tests aren't failing when run directly.

[dkubb]

Damn, I thought we were onto something there. I think you're right that the rspec runner should capture the error and display it in this situation.

[mbj]

@mockdeep, @dkubb I plan a generic "non mutation type dependant" mechanism to capture the output. My plan is to write a "log" file that contains all output / metadata in YAML format. This file will contain enough information to get the corresponding rspec output. So you can examine all details of a finished mutant run "later". The current reporter is good for interactive use, but NOT for a typical CI environment. I consider such an log file as an important build artifact.