Improving output: verbose mode

[sobolevn]

How currently the output looks like?

900 out of 1067  (wemake_python_styleguide/violations/consistency.py     should_use_text =                                                                                          FAILED: mutmut wemake_python_styleguide/violations/consistency.py --apply --mutation "    should_use_text = False⤑1"

901 out of 1067  (wemake_python_styleguide/violations/consistency.py     error_template =                                                                                           FAILED: mutmut wemake_python_styleguide/violations/consistency.py --apply --mutation "    error_template = 'Found parens right after a keyword'⤑0"

902 out of 1067  (wemake_python_styleguide/violations/consistency.py     error_template = 
903 out of 1067  (wemake_python_styleguide/violations/consistency.py     code = 313⤑0)

It shows two cases: failing and successful state.
Let's talk about both of them. I will also try to cover some general thoughts that I have.

First of all, I would like to analyze what happened. Since this process it rather long, I am not able to watch it in real time. The final output is also quite verbose, so it is hard to read it though. What do I suggest? It would be nice to have some short report that will cover:

1. What mutants managed to survive: what was changed, what was the change, in what file
2. What tests killed the most mutants (so these tests can be documented as important ones, etc), what tests did not caught any (so these tests might be removed in the future or refactored)

Secondly, it is possible to have better "inline experience". What do I mean by that?

1. Currently when reading through the test cases I am missing some required information: what source line number was changed, to what, what mutation rule caused this
2. In case that the mutant was killed I guess we can show how many tests did fail. It would be very helpful and will give you a better understanding of what is going on

I would like to say, that this tool is absolutely awesome! Thank you for building it!

[boxed]

That output looks totally broken and not at all like what it looks like on my machine. How do you run those tests?

It's supposed to show an interactive progress and one line of output per surviving mutant.

[sobolevn]

The same as in #26

I am using:

• macos
• default terminal app
• zsh

[boxed]

Strange. It's supposed to output "x out of y" that it erases and overwrites until it finds a surviving mutant and then write "FAILED:" etc on one line and then start with "x out of y" again. So the final output should not contain any "x out of y" and all lines should start with "FAILED: "

[boxed]

@asottile told me what the problem was: the output breaks when the terminal window is too narrow and you get a line break.

[sobolevn]

yes, I have narrow terminal windows. @boxed thanks for digging it!

[boxed]

I'm going to totally rework the output to remove this bug and make it nicer in some other ways.

[boxed]

I've now pushed a totally reworked UX, plus a total rewrite of the cache mechanism. I would love for you to give the new code a try! I'll wait a while to release a new version to hopefully get some feedback first.

[sobolevn]

@boxed I will have some time on weekend. I will also test how pytest-testmon + mutmut will improve timings. Thanks for your work!

[boxed]

I am very curious about the results!

[boxed]

I've released 1.0.0 of mutmut with the new UX. Please give it a try, I think you'll be pleasantly surprised at how much nicer it is to use.

[sobolevn]

Yes, it looks awesome! But, it does not work for me 🙂

I am pretty sure that this is a configuration issue. Can you please guide me?

0. Using this template: https://github.com/wemake-services/wemake-django-template
1. pytest call results in ============================= 57 passed in 13.92 seconds =============================
2. Now running mutmut run with the following config:

[mutmut]
paths_to_mutate = server/main_app
backup = False
runner = pytest
tests_dir = tests/

Here's the output:

----------------------------- Captured stderr call -----------------------------
2018-11-18 15:51:56 [ERROR] Internal Server Error: /admin/doc/
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 882, in _resolve_lookup
    current = current[bit]
TypeError: 'TemplatesPanel' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/urls/base.py", line 77, in reverse
    extra, resolver = resolver.namespace_dict[ns]
KeyError: 'djdt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 142, in __call__
    response = self.process_response(request, response)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/debug_toolbar/middleware.py", line 131, in process_response
    bits[-2] += toolbar.render_toolbar()
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/debug_toolbar/toolbar.py", line 65, in render_toolbar
    return render_to_string('debug_toolbar/base.html', context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/loader.py", line 68, in render_to_string
    return template.render(context, request)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/backends/django.py", line 66, in render
    return self.template.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 207, in render
    return self._render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/test/utils.py", line 107, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 216, in render
    nodelist.append(node.render_annotated(context))
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 322, in render
    return nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 322, in render
    return nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 1040, in render
    output = self.filter_expression.resolve(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 708, in resolve
    obj = self.var.resolve(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 849, in resolve
    value = self._resolve_lookup(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 890, in _resolve_lookup
    current = getattr(current, bit)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/debug_toolbar/panels/__init__.py", line 96, in content
    return render_to_string(self.template, self.get_stats())
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/loader.py", line 68, in render_to_string
    return template.render(context, request)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/backends/django.py", line 66, in render
    return self.template.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 207, in render
    return self._render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/test/utils.py", line 107, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 322, in render
    return nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 216, in render
    nodelist.append(node.render_annotated(context))
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 458, in render
    url = reverse(view_name, args=args, kwargs=kwargs, current_app=current_app)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/urls/base.py", line 87, in reverse
    raise NoReverseMatch("%s is not a registered namespace" % key)
django.urls.exceptions.NoReverseMatch: 'djdt' is not a registered namespace
------------------------------ Captured log call -------------------------------
exception.py               135 ERROR    Internal Server Error: /admin/doc/
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 882, in _resolve_lookup
    current = current[bit]
TypeError: 'TemplatesPanel' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/urls/base.py", line 77, in reverse
    extra, resolver = resolver.namespace_dict[ns]
KeyError: 'djdt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/utils/deprecation.py", line 142, in __call__
    response = self.process_response(request, response)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/debug_toolbar/middleware.py", line 131, in process_response
    bits[-2] += toolbar.render_toolbar()
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/debug_toolbar/toolbar.py", line 65, in render_toolbar
    return render_to_string('debug_toolbar/base.html', context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/loader.py", line 68, in render_to_string
    return template.render(context, request)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/backends/django.py", line 66, in render
    return self.template.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 207, in render
    return self._render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/test/utils.py", line 107, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 216, in render
    nodelist.append(node.render_annotated(context))
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 322, in render
    return nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 322, in render
    return nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 1040, in render
    output = self.filter_expression.resolve(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 708, in resolve
    obj = self.var.resolve(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 849, in resolve
    value = self._resolve_lookup(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 890, in _resolve_lookup
    current = getattr(current, bit)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/debug_toolbar/panels/__init__.py", line 96, in content
    return render_to_string(self.template, self.get_stats())
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/loader.py", line 68, in render_to_string
    return template.render(context, request)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/backends/django.py", line 66, in render
    return self.template.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 207, in render
    return self._render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/test/utils.py", line 107, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 322, in render
    return nodelist.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 216, in render
    nodelist.append(node.render_annotated(context))
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/template/defaulttags.py", line 458, in render
    url = reverse(view_name, args=args, kwargs=kwargs, current_app=current_app)
  File "/Users/sobolev/Desktop/test_project/.venv/lib/python3.6/site-packages/django/urls/base.py", line 87, in reverse
    raise NoReverseMatch("%s is not a registered namespace" % key)
django.urls.exceptions.NoReverseMatch: 'djdt' is not a registered namespace
====================== 2 failed, 5 passed in 2.62 seconds ======================

Update

The next time I did clean-build from this template, I have found that it works.
The only difference is that I made git init and git commit commands before starting mutmut.

[sobolevn]

@boxed considering the output. It looks very pretty! Great work!

» mutmut run

- Mutation testing starting -

These are the steps:
1. A full test suite run will be made to make sure we
   can run the tests successfully and we know how long
   it takes (to detect infinite loops for example)
2. Mutants will be generated and checked

Mutants are written to the cache in the .mutmut-cache
directory. Print found mutants with `mutmut --print-result`.

Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.

1. Running tests without mutations
⠹ Running... Done

2. Checking mutants
⠙ 6/6  🎉 2  ⏰ 0  🤔 0  🙁 4%

But, I still can not see the most important part: what mutations passed. And what tests did the dirty job killing the mutants. Maybe I am missing something 🤔

[boxed]

Well crap, the output is wrong. What you want to do is:

mutmut results

To show a diff of a mutant:

mutmut show 7

And to apply

mutmut apply 7

The mutation ids are just primary keys into the SQLite cache database now. This is a lot nicer to use imo.

[sobolevn]

@boxed so, just to be clear, there's a bug in my output?
Do you need any versions that I am using?

• hyper@2.0.0
• zsh 5.6.2 (x86_64-apple-darwin15.6.0)
• python3.6.6
• macos 10.14.1
• pytest@3.9

[boxed]

Well, there was a bug in mutmut 1.0.0 where it said "mutmut --print-result", but it should have been "mutmut results". I've released 1.0.1 to fix this and some other minor nitpicks that people found.

So the workflow is:

1. mutmut run
2. mutmut results
3. mutmut show <id> or mutmut apply <id> depending on if you just want to look at the diff or apply it directly on disk

[sobolevn]

I guess there's something wrong with mutation@1 and mutation@2.

For some reason they are marked as No mutation performed.
But if so, why then they are listed as failed? Since my test suite worked for the first time.
Sorry, for asking too many questions. I am just new to the mutation testing. 🙂

(.venv) ~/Desktop/test_project
» mutmut run

- Mutation testing starting -

These are the steps:
1. A full test suite run will be made to make sure we
   can run the tests successfully and we know how long
   it takes (to detect infinite loops for example)
2. Mutants will be generated and checked

Mutants are written to the cache in the .mutmut-cache
directory. Print found mutants with `mutmut results`.

Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.

1. Running tests without mutations
⠙ Running... Done

2. Checking mutants
⠦ 6/6  🎉 2  ⏰ 0  🤔 0  🙁 4

(.venv) ~/Desktop/test_project
» mutmut results
Timed out ⏰

Suspicious 🤔

Survived 🙁
mutmut apply 1
mutmut apply 2
mutmut apply 3
mutmut apply 4

(.venv) ~/Desktop/test_project
» mutmut show 1
No mutation performed

(.venv) ~/Desktop/test_project
» mutmut show 2
No mutation performed

(.venv) ~/Desktop/test_project
» mutmut show 3
--- server/main_app/urls.py
+++ server/main_app/urls.py
@@ -7,6 +7,6 @@
 # Place your URLs here:

 urlpatterns = [
-    url(r'^hello/$', index, name='hello'),
+    url(r'XX^hello/$XX', index, name='hello'),
 ]


(.venv) ~/Desktop/test_project
» mutmut show 4
--- server/main_app/urls.py
+++ server/main_app/urls.py
@@ -7,6 +7,6 @@
 # Place your URLs here:

 urlpatterns = [
-    url(r'^hello/$', index, name='hello'),
+    url(r'^hello/$', index, name='XXhelloXX'),
 ]

[boxed]

Huh, that IS strange. Can you post that in a new ticket with the project? Since it looks like a small test project I guess you can send it?