Mutmut doesn't test mutants

[vinceslao]

I have a problem with mutmut: the tool creates mutants but it doesn't tests them.

Mutants using command mutmut browse:

Results:

what could be the problem??

@boxed

[boxed]

Did you run mutmut run first? If so, what was the output?

[vinceslao]

This is the output:

mutmut run
⠹ Generating mutants
    done in 4ms
⠧ Listing all tests 
⠸ Running clean tests
    done
⠧ Running forced fail test
    done
Running mutation testing
⠇ 380/380  🎉 0 🫥 380  ⏰ 0  🤔 0  🙁 0  🔇 0
0.00 mutations/second

[boxed]

Is this project available somewhere so I can try this?

[vinceslao]

https://github.com/vinceslao/CitationAnalyzer

[boxed]

Just a note: you should NOT commit the mutants directory to git!

[vinceslao]

excuse me but what is the reason?

[boxed]

They are temporary files for working with mutation testing. I bet it's the same with the cosmic ray files you have committed. At least the .sqlite files seem iffy to commit.

You have also committed .DS_Store incorrectly I se.

[vinceslao]

ok, I fixed that. Any updates on the initial issue?

[MrMicc]

I'm facing something similar issue. Mutants are being created, but the tests are not being executed

Python version: 3.12.5

setup.cfg

[mutmut]
paths_to_mutate = src/
runner = pytest tests/
tests_dir = tests/

Also tested without runner at setup.cfg file

result mutmut run:

⠦ Generating mutants
    done in 5ms
⠋ Listing all tests 
⠴ Running clean tests
    done
⠦ Running forced fail test
    done
Running mutation testing
⠇ 51/51  🎉 0 🫥 51  ⏰ 0  🤔 0  🙁 0  🔇 0
0.00 mutations/second

result of mutmut results:

 domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_1: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_2: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_3: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_4: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_5: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_6: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_1: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_2: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_3: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_4: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_5: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_6: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_7: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_8: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_9: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_10: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_11: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_12: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_13: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_1: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_2: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_3: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_4: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_5: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_6: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_7: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_8: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_9: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_10: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_11: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_12: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_13: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_14: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_15: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_16: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_17: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_18: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_19: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_20: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_21: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_22: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_23: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_24: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_25: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_26: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_27: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_28: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_29: no tests
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_30: no tests
    domain.model.exception.customException.xǁNomeInvalidoErrorǁ__init____mutmut_1: no tests
    domain.model.exception.customException.xǁEmailInvalidoErrorǁ__init____mutmut_1: no tests

coverage report:

Name                                            Stmts   Miss  Cover
-------------------------------------------------------------------
src/__init__.py                                     0      0   100%
src/domain/__init__.py                              0      0   100%
src/domain/model/__init__.py                        0      0   100%
src/domain/model/exception/__init__.py              0      0   100%
src/domain/model/exception/customException.py       6      0   100%
src/domain/model/usuario/__init__.py                0      0   100%
src/domain/model/usuario/usuario.py                68      0   100%
tests/__init__.py                                   0      0   100%
tests/domain/__init__.py                            0      0   100%
tests/domain/model/__init__.py                      0      0   100%
tests/domain/model/usuario/__init__.py              0      0   100%
tests/domain/model/usuario/usuario_test.py        101      0   100%
-------------------------------------------------------------------
TOTAL                                             175      0   100%

When I run mutmut with debug=True on setup.cfg I was able to see the problem:

   result = _mutmut_trampoline(object.__getattribute__(self, "xǁUsuarioǁ__init____mutmut_orig"), object.__getattribute__(self, "xǁUsuarioǁ__init____mutmut_mutants"), *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

orig = <bound method Usuario.xǁUsuarioǁ__init____mutmut_orig of <src.domain.model.usuario.usuario.Usuario object at 0x110d86c00>>
mutants = {'xǁUsuarioǁ__init____mutmut_1': <function Usuario.xǁUsuarioǁ__init____mutmut_1 at 0x110dfa2a0>, 'xǁUsuarioǁ__init____...3 at 0x110dfa700>, 'xǁUsuarioǁ__init____mutmut_4': <function Usuario.xǁUsuarioǁ__init____mutmut_4 at 0x110dfa840>, ...}
args = ('Micci', 'email@email.com'), kwargs = {}, os = <module 'os' (frozen)>, mutant_under_test = 'fail', MutmutProgrammaticFailException = <class 'mutmut.__main__.MutmutProgrammaticFailException'>

    def _mutmut_trampoline(orig, mutants, *args, **kwargs):
        import os
        mutant_under_test = os.environ['MUTANT_UNDER_TEST']
        if mutant_under_test == 'fail':
            from mutmut.__main__ import MutmutProgrammaticFailException
>           raise MutmutProgrammaticFailException('Failed programmatically')
E           mutmut.__main__.MutmutProgrammaticFailException: Failed programmatically

But I'm not understanding why mutmut is crashing

Anyway solved doing a downgrade on mutmut 3 to mutmut 2.5.1
@vinceslao maybe this could work for you too
mutmut run

- Mutation testing starting -

These are the steps:
1. A full test suite run will be made to make sure we
   can run the tests successfully and we know how long
   it takes (to detect infinite loops for example)
2. Mutants will be generated and checked

Results are stored in .mutmut-cache.
Print found mutants with `mutmut results`.

Legend for output:
🎉 Killed mutants.   The goal is for everything to end up in this bucket.
⏰ Timeout.          Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious.       Tests took a long time, but not long enough to be fatal.
🙁 Survived.         This means your tests need to be expanded.
🔇 Skipped.          Skipped.

1. Running tests without mutations
⠋ Running...Done

2. Checking mutants
⠧ 75/75  🎉 43  ⏰ 0  🤔 5  🙁 27  🔇 0

[boxed]

@MrMicc MutmutProgrammaticFailException is on purpose. If it doesn't manage to do that, it will fail for real, so that's not it. Is that project available so I can test?

[MrMicc]

Hi @boxed ! Happy New Year! 🥳

Link to repo: https://github.com/MrMicc/Study/tree/main/python-playground

Thanks in advance

[boxed]

@MrMicc The code you have there is not correct. https://github.com/MrMicc/Study/blob/main/python-playground/tests/domain/model/usuario/usuario_test.py has imports like from src..... They should not begin with src.. src should not be a module path.

[Julien-Delavisse]

I'm continuing the discussion here (started here for information #349 (comment) ) because my problem now seems more like the one presented here.

I've updated my repository https://github.com/Julien-Delavisse/mutmut-v3-tests with a test_single.sh script to test from a mutmut github commit (using docker for better reproducibility).

I'm getting a slightly different result with this script than with my system :

⠋ Generating mutantssrc/helloworld.py
     also copying tests
     also copying test
     also copying tests.py
     also copying setup.cfg
     also copying pyproject.toml

    done in 13ms

⠙ Running stats     python -m pytest  -vv -x -q --import-mode=append --rootdir=.
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/local/bin/python3.12
cachedir: .pytest_cache
rootdir: /app/mutants
configfile: pyproject.toml
testpaths: tests
collecting ... collected 1 item

tests/test_helloworld.py::test_hello_world PASSED                        [100%]

============================== 1 passed in 0.00s ===============================
    exit code 0

    done

⠹ Running clean testspython -m pytest  -vv -x -q --import-mode=append --rootdir=.
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/local/bin/python3.12
cachedir: .pytest_cache
rootdir: /app/mutants
configfile: pyproject.toml
testpaths: tests
collecting ... collected 1 item

tests/test_helloworld.py::test_hello_world PASSED                        [100%]

============================== 1 passed in 0.00s ===============================
    exit code 0

    done

⠸ Running forced fail testpython -m pytest  -vv -x -q --import-mode=append --rootdir=.
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/local/bin/python3.12
cachedir: .pytest_cache
rootdir: /app/mutants
configfile: pyproject.toml
testpaths: tests
collecting ... collected 1 item

tests/test_helloworld.py::test_hello_world FAILED                        [100%]

=================================== FAILURES ===================================
_______________________________ test_hello_world _______________________________

    def test_hello_world():
>       assert helloworld.hello_world() == 1

tests/test_helloworld.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/helloworld.py:56: in hello_world
    result = _mutmut_trampoline(x_hello_world__mutmut_orig, x_hello_world__mutmut_mutants, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

orig = <function x_hello_world__mutmut_orig at 0x7314fdcfcf40>
mutants = {'x_hello_world__mutmut_1': <function x_hello_world__mutmut_1 at 0x7314fdcfcfe0>}
args = (), kwargs = {}, os = <module 'os' (frozen)>, mutant_under_test = 'fail'
MutmutProgrammaticFailException = <class 'mutmut.__main__.MutmutProgrammaticFailException'>

    def _mutmut_trampoline(orig, mutants, *args, **kwargs):
        import os
        mutant_under_test = os.environ['MUTANT_UNDER_TEST']
        if mutant_under_test == 'fail':
            from mutmut.__main__ import MutmutProgrammaticFailException
>           raise MutmutProgrammaticFailException('Failed programmatically')
E           mutmut.__main__.MutmutProgrammaticFailException: Failed programmatically

src/helloworld.py:9: MutmutProgrammaticFailException
=========================== short test summary info ============================
FAILED tests/test_helloworld.py::test_hello_world - mutmut.__main__.MutmutPro...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============================== 1 failed in 0.01s ===============================
    exit code 1

    done
Running mutation testing

⠼ 0/1  🎉 0 🫥 0  ⏰ 0  🤔 0  🙁 0  🔇 0python -m pytest  -vv -x -q --import-mode=append tests/test_helloworld.py::test_hello_world --rootdir=.
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/local/bin/python3.12
cachedir: .pytest_cache
rootdir: /app/mutants
configfile: pyproject.toml
collecting ... collected 1 item

tests/test_helloworld.py::test_hello_world FAILED                        [100%]    worker exit code 1

⠴ 1/1  🎉 1 🫥 0  ⏰ 0  🤔 0  🙁 0  🔇 0
44.64 mutations/second

[MrMicc]

@MrMicc The code you have there is not correct. https://github.com/MrMicc/Study/blob/main/python-playground/tests/domain/model/usuario/usuario_test.py has imports like from src..... They should not begin with src.. src should not be a module path.

Hey @boxed thanks for your support!
I didnt know about src, sorry about it! I'm a newbie at python 😅
(Ah.. if want I can add this for you at the README)

I configure pytest.ini

[pytest]
pythonpath = src

But now I'm receiving a problema similar from the issue #341
when I do mutmut run

======================================================================================= 50 passed in 0.02s =======================================================================================
    exit code 0
FAILED

with mutmut results returning:

 domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_1: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_2: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_3: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_4: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_5: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__init____mutmut_6: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_1: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_2: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_3: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_4: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_5: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_6: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_7: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_8: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_9: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_10: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_11: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_12: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_nome_valido__mutmut_13: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_1: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_2: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_3: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_4: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_5: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_6: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_7: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_8: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_9: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_10: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_11: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_12: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_13: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_14: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_15: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_16: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_17: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_18: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_19: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_20: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_21: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_22: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_23: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_24: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_25: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_26: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_27: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_28: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_29: not checked
    domain.model.usuario.usuario.xǁUsuarioǁ__checa_se_email_valido__mutmut_30: not checked
    domain.model.exception.customException.xǁNomeInvalidoErrorǁ__init____mutmut_1: not checked
    domain.model.exception.customException.xǁEmailInvalidoErrorǁ__init____mutmut_1: not checked

[boxed]

I'm getting a slightly different result with this script than with my system :

Yea. 100% mutants killed. 100% success :P @Julien-Delavisse