Mutants not tested with pythonpath or sys.path.append

[CyberiaResurrection]

Mutmut version: 22f3924
Python version: 3.9.21
OS: Fedora Linux 41

Repo: https://github.com/makhidkarun/traveller_pyroute

pyproject.toml bits:

paths_to_mutate = ["PyRoute/"]
tests_dir = ["Tests/"]
also_copy = ["Tests/", "PyRoute/Pathfinding/*.so"]
runner = "pytest -x"

After getting past #415 , I'm now actually getting mutants to run, but:

⠼ Generating mutants
    done in 44193ms
⠇ Listing all tests 
⠼ Running clean tests
    done
⠧ Running forced fail test
    done
Running mutation testing
⠴ 30097/30097  🎉 0 🫥 30097  ⏰ 0  🤔 0  🙁 0  🔇 0
0.00 mutations/second

no mutants are actually being tested.
I'm not sure how this case differs from #351 , but the following are things I can find:

1. mutants/ directory not committed to git.
2. Tests/ directory is not being mutated.
3. In the branch I'm working in, I've removed PyRoute. from imports, across both code and tests.
4. Forced-fail testing runs clean.

[Otto-AA]

What did you change with regards to the previous issue? In the previous issue the mutants were tested and it was looking okay (except for the additional segmentation fault logging, which should be shown as emoji).

[CyberiaResurrection]

I changed the code that led to the segfaults to raise exceptions when receiving bad data types, and not surprisingly, they went away.

How were the mutants being tested, in light of what you said here? #415 (comment) (namely, "It doesn't count untested mutations" )

[Otto-AA]

The status line showed them: ⠼ 30093/30093 🎉 93 🫥 29969 ⏰ 15 🤔 0 🙁 8 🔇 0
-> 93 mutants killed, 15 timeouts and 8 survived

These were the 120 mutants I talked about that were included in the time calculation.

But yes, it's weird that you have so many untested mutants as you seem to execute your whole test suite. You could try to set debug = true and check what tests it executes, or go to the mutants folder and run pytest there.

[CyberiaResurrection]

Ah, I see. After fixing the segfault-leading code, I ran mutation again and got (see above)

⠴ 30097/30097  🎉 0 🫥 30097  ⏰ 0  🤔 0  🙁 0  🔇 0

We were looking at different status lines. Now to try debug = true, and also running pytest from mutants folder

[CyberiaResurrection]

Running with debug = true resulted in

FAILED: Unable to force test failures

Got the same result without debug = true.

Now I'm wondering if the initial passes I was getting were false passes.

Running pytest in the mutants/ folder resulted in a lot of tests blowing up with

KeyError: 'MUTANT_UNDER_TEST'

[Otto-AA]

FAILED: Unable to force test failures

This usually indicates a problem with the project/test setup and imports. Mutmut copies all relevant source and test files to ./mutants and mutates the source files there. The "force test failures" check sets os.environ['MUTANT_UNDER_TEST'] = 'fail' which will cause all mutants to raise a MutmutProgrammaticFailException. This tests if the mutants can make any test fail (and thus the setup looks fine).

In your case, something in the setup did not work and all tests passed, despite mutmut trying to make all mutants fail.

The sys.path.append('../PyRoute') looks troublesome (this probably results in pytest importing the original code instead of the mutated code), but without a MRE I cannot really debug this.

To test if your tests use the mutated code, you can add this test next to your other tests (with the sys.path.append('../PyRoute') so we have the same setup as for the other tests):

import PyRoute.AreaItems.Galaxy
from pathlib import Path

def test_path():
    assert 'mutants' in str(Path(PyRoute.AreaItems.Galaxy.__file__).resolve())

If the setup is correct, then this test should pass (because mutmut should use the mutated code from the ./mutants) folder.

[CyberiaResurrection]

Removing sys.path.append('../PyRoute') has got the tests running fully through again - both clean and forced-fail. Still has no mutants actually running.

[Otto-AA]

Without a MRE I won't be able to help you further. And also I won't be available much the next two weeks, so probably I'll only get to look at the MRE in August.

For the MRE, try to remove most of the test and source code and the third party libraries, so that it is easy to understand and setup locally. If possible, reduce it to one method in the source code and one unit test for it , where pytest successfully tests your source code (and the test covers the method), but mutmut reports it as untested.
Then I can actually try it out locally

[CyberiaResurrection]

@Otto-AA , all right, I'll see what I can do along those lines. When I get a MRE, just link it in a comment on this issue?

With the following test, in testGalaxyPath.py, added to Tests/:

import unittest
from pathlib import Path

import PyRoute.AreaItems.Galaxy


class testGalaxyPath(unittest.TestCase):
    def test_path(self):
        assert 'mutants' in str(Path(PyRoute.AreaItems.Galaxy.__file__).resolve())

results in:

⠇ Generating mutants
    done in 35117ms
⠼ Listing all tests 
Found 1 new tests, rerunning stats collection
⠋ Running stats    
    done
⠧ Running clean tests
    done
⠹ Running forced fail test
    done
Running mutation testing
⠋ 30097/30097  🎉 0 🫥 30097  ⏰ 0  🤔 0  🙁 0  🔇 0
0.00 mutations/second

test_path fails when run in tests normally (as it should). but as you can see, despite it passing under mutmut clean, no mutants are tested.

Thanks heaps for your help so far.

[Otto-AA]

Yes, just link the repo / branch.

test_path fails when run in tests normally (as it should). but as you can see, despite it passing under mutmut clean, no mutants are tested.

It's good that test_path fails with the normal pytest, but succeeds in mutmut. That means mutmut uses the mutated code. I'm not sure why mutmut reports all mutants as untested. I would have guessed, if the forced fail test passes that it finds at least one mutant.

You can also try to look at mutants/mutmut-stats.json. If mutmut really found no tests, then tests_by_mangled_function_name should only contain empty lists for the mutants. Also you can verify that duration_by_test shows some tests.

[CyberiaResurrection]

tests_by_mangled_function_name in mutants/mutmut-stats.json is a single empty list. Let me see what I can cook up for an MRE.

[CyberiaResurrection]

Here's a first attempt at an MRE: https://github.com/CyberiaResurrection/traveller_pyroute_mini
It's only got SystemStar.py and testSystemStar.py under source and test dirs, respectively.

Coverage results:

Name                               Stmts   Miss  Cover
------------------------------------------------------
PyRoute/SystemData/SystemStar.py      29     13    55%
PyRoute/__init__.py                    0      0   100%
------------------------------------------------------
TOTAL                                 29     13    55%

Mutation results:

⠼ Generating mutants
    done in 112ms
⠹ Listing all tests 
⠏ Running clean tests
    done
⠸ Running forced fail test
    done
Running mutation testing
⠧ 33/33  🎉 0 🫥 33  ⏰ 0  🤔 0  🙁 0  🔇 0
0.00 mutations/second

[Otto-AA]

The cause is that you use pythonpath = PyRoute/ in your pytest.ini file.

Mutmut records following tests in mutants/mutmut-stats.json (note that you previously said this is empty for the original code, so the MRE maybe has a noticable difference to your original repo):

    "tests_by_mangled_function_name": {
        "SystemData.SystemStar.x\u01c1SystemStar\u01c1__init__": [
            "Tests/SystemData/testSystemStar.py::testSystemStar::test_is_bigger"
        ],
        "SystemData.SystemStar.x\u01c1SystemStar\u01c1is_bigger": [
            "Tests/SystemData/testSystemStar.py::testSystemStar::test_is_bigger"
        ]
    },

These names (eg SystemData.SystemStar.xǁSystemStarǁis_bigger) are based on the runtime module information we get (using the method reference to get the module + name: xǁSystemStarǁis_bigger.__module__ + '.' + xǁSystemStarǁis_bigger.__name__).

However, when generating the mutant names we do not have runtime information and use the filenames instead: to get the module name we use module_name = strip_prefix(str(filename)[:-len(filename.suffix)].replace(os.sep, '.'), prefix='src.'). In your case, for the filename PyRoute/SystemData/SystemStar.py we get PyRoute.SystemData.SystemStar.

Then later on, when we try to find the tests for the mutants we get nothing, because the module names (SystemData.SystemStar and PyRoute.SystemData.SystemStar) do not match. Thus, we do not find the tests for the mutants.

Maybe this could be changed in mutmut, to always use the filename, I am not sure about this. On your side, if you manage to get everything running without messing with import paths (pythonpath, sys.path, etc), then this should also be fixed.

I think our docs could mention that messing with import paths can break mutmut.

[CyberiaResurrection]

Thanks for looking at it, @Otto-AA I'll try what you've suggested and report back.