Collecting tests from Python modules not generated by Python files

[Kodiologist]

I'm a core developer of the Hy programming language, which is a Lisp syntax for Python, and I'm trying to migrate its test suite from Nose to pytest. The repository has a lot of tests written in Python but also tests written in Hy. Hy itself is implemented in Python and provides a meta-importer so that a Python program can import a Hy module just like a Python module. So in theory, pytest should be able to collect tests from Hy files just as well as Python files, so long as Hy's meta-importer is active. Once a Hy file is loaded, there's basically no difference between Hy and Python; a Hy module is in fact a Python module.

I found a hacky way to run the Hy tests, using this conftest.py, which calls _pytest.python.pytest_pycollect_makemodule in a pytest_collect_file hook. The catch is that this hits assert name.endswith(".py") in _pytest.python, and Hy files end with .hy, not .py. If the assertion is commented out, everything works.

Is there a better way to do this? If not, perhaps you should remove the assertion.

I'm using pytest 3.0.7. Hy is intended to support, and is tested on, Python 2.7 and Python 3.3 and up.

[Kodiologist]

I'm now just spoofing the module name to look like it ends with .py, for lack of better ideas.

[RonnyPfannschmidt]

py.test itself checks the filenames of such modules, in order to override it you will need to create a own collector that handles the import

since you left out the trackback i dont know off-hand whats needed

[Kodiologist]

Sorry, here's an example of the backtraces I get when running pytest on my repository without the module-name spoof:

_______________ ERROR collecting tests/native_tests/extra/reserved.hy ________________
../lib/python3.6/site-packages/_pytest/runner.py:163: in __init__
    self.result = func()
../lib/python3.6/site-packages/_pytest/main.py:476: in _memocollect
    return self._memoizedcall('_collected', lambda: list(self.collect()))
../lib/python3.6/site-packages/_pytest/main.py:347: in _memoizedcall
    res = function()
../lib/python3.6/site-packages/_pytest/main.py:476: in <lambda>
    return self._memoizedcall('_collected', lambda: list(self.collect()))
../lib/python3.6/site-packages/_pytest/python.py:412: in collect
    return super(Module, self).collect()
../lib/python3.6/site-packages/_pytest/python.py:332: in collect
    l.sort(key=lambda item: item.reportinfo()[:2])
../lib/python3.6/site-packages/_pytest/python.py:332: in <lambda>
    l.sort(key=lambda item: item.reportinfo()[:2])
../lib/python3.6/site-packages/_pytest/python.py:264: in reportinfo
    modpath = self.getmodpath()
../lib/python3.6/site-packages/_pytest/python.py:238: in getmodpath
    assert name.endswith(".py")
E   AssertionError

[RonnyPfannschmidt]

this one is a bit tricky. the "python module" collector of pytest expects a python module for the filename

for now its a pretty good idea to just copy the method and replace its constants replacing .py with .py

[Kodiologist]

Okay, thanks.

[nicoddemus]

Taking a quick look at the code, it seems to me removing the assertion as @Kodiologist suggests is OK:

        for node in chain:
            if isinstance(node, Instance):
                continue
            name = node.name
            if isinstance(node, Module):
                assert name.endswith(".py")
                name = name[:-3]
                if stopatmodule:
                    if includemodule:
                        parts.append(name)
                    break
            parts.append(name)

I think the assertion is there just because of the name = name[:-3] below it, which could be replaced by a safer os.path.splitext(name)[0]. If all tests pass I don't think this will bring any problems.

Would you care to give it a shot @Kodiologist?

[Kodiologist]

@nicoddemus That seems to work fine for Hy's test suite.

[nicoddemus]

Would you like to open a PR? Then we can test it against pytest test suite. 😉