Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to correctly run pytest.main() programmatically multiple times? #3143

Closed
numirias opened this issue Jan 23, 2018 · 15 comments
Closed

How to correctly run pytest.main() programmatically multiple times? #3143

numirias opened this issue Jan 23, 2018 · 15 comments

Comments

@numirias
Copy link

@numirias numirias commented Jan 23, 2018

If I invoke pytest.main(['test_foo.py']) multiple time from the same running script, it will give the same result, even if test_foo.py changes between the runs. (This was also observed in #793.)

But then what is the correct way to rerun pytest programmatically? Could you point me to an example? Starting it in a subprocess feels quite clunky and also makes it more cumbersome to process the results.

(Also, could we possibly have a note about that behavior in the docs? At least to me, it was somewhat unexpected that I can't run pytest.main() in the same script twice.)

@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Jan 23, 2018

python modules cache in side the same process and due to this property of python its not advisable to run pytest.main twice from the same process

while there may be exceptions to that rule, in general its more safe to run a new process since python is not safe for code reloading

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Jan 23, 2018

I agree, we should just add a note to the docs regarding that.

@endolith
Copy link

@endolith endolith commented Jan 4, 2019

So the recommendation is to completely close python and start a new process every time you want to test your code? I don't remember this being a problem with pytest in the past, has it changed?

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Jan 4, 2019

@endolith this does not have to with pytest, but how Python itself works: it caches the modules it imports.

For example:

# foo.py
def f(): return 1
>>> import foo
>>> foo.f()
1
>>>

Now if we change foo.py to:

# foo.py
def f(): return 2

and continue in the same session:

>>> import foo
>>> foo.f()
1
>>> foo.f()
1

I don't remember this being a problem with pytest in the past, has it changed?

Sorry, I'm reasonably sure pytest never supported this, because again it doesn't have to do with pytest but how Python works... and I don't think pytest had any special code in place to workaround this.

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Jan 5, 2019

BTW there's a note at the bottom of the docs for exactly this issue: https://docs.pytest.org/en/latest/usage.html

@endolith
Copy link

@endolith endolith commented Jan 5, 2019

@nicoddemus Apparently the reason it works for me is a side effect of running inside of Spyder: https://stackoverflow.com/questions/54009371/pytestwarning-module-already-imported-so-cannot-be-rewritten-pytest-remotedata/54011939?noredirect=1#comment94930323_54011939 and the warnings about PytestWarning: Module already imported so cannot are the only thing that's changed?

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Jan 5, 2019

Oh that explains it then. 👍

@endolith
Copy link

@endolith endolith commented Feb 19, 2019

So what's the recommended way to do this? pytest.main() prints a bunch of warnings, subprocess.call(['pytest', ... doesn't print the output, etc.

@endolith
Copy link

@endolith endolith commented Feb 19, 2019

Something like this works, but is kind of ugly:

if __name__ == "__main__":
    from subprocess import Popen, PIPE
    with Popen(['pytest',
                '--tb=short',  # shorter traceback format
                '--hypothesis-show-statistics',
                str(__file__)], stdout=PIPE, bufsize=1,
               universal_newlines=True) as p:
        for line in p.stdout:
            print(line, end='')
@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Feb 19, 2019

Indeed running as a subprocess is the recommended way to do this.

@endolith
Copy link

@endolith endolith commented Feb 19, 2019

@nicoddemus Maybe a convenience function can be written to do this? Or at least add some hints to the note at the bottom of https://docs.pytest.org/en/latest/usage.html

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Feb 19, 2019

I'm OK with a note to the docs.

Btw, do you know the pytester fixture? It may be used to invoke python tests in a "block-box" style testing, if that's what you are doing.

@endolith
Copy link

@endolith endolith commented Apr 2, 2019

@nicoddemus No I don't know about pytester, would that help me? All I'm doing is writing a program and tests simultaneously in the same file, and running the tests each time I change the file to make sure I didn't break anything.

@blueyed
Copy link
Contributor

@blueyed blueyed commented Apr 2, 2019

@endolith
Check out https://github.com/joeyespo/pytest-watch/ then maybe, or just use inotify tools/wrappers (e.g. https://github.com/blueyed/dotfiles/blob/master/usr/bin/inotify-run).
Also check out https://github.com/tarpas/pytest-testmon/ maybe to only re-run changed/affected tests.

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Apr 2, 2019

For that use case @blueyed's suggestions are on point then. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants