Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is pytest-xdist supported? #42

Open
timdiels opened this Issue Mar 24, 2016 · 16 comments

Comments

Projects
None yet
5 participants
@timdiels
Copy link

timdiels commented Mar 24, 2016

When combining testmon and xdist, more tests are rerun than necessary. I suspect the xdist runners don't update .testmondata.

Is testmon compatible with xdist or is this due to misconfiguration?

@tarpas

This comment has been minimized.

Copy link
Owner

tarpas commented Mar 29, 2016

Hi, unfortunatelly testmon is not compatible with xdist.

On Thu, Mar 24, 2016 at 12:19 PM, Tim Diels notifications@github.com
wrote:

When combining testmon and xdist, more tests are rerun than necessary. I
suspect the xdist runners don't update .testmondata.

Is testmon compatible with xdist or is this due to misconfiguration?


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#42

@tarpas tarpas added the enhancement label Oct 23, 2016

@boxed

This comment has been minimized.

Copy link
Contributor

boxed commented Nov 29, 2016

Do you know what the problem is? If it's described in this ticket, maybe someone can take a look?

@tarpas

This comment has been minimized.

Copy link
Owner

tarpas commented Dec 1, 2016

No I don't know what the problem is and it's not described anywhere. What do you expect from combining xdist and testmon? Wha'ts your use case, please?

@boxed

This comment has been minimized.

Copy link
Contributor

boxed commented Dec 1, 2016

The use case is the same as for xdist: if you have a lot of tests they can benefit from being run in parallel.

So the scenario would be:

  1. you change some file and testmon detects 100 (slow integration) tests that needs to be run
  2. testmon starts the tests with xdist enabled and the tests run in parallel
@tarpas

This comment has been minimized.

Copy link
Owner

tarpas commented Dec 6, 2016

Ok, right. I'm currently thinking about lightning fast, CPU efficient, interactive re-execution of small test suites. (which wouldn't benefit because of the cost of starting-up multiple interpreters, and because of IPC)

When I said testmon is not compatible with xdist I meant I didn't test it and didn't invest the time to think about it. The problem is what @timdiels describes. After thinking about it here are my ideas:
If the slaves run at remote machine or in different directories, then .testmondata doesn't get the data from slaves. How to solve this?
pytest-testmon would have to send the coverage data to the master in similar fashion to what xdist does with reports:
https://github.com/pytest-dev/pytest-xdist/blob/master/xdist/remote.py#L92

Poor man's solution for the cases when the slaves actually run on the same machine in the same directory, would be to just make sure the different processes don't overwrite each other, which I'm pretty sure is happening. To achieve that the .testmondata format would have to change, so that each nodedata is stored in a separate row of the "alldata" sqlite table. That would be a good change for a number of other reasons anyway.

@tarpas tarpas referenced this issue Dec 8, 2016

Merged

Refactor #54

@notestaff

This comment has been minimized.

Copy link

notestaff commented Jun 20, 2017

"nodedata is stored in a separate row of the "alldata" sqlite table." -- concurrent writes would still be a problem, no? Wouldn't you want to store each test node's data in a separate file, to avoid conflicts?

@tarpas

This comment has been minimized.

Copy link
Owner

tarpas commented Jun 21, 2017

No, as far as I know it's fine and Sqlite takes care of everything better than I could manually https://stackoverflow.com/questions/4060772/sqlite3-concurrent-access

@notestaff

This comment has been minimized.

Copy link

notestaff commented Jun 21, 2017

I've seen "database locked" errors with -n 2, that go away with disabling xdist. Maybe, adding retries would fix that?

@notestaff

This comment has been minimized.

Copy link

notestaff commented Jun 21, 2017

@tarpas btw I saw you held a fundraiser for this project. I bet travis-ci would sponsor it, since it'd save them a done of computation if people used it in conjunction with travis' caching of specified directories between builds ( https://docs.travis-ci.com/user/caching/#Arbitrary-directories )

@tarpas

This comment has been minimized.

Copy link
Owner

tarpas commented Jun 21, 2017

Is the "database locked" error the same as here?: #68 (comment) , if yes, it's been fixed in version 0.9.6

@tarpas

This comment has been minimized.

Copy link
Owner

tarpas commented Jun 21, 2017

@notestaff About travis-ci: I know, right! :) I see some downsides however. Does anybody have a tip whom to contact at Travis-CI?

@blueyed

This comment has been minimized.

Copy link
Contributor

blueyed commented Jun 21, 2017

One downside appears to be that this won't work when collecting coverage for your tests, which is rather typical I assume?!
Also for CI it is more important to really run all tests (and not skipping some accidentally because of some potential issue with pytest-testmon or how the cache/data was synced etc).

@notestaff

This comment has been minimized.

Copy link

notestaff commented Jun 22, 2017

You can save coverage info, e.g. as in https://github.com/notestaff/pytest-testmon/tree/is-save-cov
You give up sub-file resolution.

There are degrees of CI; for our use, we plan to skip tests for routine branch checkins and pull requests but clear the cache and run all tests for tagged releases. The flipside of not skipping tests is that you get wary of adding even meaningful tests to the test suite, since the commit-then-test cycle becomes too slow as the test suite grows.

@blueyed

This comment has been minimized.

Copy link
Contributor

blueyed commented Jul 10, 2017

FWIW: pytest-incremental is also not compatible with xdist.

@blueyed

This comment has been minimized.

Copy link
Contributor

blueyed commented Aug 2, 2018

It would be awesome if --testmon could be used with xdists's --looponfail, which appears to keep (potentially remote) workers ready (but re-runs all (selected) tests then).
--testmon currently does not contribute to test deselection and appears to not be notified about failures correctly.
The first issue is the case with pytest's own --lf/-ff though already (pytest-dev/pytest-xdist#326).

Are you using local "popen" workers/nodes, or something like --tx ssh=… also?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.