pytest-xdist support

[hannahdiels]

When combining testmon and xdist, more tests are rerun than necessary. I suspect the xdist runners don't update .testmondata.

Is testmon compatible with xdist or is this due to misconfiguration?

[tarpas]

Hi, unfortunatelly testmon is not compatible with xdist.
ᐧ

On Thu, Mar 24, 2016 at 12:19 PM, Tim Diels notifications@github.com
wrote:

When combining testmon and xdist, more tests are rerun than necessary. I
suspect the xdist runners don't update .testmondata.

Is testmon compatible with xdist or is this due to misconfiguration?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#42

[boxed]

Do you know what the problem is? If it's described in this ticket, maybe someone can take a look?

[tarpas]

No I don't know what the problem is and it's not described anywhere. What do you expect from combining xdist and testmon? Wha'ts your use case, please?

[boxed]

The use case is the same as for xdist: if you have a lot of tests they can benefit from being run in parallel.

So the scenario would be:

1. you change some file and testmon detects 100 (slow integration) tests that needs to be run
2. testmon starts the tests with xdist enabled and the tests run in parallel

[tarpas]

Ok, right. I'm currently thinking about lightning fast, CPU efficient, interactive re-execution of small test suites. (which wouldn't benefit because of the cost of starting-up multiple interpreters, and because of IPC)

When I said testmon is not compatible with xdist I meant I didn't test it and didn't invest the time to think about it. The problem is what @timdiels describes. After thinking about it here are my ideas:
If the slaves run at remote machine or in different directories, then .testmondata doesn't get the data from slaves. How to solve this?
pytest-testmon would have to send the coverage data to the master in similar fashion to what xdist does with reports:
https://github.com/pytest-dev/pytest-xdist/blob/master/xdist/remote.py#L92

Poor man's solution for the cases when the slaves actually run on the same machine in the same directory, would be to just make sure the different processes don't overwrite each other, which I'm pretty sure is happening. To achieve that the .testmondata format would have to change, so that each nodedata is stored in a separate row of the "alldata" sqlite table. That would be a good change for a number of other reasons anyway.

[notestaff]

"nodedata is stored in a separate row of the "alldata" sqlite table." -- concurrent writes would still be a problem, no? Wouldn't you want to store each test node's data in a separate file, to avoid conflicts?

[tarpas]

No, as far as I know it's fine and Sqlite takes care of everything better than I could manually https://stackoverflow.com/questions/4060772/sqlite3-concurrent-access

[notestaff]

I've seen "database locked" errors with -n 2, that go away with disabling xdist. Maybe, adding retries would fix that?

[notestaff]

@tarpas btw I saw you held a fundraiser for this project. I bet travis-ci would sponsor it, since it'd save them a done of computation if people used it in conjunction with travis' caching of specified directories between builds ( https://docs.travis-ci.com/user/caching/#Arbitrary-directories )

[tarpas]

Is the "database locked" error the same as here?: #68 (comment) , if yes, it's been fixed in version 0.9.6

[tarpas]

@notestaff About travis-ci: I know, right! :) I see some downsides however. Does anybody have a tip whom to contact at Travis-CI?

[blueyed]

One downside appears to be that this won't work when collecting coverage for your tests, which is rather typical I assume?!
Also for CI it is more important to really run all tests (and not skipping some accidentally because of some potential issue with pytest-testmon or how the cache/data was synced etc).

[notestaff]

You can save coverage info, e.g. as in https://github.com/notestaff/pytest-testmon/tree/is-save-cov
You give up sub-file resolution.

There are degrees of CI; for our use, we plan to skip tests for routine branch checkins and pull requests but clear the cache and run all tests for tagged releases. The flipside of not skipping tests is that you get wary of adding even meaningful tests to the test suite, since the commit-then-test cycle becomes too slow as the test suite grows.