Move pythoneval and cmdline to pytest

[elazarg]

(EDITED)
Another item from #1673, following #3780, #3788, #3861 and #3866.

After this, StubgenPythonSuite is the only data-driven test left in myunit.

As in previous PRs, I have checked that a test may fail by changing tests and running

./runtests.py eval -a -n0 -a -k -a testHello
./runtests.py eval -a -n0 -a -k -a testSimpleCoroutineSleep
./runtests.py cmd -a -n0 -a -k -a testCmdlinePackageContainingSubdir

I am somewhat less confident about this PR, since I am not entirely sure what special considerations are in play here (which aren't there for the proper unit tests).

@ambv @gvanrossum: The splitting of eval tests, introduced in PR #2635, is removed. IIUC this was the intention - please correct me if I'm wrong. (I did not time the difference).

Executing multiple pytest instances does not work well with appveyor.

[elazarg]

The diff is somewhat confusing - this part belongs to add_myunit()

[gvanrossum]

This all looks fine to me. @ambv if you have something to say, say it soon or I'll just merge this.

[ambv]

LGTM.

[gvanrossum]

Sadly there's a problem with this. When I run the tests as follows:

pytest -n0 mypy/test/testcmdline.py

every test in that file fails. Apparently the pytest-xdist extension which runs tests in parallel does something that keeps the tests from failing. The error is essentially that every subprocess invoking scripts/mypy fails with the following traceback:

Traceback (most recent call last):
  File "/Users/guido/src/mypy/scripts/mypy", line 4, in <module>
    from mypy.main import main
ModuleNotFoundError: No module named 'mypy.main'

This makes sense as the current directory is set to the temporary directory where the test files are written, so the repo root is not on sys.path. I'm guessing that the xdist extension somehow retains the current directory?

The pythoneval tests fail in the same way.

[gvanrossum]

(And yes, I spent 2 hours trying to debug unrelated code because I didn't realize -n0 always made these tests fail.)

[elazarg]

Ouch. Sorry.

I recall checking for that before opening this PR, and it did not fail on my machine :\ I use -n0 pretty often. But maybe I didn't use it for cmdline specifically; for some reason, exactly one test (testCoberturaParser) fails for me now:

Expected:
  <coverage timestamp="$TIMESTAMP" version="$VERSION" line-rate="0.8000" branch-rate="0"> (diff)
  ...
Actual:
  <coverage timestamp="$TIMESTAMP" version="0.530" line-rate="0.8000" branch-rate="0"> (diff)
  ...

[elazarg]

That test indeed does not fail without -n0.

[gvanrossum]

I wonder if you have mypy installed? That version looks like it comes from the installed version. If you pip3 uninstall it, do more tests fail?

[elazarg]

Yes they are! Or actually, they didn't fail - until I've commented out the patch :)

[elazarg]

But actually this means my patch is the wrong solution. We need to make sure the folder is the right one, and simply adding it to PYTHONPATH does not guarantee it, I assume.

[gvanrossum]

Use mypy.api.run()? (See other issue.)