Use a real test runner

[iankronquist]

I believe it would be beneficial to use a test runner like py.test or nose. Test runners provide more detailed feedback, and will run all of the tests even if one fails. Test runners also show test progress instead of leaving developers wondering whether anything is happening.
This would replace test.py and would require updating the readme accordingly.

[JukkaL]

The current test runner actually runs all tests even if some fail, and it can also run only a subset of tests. It could easily be modified to display progress. However, pytest or nose would be more flexible and if somebody wants to port the test cases, I'm fine with it. I want to continue using test description files, however.

It would be good to first have a discussion of the relative merits of pytest/nose/unittest. I've only used unittest and pytest.

Historical background: Many eons ago, mypy was not written in Python but in Alore, and the test framework that mypy currectly uses is a direct port of the Alore unit test framework.

[iankronquist]

I stuck an assert(0) statement in a very early test in the test suite and it skipped 37 other tests. This sounds like it should be two issues:

• Use a test runner
• Port the test suite to unittest (this is big)
  I am opening another issue accordingly.

[refi64]

I vote for using nose or py.test. unittest tends to end up creating overly large tests with lots of long boilerplate code, in addition to having to memorize 200 assertion functions.

nose is more of a test runner, but it's a good one at that. py.test is kind of like nose on steroids+a testing framework.

You might want to read this. It talks about embedding nose, so it might be runnable under mypy.

[gvanrossum]

I favor unittest -- fewer dependencies. It would still be nice if py.test
could optionally be used as a test runner though.

[refi64]

Nose can run unittest tests, too. In reality, all 3(unittest, nose, and py.test) and be swapped in most cases. I just ran Hy's tests with py.test, and they (mostly) passed.

[gnprice]

This is duplicated by #1673, which has current discussion; and this task is now stale because we got a new test runner in 2015. I'll close this one and we can discuss on #1673.