Iptest refactoring

[takluyver]

This refactors iptest to bring it a bit more up to date. There's probably more work that can be done, but I think this is a good point to do a first PR on it.

The main changes:

• The machinery to run a testing subprocess has been split out into iptestcontroller and rationalised. Now you create an IPTestController instance, set up its command line arguments, Python code and environment variables, and then launch it.
• Command line parsing is now done by argparse, rather than lots of if '--with-xunit' in argv:. This makes it easier to add more options later, and also gives us simple ways to get combined coverage reports (and we should later be able to get coverage from subprocesses, but that's a question for another PR).
• Instead of a list of test groups and a list of exclusions, we now have a list of TestSection instances, each of which has inclusions and exclusions, as well as a list of requirements. This allows us to exclude tests from one section but include them in another (e.g. autoreload, kernel.inprocess) in a clean way. I've rearranged the specification of dependencies and exclusions to group them by section.
• Exclusions are now handled by a separate nose plugin (ExclusionsPlugin) rather than being an exciting side-effect of our ipdoctest plugin.

[takluyver]

• Ensure that we can still run more specific parts of the test suite.

[minrk]

I presume IPTestApp should be a BaseIPythonApplication subclass, consistent with the rest of IPython.

[ellisonbg]

Cool, I am excited to have a look at this. @ivanov have you talked to @takluyver about how we are going to build the JS/casper/phantom tests into this architecture. It is almost like we need a more general abstraction of a test seciton - one that is not run by nose.

[ellisonbg]

It would be good to look at what the needed abstractions are for that. Do you think you could look into that and post back on this PR?

[takluyver]

IPTestController is fairly general - it should be easy to split it into a couple of subclasses for running different kinds of tests.

[takluyver]

I've refactored it so there's a TestController base class and a PyTestController subclass. When we've got JS tests, we should be able to make another subclass which runs the JS tests.

[takluyver]

There's a problem with a debugger doctest printing different output when iptest runs in a subprocess versus being run directly, which I'm working on debugging.

[takluyver]

Found and worked around the doctest issue, details in commit message.

[ivanov]

i've got a bunch of feedback here, just working on it locally today for now. Once I have all my ducks lined up in a row and figure out what modifications make sense, I'll make a PR against this branch

[takluyver]

It's now also possible to run more than one section together, e.g. iptest core lib utils.

I should also have mentioned before, this allows iptest core instead of iptest IPython.core. The latter still works, in case we're too used to typing that.

[ivanov]

sweet, thanks for accommodating the changes from the peanut (butter pretzel puffs) gallery. looks great now, being able to run multiple test groups in parallel will also come in handy. merging.