need misc updates for dev testing documentation

[mattvonrocketstein]

This issue refers to the documentation here: https://github.com/ipython/ipython/wiki/Dev%3A-Testing

Hopefully I'm looking at the right stuff. One example mentioned is

$ iptest -v IPython.utils
..........................SS..SSS............................S.S...
.........................................................
----------------------------------------------------------------------
 Ran 125 tests in 0.119s

This doesn't work for me, because it should be iptest IPythin.utils -- -v. I see

> iptest -v IPython.utils
usage: iptest [-h] [--all] [--slimerjs] [--url URL] [-j [FAST]] [--xunit] [--coverage [COVERAGE]]
              [--subproc-streams SUBPROC_STREAMS]
              [testgroups [testgroups ...]]
iptest: error: unrecognized arguments: -v

Another broken example is python tools/test_pr.py -p 1234, which broke for me because I didn't have the keyring module installed. Requirements are mentioned in the documentation nearby, but shouldn't this be installed as part of "ipython[all]" requirements? Once I installed keyring, the example still didn't work. Does this run on the official travis, and is only intended for core committers with extra privs, or can I expect to do this on my local env? I see:

 > python ipython/tools/test_pr.py -p 7125
no cache
Please enter your github username and password. These are not stored, only used to get an oAuth token. You can revoke this at any time on Github.
Username: mattvonrocketstein
Password: 
Traceback (most recent call last):
  File "ipython/tools/test_pr.py", line 289, in <module>
    test_pr(args.number, post_results=args.publish, extra_args=extra_args)
  File "ipython/tools/test_pr.py", line 259, in test_pr
    gh_api.get_auth_token()
  File "/home/vagrant/code/ipython/tools/gh_api.py", line 67, in get_auth_token
    response.raise_for_status()
  File "/home/vagrant/code/ipython/ipy_venv/local/lib/python2.7/site-packages/requests/models.py", line 825, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity

Finally and probably most importantly, I think a section on testing extensions is missing. Extension writers can unittest their own stuff however they wish, but obviously the more integration-style tests used in ipython core could be a very useful for complex extensions. Are there any existing thoughts about best practice here or better yet examples of third-party code doing this? Is iptest pretty agnostic about whether it's invoked on IPython.* or MyExtension.*? etc

Thanks

[takluyver]

• I've fixed the -v thing. It is a wiki, if there are obvious problems like that, feel free to fix them yourself.
• I guess a change in the Github API broke test_pr.py. Maybe we should get rid of that script; we used it before Travis did PR testing. In principle, it's still useful if people modify the parallel code, because that isn't tested on Travis, but in practice we very rarely use it now. I'm just a bit sentimental about it because I wrote it. ;-)
• I can imagine it may be useful for extension authors to use parts of our testing machinery, like IPython.testing.globalipapp. I imagine they would do this from a Python API inside their tests, rather than actually calling iptest. Do you have an extension you'd like to create more tests for?

[mattvonrocketstein]

I've fixed the -v thing. It is a wiki, if there are obvious problems like that, feel free to fix them yourself.

Great, I did not realize it was a wiki, thought it was markdown in the main source. As for the test_pr.py thing, I don't really have a dog in that fight, and I just thought someone might like to know. All I can say is I guess I might use it if it worked.. I'm always nervous my testing invocation will differ in some significant way from whatever travis uses and it's just inelegant to waste travis build time/cycles.

I can imagine it may be useful for extension authors to use parts of our testing machinery, like IPython.testing.globalipapp. I imagine they would do this from a Python API inside their tests, rather than actually calling iptest. Do you have an extension you'd like to create more tests for?

I am the author of smash which does involve several extensions, but also digs really deep into the guts of ipython (trying hard to avoid a fork). Just for instance see some of my patches and core overrides. Note that while I'm twisting up the ipython core with extra features, I also want to ensure total backwards compatibility.

So I'm struggling with a good approach for general testing, and hoped to improve documentation for posterity while I was learning my way around. It seems to me that vanilla unit tests (by which I mean tests that do NOT use _ip or get_ipython()) will not work for me in the long term. I've been looking over globalipapp a little, but I thought that perhaps using that would involve iptest anyway to setup other aspects of a bigger testing harness. Any advice/thoughts would be much appreciated. The point of opening this ticket is that I planned to return the favor with fresh docs on the wiki and/or docs pull requests :)

[ellisonbg]

Any progress on this? Is there anything we need to do?

[mattvonrocketstein]

Needs info from me you mean? I'd say nothing has been resolved. Trying to make things more concrete:

1. Does test_pr.py get to stay or does it go?
2. Should there or should there not be documentation on strategies for writing extension tests?
3. If there should be documentation for it, I'm trying to write it but I need some hints first.

Unit testing is straightforward for third party apps since they can do whatever, but, how best to write and run IPython.core style integration tests outside of IPython development?

[ellisonbg]

Sorry, I just went through a couple hundred issues and ended up adding needs-info to issues that I wasn't quite sure what the state was. Thanks, your note helps to clarify what needs to be done though.

[ellisonbg]

I should note I am fine getting rid of test_pr. @fperez do you still use it?

[takluyver]

test_pr is theoretically still useful for things we don't test on Travis, like IPython.parallel, or if Travis is down/achingly slow. We don't often use it, however, and presumably there'll be even less need for it after the repo split.

[fperez]

Now that I have git mpr and most cases are taken care of by Travis, the need for test_pr is significantly lower. I'd be OK if it goes, if that helps simplify things.

[ivanov]

Hey there. I'm going through old issues and it seems to me that it makes sense to close this one. There were multiple threads here, some of which have been addressed: wiki was edited, and a decision was made that removing test_pr.py is fine (I've just posted #10041 to do just that). The last bit which wasn't addressed - which is to document strategies for writing extensions, is clouded by the other stuff. Documentation updates are always welcome, no point in having an issue open to state that (and in a roundabout manner.)

[ivanov]

oh, man, I forgot the two most important parts: thank you everyone and
happy hacking!