write cram tests

[myusuf3]

https://bitheap.org/cram/

[dbr]

This seems interesting, but I'm not sure how helpful it'll be for the main octogit tests..

To properly test the various commands, the hardest part will be stubbing out the returned data from the Github API, so we can test stuff like octogit login or octogit create (or even just octogit issues without having to create a dummy Github account)

Also, since cram requires the command output look pretty much the same, I suspect it'll mean all the tests break as soon as you tweak the UI in any way.. Just checking octogit's exit code might be enough in most cases (and for things like octogit issue 12, additionally checking if the issue title is in stdout)

For tvnamer, I wrote a function which takes some arguments (like the input filenames, a list of command line arguments, a config file which gets written to a temp file and passed as an argument). Then it turns these into shell args, and executes main.py like..

subprocess.Popen([sys.executable, path_to_mainpy, 'inputfile1.avi', 'inputfile2.avi', '--config', '/tmp/blah.json'])

..all of which lets me write tests like this.

It's worked really nicely - the tests are flexible and easy to write, and easy to debug. Also they are executed like any other of the regular test-cases, via nose

[myusuf3]

@dbr I really like this idea. I am going to be refactoring the way the cli works for to work with docopt so as soon as I do that maybe you an spear head setting up something like what you have setup for tvnamer

[Satshabad]

@dbr, @myusuf3 Is there a reason it's not a good idea to create a dummy account for testing? It seems like as long as the test repos and issues are cleaned up it wouldn't be too bad. I suppose it might be a little slow to run all the tests though, and it might take some work to conceal the credentials from everyone.

If that's not a great idea we can use a monkey patch like this http://stackoverflow.com/a/295481 to stub out the github api with some fake data. The disadvantage of that is, we won't know if the github api works from the tests.

Thoughts?

[myusuf3]

I think we should go the monkey patching route, and keep a close eye for api changes, you will want to take a look at things like Mock.

[dbr]

Agreed. Having a secret dummy account will make it difficult for contributors to run the tests, stubbed responses means they can be run by all, and quickly.

Would be good to have a script which grabs the stubbed responses, and also a way to easily run against a live dummy-account (maybe run like this on travis-ci with the secure env-vars?)

..and would also be good if I can find the time to actually implement this.

[Satshabad]

Is this kind of what you are thinking? I am having trouble wrapping my head around mocks, especially since so much of the code deals with the file/config system and the github api. Could someone maybe give me some pointers here? I tried mocking some requests stuff in this commit: https://github.com/Satshabad/octogit/tree/2ebc7101e8c6953679bf82c7d4de1f0e410817dc, Is that kind of what we want?

I am also having trouble importing the local octogit package as opposed to the system-wide installed package. Any ideas on how to fix that? Not even an hour of googling helped me there.

Super new to this mock testing idea so any help would be appreciated, thanks!

[Satshabad]

OK I've learned a ton in the past few days. I think a combination of stubs and mocks will suffice for the tests. Now the question I have is: What should we test? I started writing tests for the config module, but I think it's way too tedious to write the mock/stub tests for every simple function.

Any suggestions?

[myusuf3]

Sorry been super busy. I will comment here ASAP.

[Satshabad]

No worries. I submitted a pull request so perhaps some other can also look at the code as well. Definitely not ready, but it's in the style of early/often review.