adding a autoplay tests to verify the prompt output is not changing

[coryb]

@AlecAivazis, this stuff is for #46. As mentioned in the issue, expect doesn't look like it will work out after all, so I wrote something that will basically simulate the autoexpect functionality to record a program interaction and then generate go-code that we can use for verifying prompts.

This PR really consists of 2 things:

1. the record/playback generator in tests/recorder/recorder.go
2. the playback generated tests in tests/autoplay/*.go

To run the autoplay tests you need to be in the tests directory so:

cd tests
go run ./autoplay/selectThanInput.go

To generate a new test or to update an existing test after changes you can run:

cd tests
go run ./recorder/recorder.go -- selectThanInput.go

Which will update the existing tests.

The test structure is all identical. We capture input and output to the program via the recorder process which when played back will re-execute the program and provide the input while waiting for the output. If the output does not match the autoplay tests will just panic with a message about what it was expecting and what it got (and fail the travis build).

I would just look at tests/autoplay/selectThanInput.go to get an idea of what is generated and tests/recorder/recorder.go to see how it is generated. All the rest of the stuff in tests/autoplay is just the same stuff generated with different input/output.

Anyway, play around with it and let me know what you think.

[AlecAivazis]

This package is really cool! I have been suffering with the problem of how to test clis for close to half a year and this is really nice approach. How would you feel about making this an external package? I would love to use it for some projects i'm working on

[coryb]

Yeah, I thought about it. I do make some assumptions about what is being recorded (ie go run is hardcoded) so some abstractions need to be made. Now the hard part, what to call it. I am leaning towards oscillo (as in the scope for tracing), or we could be puny and call it goscillo which seems a bit obnoxious :)

Btw, if you need basic cli testing and dont need to deal with fancy interactive prompts, then I have a useful project I use for testing most of my cli projects: https://github.com/coryb/osht, it makes testing via bash very easy.

[AlecAivazis]

Oh nice! Thanks for sharing. This looks like a great option for "not fancy" prompts. And I agree goscillo seems a bit... forced. Naming things is hard, survey used to be called probe which changed for obvious reasons haha

[AlecAivazis]

FWIW, i really liked autoplay although I also like the connection between "scope" and watching the cli do its thing

[coryb]

autoplay works for me. I am in the process of making the abstractions necessary to make it a generic replay tool. I will update the PR when it is done.

[coryb]

Hey, created github.com/coryb/autoplay repo with the recorder code moved over, I added you as a contributor to that repo. I have updated the tests.

The travis tests seem to be failing, but it works on my system, so I will have to debug that one now...

[AlecAivazis]

Awesome! Got the invite, will take a look now. Thanks! I think it would be really cool to build this library up a bit to support writing test suites around it. For example, I might want to create an in-memory server before running a recorded test to verify that its output matches what the server is sending

[AlecAivazis]

I haven't gotten a chance to run the tests yet but they pass in travis so I assume it works. I'll pull this down and test it tonight or tomorrow at the latest

[coryb]

I am struggling with this one, shooting in the dark at the moment. On travis in mid execution rl.Readline() is getting an EOF. I suspected maybe it was a buffering issue and implemented a retry on EOF, but now the test just hangs forever. I also suspected it was just the autoplay or pty having issues, but when I tried running the tests on Linux manually without autoplay then I still saw the EOF errors. I very rarely see them on OSX (although when I run one of the autoplay tests in a tight loop it fails about 1% of the time with an EOF error) but on Linux it seems to happen much more frequently.

[coryb]

I am going to put this work on hold, too many odd issues with Readline. I started playing around with survey on a Windows VM and it looks like the recent changes for the renderer are not working correctly (prompt rewriting not working right). So I am going to borrow a Windows laptop for a while and see if I can figure out a solution that will work reliably on all the platforms.

[AlecAivazis]

Damn, okay - thanks for trying. I have been struggling with this for awhile so i know the pain you've been experiencing. Thanks for looking into the windows bug, I thought I reviewed that PR on my windows machine before merging it but I guess something slipped through.

[AlecAivazis]

@coryb i just tried to run the tests on a windows laptop and was not running into errors apart from poor unicode support, what sort of errors are you seeing on your side?

[coryb]

Very strange, my only windows access currently is a vagrant VM, but when I ran go run tests\confirm.go and saw:

Enter 'yes'
? yes: (y/N) 
             ? yes: Yes
Answered true.
---------------------
Enter 'no'
? yes: (y/N)
             ? yes: No
Answered false.
---------------------

This is via cmd.exe. So the prompts are not echoing (when I type yes it is not displayed). Then the cursor is not moving up or returning to position 0 to rewrite the prompt (it is odd that the \n is echoing though). I thought the echoing might just be black on black, but when I highlight for copy/paste I don't see the echo'd answers either.

Not sure how you were testing. So this could be something via vagrant although I dont see how, or perhaps you are using a different terminal other than cmd.exe?

[AlecAivazis]

Hm, so I only ran the various multiline prompts, i'll check confirm later tonight. I am running cmd.exe too - which version of windows are you testing on? Let's open a new ticket for this bug so we dont clutter the PR with non-sequitors

[coryb]

Woot! Updated autoplay tests and they are passing under Travis now.

[AlecAivazis]

Sweet, I think it's probably rather out of date with the recent changes, but once it's ready i'll get this in.

[coryb]

Hmm, I did the merge but the PR seems pretty messed up. I think I will just nuke this PR and make a new one that is clean.