Add simple test mechanism for interactive commands

[briandfoy]

There are some commands that take interactive input, such as bc or ed. I'd like to have a test mechanism where we can either provide input line by line and test the result (output), or provide a series of lines and test the final output. This sort of thing already exists and just needs to be here.

[jgpuckering]

Ok, I think I have one for you. It's called Test::Cmd -- although it isn't a core routine. It is in CPAN though and works down to perl 5.6. Written by Neil Bowers and with over 6000 testers. You can try out the attached test script, which has 36 tests of bc.

bc_t.zip

By default it will attempt to locate bc in ../bin/bc relative to the bin in which it is executing-- which is expected to be PerlPowerTools/t. But you can put the path to bc on the command line and it'll use that.

If you think this is worth adding, I'd like to create a branch called test_scripts (or maybe issue_140 if you prefer), I'll then write whatever test scripts I can, commit them to that branch, and then push it to the repo for review and possible merge into master.

[briandfoy]

As far as I can tell, Test::Cmd doesn't do what I want. I'd like something where I can send one line of input to a command then get it's output, and then send another line, and get it's output. Think about testing things like bc or ed where you want to see the state of their buffers after every input.

The answer is probably something with IPC::Open3, but I'd like a better interface to it so it makes sense in our tests.

[jgpuckering]

If I understand you correctly, I can do just that with Test::Cmd. For example:

my $test_obj = Test::Cmd->new( prog => $Script, interpreter => 'perl', workdir => '' );

$test_obj->run( stdin => <<"__data" );
v=3
++v
__data

is( $test_obj->stdout, "3\n4\n" );

I could have set up the 36 tests of bc expressions that I included with the file I sent you as a single test like this, but a failure anywhere would result in a very messy report.

My script produces this output:

C:\Sandbox\PerlPowerTools> perl ..\bc.t bin\bc
1..36
ok 1 - negation: -1
ok 2 - variable assignment: var=12
ok 3 - prefix increment: v=3; ++v
ok 4 - prefix increment: v=3; --v
ok 5 - postfix increment: v=3; v++; v
ok 6 - postfix increment: v=3; v--; v
ok 7 - postfix increment: v=3; v+=5; v
ok 8 - postfix decrement: v=5; v-=3; v
ok 9 - addition: 1+2
ok 10 - subtraction: 5-3
ok 11 - multiplication: 3*5
ok 12 - division: 15/3
ok 13 - remainder zero: 15%3
ok 14 - remainder four: 29%5
ok 15 - exponentiation: 2^15
ok 16 - expr without parens: 3*2+5
ok 17 - expr with parens: 3*(2+5)
ok 18 - sqrt: sqrt(16)
ok 19 - boolean not: !0
ok 20 - boolean not: !1
ok 21 - boolean and: 1 && 1
ok 22 - boolean and: 1 && 0
ok 23 - boolean and: 0 && 1
ok 24 - boolean and: 0 && 0
ok 25 - boolean or: 1 || 1
ok 26 - boolean or: 1 || 0
ok 27 - boolean or: 0 || 1
ok 28 - boolean or: 0 || 0
ok 29 - equals (true): 9 == 9
ok 30 - equals (false): 9 == 8
ok 31 - not equals (true): 9 != 8
ok 32 - not equals (false): 9 != 9
ok 33 - less than: 5 < 7
ok 34 - less than or equal: 5 <= 5
ok 35 - greater than: 7 > 5
ok 36 - greater than or equal: 7 >= 7

What my script doesn't do is catch a problem arising from some kind of state error, since every execution is independent of the previous one. But those errors are highly subject to the order of input. I was striving for basic functionality testing first.

It takes 3.8 seconds to run that set of tests on my desktop computer. Sure, it might be faster if you could set up a pipeline between the input generator and bc and the tester, but is it worth the hassle? Will it be reliable across platforms?

Another way to approach this is to do what I did in the units script. I modified it to allow it to be require'd in the test script, and provided a test( ) function that essentially does what run( ) would do except it gets input from parameters instead of stdin, and it delivers output as data instead of printing it. That could probably be done with bc as well.

You should give bc.t a try. I think it's a solution that's "good enough" -- and certainly better than no tests at all!

P.S. As a point of comparison, the 28 tests in units.t take 1.47 seconds to execute on my machine vs 3.8 seconds for the 36 tests in bc.t.

[jgpuckering]

While I'm here I may as well save you the trouble of trying bc.t yourself. There's not much to it besides the table that defines the tests, Here's what that looks like:

my @test_table = (
    [ '-1',                '-1', 'negation'                 ],
    [ 'var=12',            '12', 'variable assignment'      ],
    [ 'v=3; ++v',        "3\n4", 'prefix increment'         ],
    [ 'v=3; --v',        "3\n2", 'prefix increment'         ],
    [ 'v=3; v++; v',  "3\n3\n4", 'postfix increment'        ],
    [ 'v=3; v--; v',  "3\n3\n2", 'postfix increment'        ],
    [ 'v=3; v+=5; v', "3\n8\n8", 'postfix increment'        ],
    [ 'v=5; v-=3; v', "5\n2\n2", 'postfix decrement'        ],
    [ '1+2',                '3', 'addition'                 ],
    [ '5-3',                '2', 'subtraction'              ],
    [ '3*5',               '15', 'multiplication'           ],
    [ '15/3',               '5', 'division'                 ],
    [ '15%3',               '0', 'remainder zero'           ],
    [ '29%5',               '4', 'remainder four'           ],
    [ '2^15',           '32768', 'exponentiation'           ],
    [ '3*2+5',             '11', 'expr without parens'      ],
    [ '3*(2+5)',           '21', 'expr with parens'         ],
    [ 'sqrt(16)',           '4', 'sqrt'                     ],
    [ '!0',                 '1', 'boolean not'              ],
    [ '!1',                 '0', 'boolean not'              ],
    [ '1 && 1',             '1', 'boolean and'              ],
    [ '1 && 0',             '0', 'boolean and'              ],
    [ '0 && 1',             '0', 'boolean and'              ],
    [ '0 && 0',             '0', 'boolean and'              ],
    [ '1 || 1',             '1', 'boolean or'               ],
    [ '1 || 0',             '1', 'boolean or'               ],
    [ '0 || 1',             '1', 'boolean or'               ],
    [ '0 || 0',             '0', 'boolean or'               ],
    [ '9 == 9',             '1', 'equals (true)'            ],
    [ '9 == 8',             '0', 'equals (false)'           ],
    [ '9 != 8',             '1', 'not equals (true)'        ],
    [ '9 != 9',             '0', 'not equals (false)'       ],
    [ '5 < 7',              '1', 'less than'                ],
    [ '5 <= 5',             '1', 'less than or equal'       ],
    [ '7 > 5',              '1', 'greater than'             ],
    [ '7 >= 7',             '1', 'greater than or equal'    ],
);

The rest is a foreach loop and an invocation of Test::Cmd run() as described in my previous message.

[briandfoy]

You haven't done what I'm imagining with Test::Cmd: you gave it all the input at once and got the final output. This issue is about not doing that. I want to give one line of input, get the output, give another line of input. This way you know if a particular line of input did what it should do without having to do a lot of ad-hoc work to match up input with output.

---

The tests for bc are fine, but they aren't what I'm talking about here. But, we don't really need Test::Cmd for that. All of that can be done with IPC::Open3 just as easily, and if we can avoid a dependency we should. Installing things in the GitHub Actions already takes too long.

I looked at the tests you had for t/units.t and I'm about to merge a major refactor of it so it's table-oriented like you just saw. There are some other modulino tricks I added too. It's fine that you added the test method to units, but that works because units is doing something very simple. It's not appropriate for most other programs.

[briandfoy]

Besides what I just wrote, you should add those tests to bc. That's good work, just for a different issue. :)

[jgpuckering]

Ah, ok that helps. I wasn't against a solution using IPC::Open3. I'd love to see a working example. Maybe I can use it instead of Test::Cmd.

I do get your point about minimizing dependencies in the CI environment.

A drawback to using Test::Cmd is the process overhead. Even though it might only take 3 seconds to run 30 or so tests, we have 120 scripts to test and so that adds up. A faster solution would be better.

I fiddled with Open3 a bit but so far a working example has eluded me. Maybe I'll have better luck tomorrow.

[briandfoy]

After working through this issue in multiple ways, I've come to the conclusion that it's not possible to get this working on Windows. If it were, we'd have a CPAN module to do it. If it can't work on Windows, it's not something we can do here.