Black box tests on Tomty

[melezhik]

Hi! the idea is to have end user tests via black box tests.

The additional advantage is testing
infrastructure is fully integrated into RakuDist and no efforts are required to spin up
test environment (but you could run tests locally if you need too).

I know current unit tests could achieve pretty
much the same, at least at this simple
case - run ake and parse an output ...

But again idea is to have tests
decoupled from implementation. And there are
other cases where unit tests are not
the best choice.

Anyway, take it or leave it 😀

It"s just a thoughts and maybe it could be
somehow useful ...

I'd aporeatite your feedback.

[melezhik]

I forgot to attach a link to RakuDist build.

http://rakudist.raku.org/sparky/report/debian/315

[AlexDaniel]

I really don't understand how this is any better than black box tests that we already have: https://github.com/Raku/ake/tree/master/t. In fact, it looks worse to me because there are so many files? I don't understand, sorry.

[melezhik]

@AlexDaniel this is a very Sparrow design. You wrote tests on the language that is closest to domain ( shell because this is how people normally would run Ake scripts ), you parse stdout using check files ( optional step ), you orchestrate tests using Raku. Yeah there are more files in average in comparison to unit tests approach, but you have pure back testing.

As result:

Those tests are much easier to maintain, they are just simpler, they are 100 black box ( 'master/t' are not because they are coupled with implementation ) and they test everything from end user prospective.

[AlexDaniel]

'master/t' are not because they are coupled with implementation

Look at them:

ake/t/21-tasks.t

Lines 9 to 12 in a598347

	given make-ake-directory ｢task ‘foo’, { put ‘hello’ }｣ {
	test-run ‘simple task’,
	<ake foo>, :out(“hello\n”)
	}

It dumps the given string into a file named “Akefile”, then it runs ake foo and checks the output. I don't see how you can get more black box than this, and I see nothing that needs to be improved.

[melezhik]

They are coupled with an implementation in a sense you use suplimetal methods to run Ake task. Even this simple example you reference has make-ake-directory and test-run functions. it's OK, but it's extra code. Chances are with more complex scenarios you eventually add more layers. Again it's OK for unit tests where you try to test inner structure of a code. But is it OK for black box testing? This is a question.

With Sparrow you almost always get more simple and more closer to an end user approach. Just get an Ake file and get it run as a shell script. There is no even difference between Sparrow task ( shell script ) and Ake scenario gets run as a shell script. They are the same. Sparrow framework just adds some validation logic out of the box.

[AlexDaniel]

Even this simple example you reference has make-ake-directory and test-run functions

They're not part of the implementation, but part of the tests: https://github.com/Raku/ake/blob/master/t/lib/AkeTester.rakumod. I hope we can agree that 34 lines of code is not a lot. And with these 34 lines I have full control of how these tests are done.

But is it OK for black box testing? This is a question.

Yes.

and Ake scenario gets run as a shell script. They are the same.

No, shell is completely avoided here, which is good.

[AlexDaniel]

Just to clarify, there are probably bigger projects where Sparrow can indeed be very useful, but Ake is not one of them. I really fail to find any justification for changing the way tests are done here.

[melezhik]

hi Alex. NP. i'm just testing a water and any feedback is ok.

[AlexDaniel]

@melezhik I think Sparrow needs a way to write single-file tests or even to pack more than one test into a single file. This is an approach that's taken by some modern projects. For example, see Vue.js (it's different, but it's a good example how different files that define a singe thing can be grouped into one file). Without this users are forced to split their tests across many files, which makes everything just messy and hard to follow.

[melezhik]

yeah. the thing is Sparrow is not a testing framework ( though it has testing framework features). it's an automation framework that effectively glues various languages into Raku high level scenarios. one can always have a list of tasks in one Raku file.

i understand you concern about having more then one file, still don't see it messy.
it's just a separation for different types of entities - check files, tasks ( primitives) and high level Raku scenarios.

[niner]

On Sonntag, 14. Juni 2020 15:38:28 CEST Alexey Melezhik wrote: yeah. the thing is Sparrow is not a testing framework

Then why do you push for replacing actual testing frameworks with Sparrow?

[melezhik]

hi @niner i never said replacement. i am suggesting one more way to test it with some possible benefits. both solutions could coexist in parallel.

[JJ]

Is this still relevant? Should we merge it?

[melezhik]

@JJ i will probably add tomty support to sparkyci so these tests could be run additionally to standard unit tests …

[JJ]

So, for the time being, can we close this?

[melezhik]

Yes