Port home-grown test framework to Busted

[alerque]

@Tieske, @hishamhm —

How would you guys feel about it if I were to port all the current tests from the current in-project test runner to Busted? This would not add a runtime dependency, but it would be an extra build / test / developer dependency.

The issues I have with the current system are that...

• output is pretty inconsistent. Busted can be run with several modes with output suitable for remote CI jobs, local debugging, quick live checks while coding, etc.
• the runner dies of first failure. Busted isolates tests in such a way that one can fail and you still get results from all the tests. This makes it a lot easier to get a feel for the scope of whatever you broke.
• expected failure conditions are not too easy to test. Yes a number of the tests use manual pcall() wrappers, but Busted's expectations for error messages and various conditions are quite a bit more robust.
• testing complex output isn't easy. Partly because table iteration order is non-deterministic in Lua and also for other reasons, more complex tests are cumbersome to write from scratch. Busted has some utilities that ease this pain.
• the runner is single threaded. It's doesn't take that long but test results would be more than $(nproc)× faster if it was easier to run them in parallel. The current framework makes that tricky. Busted doesn't have this built in (yet) either, but the different output options make it much easier to execute from something that does support job control (GNU parallel, GNU Make's test harnesses, etc.)

I make no promise to get around to this right away as it would take a bit of doing, but I thought I'd ask whether such a thing would be acceptable before I get started.

[Tieske]

How would you guys feel about it if I were to port all the current tests from the current in-project test runner to Busted? This would not add a runtime dependency, but it would be an extra build / test / developer dependency.

I've had this thought many times, but lacked the time. So fwiw please go ahead

• the runner is single threaded. It's doesn't take that long but test results would be more than $(nproc)× faster if it was easier to run them in parallel. The current framework makes that tricky. Busted doesn't have this built in (yet) either, but the different output options make it much easier to execute from something that does support job control (GNU parallel, GNU Make's test harnesses, etc.)

that's over-engineering imho, but doesn't hurt either

[Tieske]

I think we can close this now? the whole thing will be a work in progress for a while

[alerque]

Yes. The basics are in place and all new tests should be in Busted. Legacy ones can continue to be ported as we get a chance to refactor them.