Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Functional and unit tests for CI #414
following #222 and all the others discussion about testing. It took me some time, but I finally found out a decent way to implement functional and unit testing for (neo)mutt.
Those are tests being done using the mutt runtime, calling commands using mutt's eventloop. To achieve that, the Lua API is a good way to go #298.
So I've done exactly that: https://gist.github.com/35be5c581aaabf6f1c453cbc2a6e6147
A few tests that are ran by calling:
Those tests will not be included in the Lua PR #298.
those ones are a bit trickier. The idea is to to access each header of mutt and test the API exposed through the compiled objects. I've achieved this by using Luajit+FFI, and compiling mutt as a big shared library. To complete the testing toolchain (and load all the headers and symbols), I've stolen a bunch of lua files from neovim/neovim
And then I wrote a test case for the lists (babysteps!): https://gist.github.com/b12b948616341d8dffd825ae9a5efcea
referenced this issue
Feb 19, 2017
While I think using Lua for functional tests could be a fair solution and something worth investigating, I object to using any foreign language + FFI for unit testing. There are a few reasons for that.
I strongly disagree about the two above. If tests are the best source of documentation for your code, and not the code itself (and the few comments to help understanding of the function), then the project has real design and usabality issues. Furthermore, tests are not designed to be showing HOW the tested function works, but cover not only the general usecase, but all the edge cases. That would mean a lot of redoing the same test with very little differences.
Which is why I'm strongly opposed to keep code and testing of the code in the same compilation unit. Because then, you'd 90% of the source file only being about testing the 10% that are drown within the tests.
N.B.: to be clear, I'm not opposed to the discussion, I'm strongly opposed to keep unit tests within the same compilation unit as the code being tested. For the reasons exposed above.
You can work with pointers in luajit+ffi, but that's right that you need some things are a bit more difficult, and for those you can introduce C helper functions. Then you'd argue it's adding a layer of abstraction, and you'd be right. But what would really be the harm?
that code can eventually be put as an external library that would be shared by projects like neovim and ours, pulled from the luarocks package repository.
so you asked what's good about lua test harness?
The bad sides?
What really led me to this choice are the two first good points listed above, as I believe consistency and accessibility are two strong feat of this solution. Then, I also have to admit that implementing this test harness has been driven by neovim's experience over the last 3 years, as they followed a similar path with another codebase.
For all the parts that are adding complexity there's the possibility to factorise the work, and take advantage of having a similar effort on both projects to make that extra complexity an external lua dependency (on luarocks).
to prove I'm not narrow minded, I stumbled across that small test framework:
that ticks a few checks:
might be worth trying it out.
would keeping unit tests inside the source code have any advantage?
This branch has been archived to: https://github.com/neomutt/neomutt-old/tree/testing/testing