New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Functional and unit tests for CI #414

Closed
guyzmo opened this Issue Feb 19, 2017 · 7 comments

Comments

@guyzmo
Contributor

guyzmo commented Feb 19, 2017

following #222 and all the others discussion about testing. It took me some time, but I finally found out a decent way to implement functional and unit testing for (neo)mutt.

functional tests

Those are tests being done using the mutt runtime, calling commands using mutt's eventloop. To achieve that, the Lua API is a good way to go #298.

So I've done exactly that: https://gist.github.com/35be5c581aaabf6f1c453cbc2a6e6147

A few tests that are ran by calling:

build/mutt -F tests/functional/test_runner.muttrc --batch

Those tests will not be included in the Lua PR #298.

unit tests

those ones are a bit trickier. The idea is to to access each header of mutt and test the API exposed through the compiled objects. I've achieved this by using Luajit+FFI, and compiling mutt as a big shared library. To complete the testing toolchain (and load all the headers and symbols), I've stolen a bunch of lua files from neovim/neovim 😀 like helpers.lua.

And then I wrote a test case for the lists (babysteps!): https://gist.github.com/b12b948616341d8dffd825ae9a5efcea

@gahr

This comment has been minimized.

Member

gahr commented Feb 20, 2017

While I think using Lua for functional tests could be a fair solution and something worth investigating, I object to using any foreign language + FFI for unit testing. There are a few reasons for that.

  • unit tests are often the best documentation available on a particular function or data structure. In contrast to comments-based documentation (read: doxygen)- which we all know tends to rot away - unit tests are forced to be kept in sync with the functionality the test when this evolves. Having them in a different language prevents them from offering concrete examples on how to use or not use a particular API;
  • having unit tests in C would allow us to at least have a discussion about where the unit tests code should go, either in separate files or near the code they test. Using a foreign language would preclude even the possibility to have such a discussion;
  • unit tests in C would allow for coverage analysis. I understand not everybody is enthusiastic about coverage analysis and the level of confidence it gives about the goodness of code. However, what I don't think is arguable is that they provide a nice way to figure out what is fairly tested and what needs deeper inspection. As far as I know, FFI can't give you that;
  • unit tests in C would allow for a much deeper introspection of data structures and would forcibly be much more integrated with C types system (think of pointers);
  • unit tests in C would free us from having to maintain code to generate FFI code (...) and data structures;
@guyzmo

This comment has been minimized.

Contributor

guyzmo commented Feb 20, 2017

• unit tests are often the best documentation available on a particular function or data structure. In contrast to comments-based documentation (read: doxygen)- which we all know tends to rot away - unit tests are forced to be kept in sync with the functionality the test when this evolves. Having them in a different language prevents them from offering concrete examples on how to use or not use a particular API;
• having unit tests in C would allow us to at least have a discussion about where the unit tests code should go, either in separate files or near the code they test. Using a foreign language would preclude even the possibility to have such a discussion;

I strongly disagree about the two above. If tests are the best source of documentation for your code, and not the code itself (and the few comments to help understanding of the function), then the project has real design and usabality issues. Furthermore, tests are not designed to be showing HOW the tested function works, but cover not only the general usecase, but all the edge cases. That would mean a lot of redoing the same test with very little differences.

Which is why I'm strongly opposed to keep code and testing of the code in the same compilation unit. Because then, you'd 90% of the source file only being about testing the 10% that are drown within the tests.

N.B.: to be clear, I'm not opposed to the discussion, I'm strongly opposed to keep unit tests within the same compilation unit as the code being tested. For the reasons exposed above.

• unit tests in C would allow for coverage analysis. I understand not everybody is enthusiastic about coverage analysis and the level of confidence it gives about the goodness of code. However, what I don't think is arguable is that they provide a nice way to figure out what is fairly tested and what needs deeper inspection. As far as I know, FFI can't give you that;

Can't it? We can just link against gcov and it will provide a coverage report. That works with luajit+ffi. And that's how neovim does it.

• unit tests in C would allow for a much deeper introspection of data structures and would forcibly be much more integrated with C types system (think of pointers);

You can work with pointers in luajit+ffi, but that's right that you need some things are a bit more difficult, and for those you can introduce C helper functions. Then you'd argue it's adding a layer of abstraction, and you'd be right. But what would really be the harm?

• unit tests in C would free us from having to maintain code to generate FFI code (...) and data structures;

that code can eventually be put as an external library that would be shared by projects like neovim and ours, pulled from the luarocks package repository.

@guyzmo

This comment has been minimized.

Contributor

guyzmo commented Feb 20, 2017

so you asked what's good about lua test harness?

  1. consistency: to build the functional test harness, taking advantage of the embedded lua interpreter in mutt, I've added lua functional tests. Making unit tests in lua as well makes sense, so we have consistency in:
    • how test specs are written and,
    • how the reporting is done.
  2. accessibility: the busted test framework offers a BDD driven approach, that offers a more tangible understanding of what the test harness offers. That forces the feature implementer to be in a usage drives implementation state of mind when writing the test specs.
  3. integration: lua+busted is compatible with TAP which provides a standard API for testing that integrates well with external frameworks.
  4. portability: lua runs about as many architectures and compilers as mutt, the language being about as old as mutt (2 years older actually).
  5. abstraction: It's great to abstract from the implementation details to actually be able to focus and automize testing of an algorithm's behaviour taking advantage of the greater flexibility and stronger typing Lua offers.

The bad sides?

  1. complexity: to run the tests we're introducing luajit, the need to use luajit, busted that pulls a bunch of external frameworks, which complexify somehow the work of the build scripts maintainer ;
  2. abstraction: It's also a weak point, because it's adding layers between testing and tested, which can occasionally give more work to the contributor, when testing dark corners of the implemention (the lua header parser does not support everything yet).

What really led me to this choice are the two first good points listed above, as I believe consistency and accessibility are two strong feat of this solution. Then, I also have to admit that implementing this test harness has been driven by neovim's experience over the last 3 years, as they followed a similar path with another codebase.

For all the parts that are adding complexity there's the possibility to factorise the work, and take advantage of having a similar effort on both projects to make that extra complexity an external lua dependency (on luarocks).

N.B.: Of course, and I repeat, this is a proposal that I've implemented and I'm ok with withdrawing the unit tests of that PR, if we come up with a better solution.

@guyzmo

This comment has been minimized.

Contributor

guyzmo commented Feb 20, 2017

N.B.: Of course, and I repeat, this is a proposal that I've implemented and I'm ok with withdrawing the unit tests of that PR, if we come up with a better solution.

to prove I'm not narrow minded, I stumbled across that small test framework:

https://github.com/grassator/bdd-for-c

that ticks a few checks:

  • BDD oriented
  • TAP support
  • Smaller and simpler

might be worth trying it out.

@toogley

This comment has been minimized.

Contributor

toogley commented Feb 23, 2017

@gahr

having unit tests in C would allow us to at least have a discussion about where the unit tests code should go, either in separate files or near the code they test. Using a foreign language would preclude even the possibility to have such a discussion;

would keeping unit tests inside the source code have any advantage?

@guyzmo

This comment has been minimized.

Contributor

guyzmo commented Feb 25, 2017

I have built 3 branches, following that comment:

@gahr offers an alternative solution #428.

Now we need to think what would be the best solution.

@flatcap

This comment has been minimized.

Member

flatcap commented Feb 21, 2018

@flatcap flatcap closed this Feb 21, 2018

@wafflebot wafflebot bot removed the status:in-progress label Feb 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment