unit testing

[ktsaou]

@philwhineray

I suggest to have a set of configurations which will be used for testing all tools prior to release.

For example,

1. we keep a folder of firehol, fireqos and link-balancer config files.
2. we write a script which calls each tool to generate the final statements for all config files
3. this script compares the newly generated statements to the statements that were generated using the previous release (or the previous commit).
4. it reports any differences it finds in file that is also committed to git (so that we will have a log of the differences for every commit)

I hope that this way we will be able to find corner cases in our code quickly and easily.

What do you think?

[philwhineray]

I think it is a very good idea, in fact for firehol I run such a setup regularly from a local branch.

I have rebased and pushed the branch regression so you can see what is there.

The regress command allows running any version of firehol from any local branch, tag or commit, so rather than storing to git I tend to run several runs and use recursive diff the output directories. Then I have a mechanism to save expected output where I have done a manual audit and a way to do testing of statuses when a failure is expected.

It may take some effort to bring it in line with exactly what you have in mind but maybe it will do as a start.

[philwhineray]

The most recent changes to mark will have broken the audited outputs of course. That is the price of progress :)

[philwhineray]

With your unit testing suggestion, quite a lot of the code could be removed i.e. there is quite a lot to do with replicating results between versions which support ipv6 or not and other features: if the tests only have to relate to the current version it can be much simpler.

I should add that if you want to start over I won't be offended. I threw this code together mainly to prove that I was getting the same results between ipv4 and ipv6 and that I had not broken anything compared to the ipv4-only version.

[philwhineray]

Finally for now... your point 4 seems over complex - why not just commit the updated results with a comment about why the change is acceptable and then git will maintain the history of the differences and reasons for them.

[ktsaou]

@philwhineray ok.

I don't expect to have time for this in the near future. I just wrote down the idea to follow it up later.
I can help if you have some time to spend on this, but I cannot take it on my own.

First, I plan to work a bit on link-balancer, possibly adding static routes configuration to it per interface and gateway.

Later, I plan to spend some time on netdata. I hope to modularize it a bit, cleanup the code, support a new charting mechanism, allow it to handle data for longer durations in an effective way and even provide an events interface to it. Once this is done, I hope you will accept it for inclusion in firehol too.

Also, I would like to work a bit on the documentation and the site of firehol. Even change the look and feel of it (for example use bootstrap as a css engine).

The above will most probably take a few months...

[ktsaou]

I hope however we can release a version of firehol in the mean time. What do you think?

[philwhineray]

@ktsaou

No problem, I did not expect an immediate action - I just did not want you to put in extra effort if there are bits that can be re-used rather than keeping it private on my system.

I may not have much time shortly - we have a new member of the family due soon. If I do have time to extend the system / make it more like unit tests I will do so. More likely I will continue to run it as is before releases and we can revisit later.

I would also like to see a firehol release soon since I think a number of quite big improvements have gone in recently. For me a priority is working out firewalling for a mixed bridge / router first - there are still problems with that.

So once that is done, I would suggest we call it 2.1.0, including link-balancer (I have not even looked at this yet). Including netdata should not be a problem either, once you are happy with it. We just need to work out where it lives in the tree.

Finally for the documentation and the website: if I get some time I will look at combining nanoc and bootstrap. If not, the basic framework is pretty easy to work with when you have the tools - I think the important info is in README.md. In any case if you do any experiments in the test branch the pages will automatically publish to test.firehol.org so it is safe to try things out and make mistakes without affecting the main site.