Rework NetmaskTree for better CPU and memory efficiency.

[stephanbosch]

Short description

The existing implementation of class NetmaskTree was inefficient because:

• it created a new tree node for each bit in the Netmask
• used a redundant std::set to provide iterator support.

These changes mend both these problems, yielding an overall decrease in CPU usage and memory requirements.

Checklist

I have:

• read the CONTRIBUTING.md document
• compiled this code
• tested this code
• included documentation (including possible behaviour changes)
• documented the code
• added or modified regression test(s)
• added or modified unit test(s)

[stephanbosch]

oh, seems like I didn't actually compile the recursor. need to fix a few things.

[lgtm-com]

This pull request introduces 1 alert when merging 7e133b3 into ef9ed3b - view on LGTM.com

new alerts:

• 1 for Inconsistent definition of copy constructor and assignment ('Rule of Two')

[stephanbosch]

For some reason, GitHub doesn't list the commits in the proper sequence. Not practical for review.

Proper order from last to first commit:

3d12b855d iputils.hh: NetmaskTree: Drop the internal std::set.
7061cda48 iputils.hh: NetmaskTree: Add iterator class and use it for begin() and end() methods.
e1c261ca4 iputils.hh: NetmaskTree: Keep track of the left-most node in the tree.
94085eed0 iputils.hh: NetmaskTree: Copy the tree using tree traversal.
23992be69 iputils.hh: NetmaskTree::TreeNode: Implement tree traversal methods.
85fe4f504 iputils.hh: NetmaskTree: Make tree cleanup mandatory.
462366721 iputils.hh: NetmaskTree: Reduce the number of tree nodes.
a503e85a5 iputils.hh: NetmaskTree: Initialize TreeNode::node immediately.
ed2e08993 iputils.hh: NetmaskTree: Use for loops instead of while loops.
e8783ead0 iputils.hh: NetmaskTree: Restructure the tree with separate branches for for IPv4 and IPv6
51ab542b4 iputils.hh: NetmaskTree: Rename field "root" to "d_root"
39599b68b iputils.hh: NetmaskTree: Use ComboAddress::getBit() and Netmask::getBit()
28319d29c iputils.hh: Netmask: Add masked copy constructor.
422b693b7 iputils.hh: Netmask: Prevent the construction of a 128-bit IPv4 netmask.
25d2498a5 iputils.hh: Netmask: Add getBit()
5d3168541 iputils.hh: Netmask: Add getAddressBits()
ab4073c00 iputils.hh: ComboAddress: Add getBit()
af340ccc9 iputils.hh: ComboAddress: Add getBits()
61a508dc7 test-iputils_hh.cc: Add tests for NetmaskTree copy, swap and iterator operations.
0313b0e1a test-iputils_hh.cc: Test 0.0.0.0 address at serveral network bit ranges.
9c405db9a test-iputils_hh.cc: Better verify the NetmaskTree container size during tests.
90b0422cc Reformat test-iputils_hh.cc.
ca717c62f Reformat iputils.hh

[stephanbosch]

Fixing the LGTM alert.

[stephanbosch]

OK, worked around the GitHub commit order bug by rebasing and rewording every commit.

[rgacogne]

Looks great, a few comments but nothing bad.
A very quick test shows a very nice decrease in memory usage and insertion/lookup time, with a very small increase of the deletion time (even when compared to the existing "cleaning up" case) that probably doesn't matter to us.
I'll do more tests later.

[stephanbosch]

OK, I'll address these soonish. I also have a few other small things I'd like to change still.

[stephanbosch]

I addressed most comments and changed a few things:

• The original pr added a Netmask(other, bits) copy constructor that could used to make a copy with equal or less network bits. That method is called getSuper() now.
• Netmasks are now stored normalized in the tree (bits outside the netmask are forced to zero). So, the pair->first values returned by the insert() and lookup() are also always normalized.
• I added a new commit that makes the iterator API use a proper reference type (not a pointer). We can do this now that there is no std::set anymore that dictates the iterator type. This makes the iterator more standard, less cumbersome and particularly less confusing. This commit removes a question comment that is illustrative of the problem this solves.
• Made the key (first) value in the node_type const, so that the application cannot mess with it. That could have corrupted the tree.

[rgacogne]

Thanks! Just a nit but amending the commits makes it hard for me to spot what actually changed since my initial review :-)

[stephanbosch]

Thanks! Just a nit but amending the commits makes it hard for me to spot what actually changed since my initial review :-)

Yeah, that is a bit easier in GitLab it seems...

[Habbie]

This branch has conflicts that must be resolved

[omoerbeek]

Loving this PR and would even love it more if it got finished :-) @stephanbosch do you think you'll have some time to spend to make it ready for merge?

[stephanbosch]

Rebased and CI seems to like it still.

[rgacogne]

Looks great, many thanks! Just one nit but we can fix it later.