Group freelist contention

[eugeneia]

Currently, the group freelist—which we use to allocated rebalance packet buffers between Snabb process within a group—is not able to sustain unidirectional, high-throughput use cases.

I.e. this is fine because the group freelist is not used much:

procA <--pair of interlinks --> procB (bidirectional traffic)

while this use-case is a problem:

procA --single interlink--> procB (procA needs to reclaim packets from procB)

The issue manifest as overly long pauses while the group freelist spinlock is held.

A workaround is currently to manually feed back packets from procB to procA and freeing them in procA (via a dedicated interlink).

It would be neat to solve this automatically however. Some possible approaches include:

• reducing pause times by setting small upper limits on the number of packets rebalanced during rebalance_freelists and allocate
• using a better lock-free data structure for group freelist
• eliminating a global group freelist altogether and doing rebalancing between processes directly

Cc @alexandergall