tradeoff between memory copy and cpu contention

[exceptionplayer]

Recently we found an issue: we used a shared buffer for all the eventloops, then all the eventloop threads tried to release the buffer which caused high latency.

We fixed this issue by replacing the buffer with an unreleasableBuffer. Although the issue is resolved for now, it makes us thinking about the balance between avoiding memory copy by sharing buffer and avoiding cpu contention.

We need to send same data to different users, usually we use shared buffers, but recently we upgraded our server machines to a new generation that has more cpu cores, the cpu contention issue occurred.

Do you have any ideas?

[franz1981]

Do you have any profiling data?
What type of cpu is?
The contention on the shared ref counter updates across sockets can be pretty bad - and there is little to do there; our ref counter is not designed to be scalable across cores unless the work performed is enough to not making it a bottleneck

[exceptionplayer]

@franz1981 Hi, we are using Intel(R) Xeon(R) 6892P-C, there is 96 cores, and 3 NUMA nodes.
The flamegraph showed that all the eventloop threads are busy invoking ReferenceCountUpdater.tryRelease0 method.

[franz1981]

Yep. As expected...in your case it would be better performing a copy and getting a big cache miss while reading it, instead. It will he slower but scalable