Consider setting CPU affinity for fdbserver processes

[alexmiller-apple]

From https://forums.foundationdb.org/t/setting-cpu-affinity-for-fdbservers/84, it was noted that we don't set CPU affinity for any of the processes we start, and there's likely advantages to doing so. It appears once upon a time, we had code in the resolver that would pin itself to a core, but it was commented out, and I agree doing it from fdbmonitor sounds like a better solution.

Implementing this doesn't sound like too much work, and a quick run through a performance test should tell us if it makes any noticeable impact.

[cespare]

I'd guess the difference will be small on an otherwise-unloaded machine unless you manage to make an fdbserver allocate some memory on the wrong NUMA domain. That should definitely cause noticeable overhead.

[davidscherer]

The performance effects of this sort of tuning tend to be fairly hardware and OS version specific, so I would recommend diverse testing before making this the default behavior out of the box. But an fdbserver parameter (and thus, config file option) to set affinity might be low risk and convenient.

Ideally operating systems should be able to do a reasonable job placing essentially single-threaded programs, even on NUMA machines. But I think setting affinity is a legitimate solution to a particular situation with network card interrupt delivery. It rings a bell for me. There might have been something written on this subject at some point in FDB history, if the Apple team wants to try to dig it up.

[daverosenthal]

We messed with pinning the network queues to cores, etc. It only made minor differences with highly unusual test workloads. Never a practical improvement. I agree with @davidscherer 's suggestion of adding an option for CPU affinity.

[panghy]

This turns out to be rather significant for us since we use the memory storage engine. Kernel-level automatic NUMA balancing "kernel.numa_balancing" doesn't help us (enough), basically we discovered that hot spots (certain processes would be much hotter CPU-wise than others) are largely the result of the kernel scheduling the process to a node and having memory allocation local to that node and then subsequently moving the process to another node (the latency impacts are significant in that case and difficult to debug).

[fdbserver.4500]
command = /usr/bin/numactl -m 0 -N 0 /usr/sbin/fdbserver

We are directly managing core and memory affinity with numactl hence.

[xumengpanda]

The numactl command can also handle the set-cpu-affinity work.
Unless we want to start a (fdbserver) process on a particular core, it seems using the linux tools to set the affinity is enough.