CPU governor usage does not consider SMT (aka hyperthreading)

[jimharris]

With SMT enabled, the SPDK schedulers can get confused when using the CPU frequency governor, because it will often try to change the frequency on one CPU thread without changing its sibling.

With the dynamic scheduler, the following occurs when the system is idle:

1. all spdk_threads are moved to the scheduling reactor/core
2. all other cores go to sleep
3. scheduler detects that even the scheduling reactor is idle, and will continuously decrease its frequency

On my AMD platform, spdk_top shows that the scheduler thinks it has lowered the CPU frequency all the way to 400. But in reality since its sibling thread's frequency hasn't been changed, the scheduling core's frequency stays at maximum.

Similarly for gscheduler, two reactors have differing busy percentages, they will be setting a frequency based on their own busy percentage, but that frequency will not take effect unless the sibling sets the same frequency.

I think we need the following at a minimum to fix this bug:

1. Some kind of SMT detection code, probably with APIs in env.h. Note that DPDK to my knowledge has no APIs for determine SMT siblings. So we can decide whether to try to upstream sibling APIs into DPDK, or just implement them ourselves in our env_dpdk library. But they certainly belong in env.h, we already have other CPU-related APIs there.

2. We should fail to init the governor if we detect the application includes any SMT sibling threads.

The dynamic scheduler already properly handles case where the governor is not available. It will continue to move spdk_threads, but just won't try to adjust CPU frequency when the governor isn't available.

The gscheduler scheduler already fails init() if the governor cannot be installed.

I would consider adding SMT support to the schedulers as an enhancement, which would look different for each scheduler and is outside the scope of this issue.

[mikeBashStuff]

Just to note, for the very same reason - to force drop in the cpu(s) frequency in tests (so basically workaround DPDK's limitations in regards to smt) - this was done: 7832b34. Of course, the intention here was to simply put an emphasis on the actual limitation not to "fix" anything since the test now simply emulates what an actual user would have to do to see drop in the frequency. This can be seen in action across most of the nvme-phy-autotest jobs (under dpdk_governor test) since most of our phy systems have smt enabled.

[jimharris]

Just to note, for the very same reason - to force drop in the cpu(s) frequency in tests (so basically workaround DPDK's limitations in regards to smt) - this was done: 7832b34. Of course, the intention here was to simply put an emphasis on the actual limitation not to "fix" anything since the test now simply emulates what an actual user would have to do to see drop in the frequency. This can be seen in action across most of the nvme-phy-autotest jobs (under dpdk_governor test) since most of our phy systems have smt enabled.

Thanks for the heads-up @mikeBashStuff.

I have a patch ready that does the SMT detection. Just need to write patch 2 which will be trivial.

But we can't break the tests in the meantime, maybe step 2 checks for an undocumented environment variable, that allows the governor to load even in the presence of SMT? And then I can set that environment variable in the those scheduler test scripts.

[jimharris]

Pushed https://review.spdk.io/gerrit/c/spdk/spdk/+/23407 out to CI. Let's see if the scheduler tests still pass.

[tomzawadzki]

The series above were merged, the issue can be closed.