-
Notifications
You must be signed in to change notification settings - Fork 896
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added module to enable CPU affinity #1641
Conversation
cpu-affinity/src/main/java/org/apache/bookkeeper/utils/affinity/CpuAffinity.java
Outdated
Show resolved
Hide resolved
Ping. Please take a look again. |
3a3056a
to
5119c88
Compare
5119c88
to
615f94e
Compare
circe-checksum/src/main/java/com/scurrilous/circe/utils/NativeUtils.java
Show resolved
Hide resolved
cpu-affinity/src/main/java/org/apache/bookkeeper/common/util/affinity/impl/CpuAffinityImpl.java
Show resolved
Hide resolved
cpu-affinity/src/main/java/org/apache/bookkeeper/common/util/affinity/impl/CpuAffinityImpl.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The patch itself looks good. A few nits.
However, I think we need an overarching document for these "performance" improvements, with benchmark results, detailed instructions to show how to reproduce the scenario in which performance changes, and the overall goals and approach for the perf improvements.
The current patches feel very piecemeal and handwavey, and this becomes a problem later when someone wants to change something, and the change is objected to on performance grounds but no solid facts on why it can't change.
circe-checksum/src/main/java/com/scurrilous/circe/utils/NativeUtils.java
Show resolved
Hide resolved
</profile> | ||
|
||
<profile> | ||
<id>Linux</id> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is the profile capitalized, while the other isn't
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This pom was copied from the existing circe-checksum one. The profile name is not that important since it's automatically enabled based on the current OS.
cpu-affinity/src/main/java/org/apache/bookkeeper/common/util/affinity/impl/NativeUtils.java
Show resolved
Hide resolved
@ivankelly @eolivelli @sijie Please take another look.
This is just the underlying implementation. I will work on the docs and instructions when the whole busy-wait change set is ready. |
Do you have some preliminary numbers then to show why this change is valuable? |
This are the numbers on my local tests. It's not using the BK client lib directly, but rather through ManagedLedger (from Pulsar) which has one more thread handoff, to have callbacks serialized to 1 thread hashed on topic name. Conditions:
The remaining bulk of the median latency I think is related to SO_BUSY_POLL (netty/netty#8268) being ineffective on loopback interface. |
run pr validation |
@merlimat thanks. Do you have any numbers for just this change? Like without the spin-wait stuff and HT disablement? |
@ivankelly All the changes in the table above are incremental, so the last 2 lines show the effect of Cpu isolation:
I think it doesn't make sense to test cpu isolation without the other changes because the baseline noise will make the effects not visible. To summarize: cpu-affinity is used to remove interference from other tasks (OS thread and other processes) from a particular thread in order to avoid context switches and reduce the latency in the long tail. Without busy-poll, the bulk of context switches will still be there (eg: when sleeping while waiting for items in a queue), so the cpu-isolation won't provide much benefits. |
Motivation
This is part of a set of changes aimed at reducing latency in BK at the expense of other aspects (eg: max throughput). While not intended to be used as default settings, they might be good to have whenever the latency becomes critical.
Pinning a thread to a particular CPU will ensure no other process will execute on that CPU reducing all scheduler induced context switches that will cause latency jittery.
A given thread that wants to get pinned to a CPU just needs to call:
It's called
acquireCore()
because it will also disable hyper-threading on the pinned cpu.Subsequent PRs will use this module to have the option to pin critical threads to available CPUs.
Changes
sched_setaffinity()
to pin a thread to a particular CPU