Introduction of CPU thread affinity

[pveentjer]

Provides CPU thread affinity.

So certain threads can have affinity for particular cpu's.

Server

-Dhazelcast.io.input.thread.affinity=1-3 
-Dhazelcast.io.output.thread.affinity=3-5 
-Dhazelcast.operation.thread.affinity=7-10,13 
-Dhazelcast.operation.response.thread.affinity=15,16

Client

-Dhazelcast.client.io.input.thread.affinity=1-4 
-Dhazelcast.client.io.output.thread.affinity=5-8
-Dhazelcast.client.response.thread.affinity=7-9

If you don't configure affinity for a category of threads, it means they can run on any CPU.

This gives a lot better control on latency and provides better throughput. The number of threads configured in the affinity settings overrides the number of threads that have been configured explicitly.

It matters a lot on which NUMA node the IO threads are. It can make performance very bad, but also very good. More research is needed.

For the time being this will not be a public feature. We can use it internally to exclude certain problems like threads moving over CPUS and control NUMA locality of memory.

When using the affinity syntax it is important to realize that it also includes the number of threads. So it will override the corresponding thread count.

The affinity syntax is as follows:

• 1,2,3,4: We get 1 thread on CPU1, one thread on CPU2 etc. In total there are 4 threads configured which will each run on a different CPU.
• 1-4: this is a shortcut for the above syntax.
• [1-4]: this is a group. This will give 4 threads and each thread can run on CPU's 1-4.
• [1-4]:2: this is also group but with 2 threads that can run over CPUS 1-4.

You can also make combinations
1,3,5-8,[10-20]:2,[31,41]:5

It is important to realize that 3-4 is something else than [3-4]. The first configuration will give you 1 thread assigned to CPU 3 and another to CPU 4. The second configuration will give you 2 threads that both can run on CPUS 3 and 4.

If you have the following CPU NUMA node mapping:

• CPUS 0-19 on NUMA node 0
• CPUS 20-39on NUMA node 1
  You could configure 20 threads on NUMA node 0 and 20 threads on NUMA node 2 like this:
  [0-19],[20-39].

If you want to see the mapping, run
numactl --hardware

Example from a dual socket machine:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
node 0 size: 393090 MB
node 0 free: 372729 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
node 1 size: 393216 MB
node 1 free: 343296 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

If you want to confine the partition threads to the NUMA nodes for the above configuration, you would use:

-Dhazelcast.operation.thread.affinity=[0-9,20-29],[10-19,30-39]

The affinity properties can only be set from the command line. E.g.

 -Dhazelcast.client.io.input.thread.affinity=1,2,3

A few warnings:

• Pinning a single thread down to a single CPU is great for experiments. But can be a problem in a production setup because the thread has no other CPU it can run on. So if another thread or the OS would be running on that CPU (there are many more threads in HZ and in the JVM), this thread will get stalled. So in most cases it will be better to use the group syntax i.e. [from-to]:threadcount
• Be careful pinning a thread down to CPU 0 especially if that is the only thread it can run on. In some environment this CPU is used for other purposes.
• Since this is not a public feature, breaking changes can be made without warning.
• Mileage may vary! Thread affinity is not a guarantee for reduced latency or increased throughput.

You need to following jars on your classpath:

  <!-- needed for thread affinity -->
        <dependency>
            <groupId>net.openhft</groupId>
            <artifactId>affinity</artifactId>
            <version>3.2.2</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.25</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.5.8</version>
        </dependency>

If you are using the Simulator, you don't need to do anything extra since Simulator has these on the classpath.

In the System.out you will find information if the affinity has been picked up correctly. E.g.

hz.heuristic_mccarthy.partition-operation.thread-0 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-1 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-2 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-4 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-3 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-6 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-5 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-7 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-8 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-9 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-11 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-12 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-10 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-13 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-14 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-15 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-17 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-16 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-18 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-19 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-24 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-27 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-25 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-30 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-21 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-38 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-39 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-22 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-31 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-28 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-20 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-26 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-23 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-37 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-36 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-34 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-29 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-32 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-33 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.partition-operation.thread-35 has affinity for cpus:{20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59}
hz.heuristic_mccarthy.IO.thread-in-0 has affinity for cpus:{1, 2, 3, 4}
hz.heuristic_mccarthy.IO.thread-in-1 has affinity for cpus:{1, 2, 3, 4}
hz.heuristic_mccarthy.IO.thread-in-2 has affinity for cpus:{1, 2, 3, 4}
hz.heuristic_mccarthy.IO.thread-out-0 has affinity for cpus:{5, 6, 7, 8}
hz.heuristic_mccarthy.IO.thread-in-3 has affinity for cpus:{1, 2, 3, 4}
hz.heuristic_mccarthy.IO.thread-out-1 has affinity for cpus:{5, 6, 7, 8}
hz.heuristic_mccarthy.IO.thread-out-3 has affinity for cpus:{5, 6, 7, 8}
hz.heuristic_mccarthy.IO.thread-out-2 has affinity for cpus:{5, 6, 7, 8}

[ghost]

The job Hazelcast-pr-builder of your PR failed (log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file

--------------------------
-------TEST FAILURE-------
--------------------------
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ClientMapEventJournalBasicTest>AbstractEventJournalBasicTest.allowReadingWithFutureSeq:512->HazelcastTestSupport.assertOpenEventually:1108->HazelcastTestSupport.assertOpenEventually:1123 CountDownLatch failed to complete within 30 seconds, count left: 1
[INFO] 
[ERROR] Tests run: 27743, Failures: 1, Errors: 0, Skipped: 1001
[INFO] 
[ERROR] There are test failures.

[blazember]

I approve this as an experimental feature. For a product feature, I propose the changes I commented in the pom.xml (we discussed it in private too). Good to see this coming to make drawing conclusions from our perf tests easier 🎉

I made some comments on minor things (mostly logging).

[blazember]

Would make sense to add a test for negative numbers too. Both for groups and thread count.

[pveentjer]

Negative integers are not parsed by the parser.

[blazember]

Shouldn't we get an InvalidAffinitySyntaxException for such a setting?

[pveentjer]

You will. Check the syntax for an integer. it will expect 1 digit followed by zero or more digits. If it sees a '-', it won't recognize it and you get a syntax error.