Background
I am testing a high performance, low latency benchmark between a c5n.4xlarge client and a c5n.xlarge server. For most of my testing I just put them both in a cluster group, and that was good enough, but for the final round of tests I wanted to guarantee that the latency between hosts doesn't change much (even across stop/starts), and I wanted to avoid any potential noisy neighbor issues. To achieve this I decided to launch both instances on the same dedicated host.
For some reason the console prevents me from launching a c5n.4xlarge and c5n.xlarge on the same dedicated host. In order to work around the issue I decided to just go ahead and launch a c5n.9xlarge for the server, but then use the EC2 CPU options configuration to restrict it to only use 4 vCPUs (2 cores, 2 threads per core). This seemed to work fine at a first glance, only 4 vCPUs were reported by the OS, only 4 network queues were reported by ethtool, and stress-ng tests showed performance to be consistent with a standalone c5n.xlarge (or just a little faster).
Issue
When I run my benchmark between the two instances on the same dedicated host, overall performance is 20% lower than expected, and p99 latency is almost twice as much as it should be.
On further investigation I realized that in this configuration (9xlarge pretending to be xlarge), hardware interrupts were no longer being distributed evenly across the 4 CPUs. Even though the OS is reporting 4 vCPUs and 4 network queues, the actual interrupts were not being distributed evenly. Additionally the distribution of the interrupts was changing while the benchmark was running even though a consistent set of connections is used for the entire benchmark.
At first I thought the issue was related to the fact that I was using EC2 CPU Options to restrict the number of CPUs, but further investigation confirms this is not the case. Instead the fact that I am running the benchmark between two instances on the same dedicated host seems to be the source of the decline in performance.
Background
I am testing a high performance, low latency benchmark between a c5n.4xlarge client and a c5n.xlarge server. For most of my testing I just put them both in a cluster group, and that was good enough, but for the final round of tests I wanted to guarantee that the latency between hosts doesn't change much (even across stop/starts), and I wanted to avoid any potential noisy neighbor issues. To achieve this I decided to launch both instances on the same dedicated host.
For some reason the console prevents me from launching a c5n.4xlarge and c5n.xlarge on the same dedicated host. In order to work around the issue I decided to just go ahead and launch a c5n.9xlarge for the server, but then use the EC2 CPU options configuration to restrict it to only use 4 vCPUs (2 cores, 2 threads per core). This seemed to work fine at a first glance, only 4 vCPUs were reported by the OS, only 4 network queues were reported by ethtool, and stress-ng tests showed performance to be consistent with a standalone c5n.xlarge (or just a little faster).
Issue
When I run my benchmark between the two instances on the same dedicated host, overall performance is 20% lower than expected, and p99 latency is almost twice as much as it should be.
On further investigation I realized that in this configuration (9xlarge pretending to be xlarge), hardware interrupts were no longer being distributed evenly across the 4 CPUs. Even though the OS is reporting 4 vCPUs and 4 network queues, the actual interrupts were not being distributed evenly. Additionally the distribution of the interrupts was changing while the benchmark was running even though a consistent set of connections is used for the entire benchmark.At first I thought the issue was related to the fact that I was using EC2 CPU Options to restrict the number of CPUs, but further investigation confirms this is not the case. Instead the fact that I am running the benchmark between two instances on the same dedicated host seems to be the source of the decline in performance.