-
Notifications
You must be signed in to change notification settings - Fork 0
Network ring buffers
To scale your custom Go system or your MySQL environment to handle extreme network throughput (e.g., 100,000+ packets/operations per second), the application-level ring buffer is only half the battle. If the Linux operating system kernel or the network interface card (NIC) drops packets before they even reach your code, application optimizations won't matter. When a network packet arrives, it is placed into a hardware Network Ring Buffer (RX Ring) on the NIC, processed via an interrupt, and pushed to the Kernel Socket Receive Buffer before your Go or MySQL process can read it. Here is how to configure and tune these layers for high-throughput systems.
Your network interface card has an on-board hardware ring buffer. If this buffer fills up, the network card will drop packets at the wire level (visible as rx_fifo_errors or rx_dropped in ifconfig). First, inspect your current ring buffer capacity and maximum limits:
ethtool -g eth0
If the Current Hardware Settings are lower than the Pre-set Maximums, max them out (commonly up to 4096 on enterprise cards):
sudo ethtool -G eth0 rx 4096 tx 4096
(Replace eth0 with your active network interface identifier).
In Go, reading directly from a raw TCP network socket inside a loop creates excessive syscall and scheduling overhead. Instead, bypass standard blocking reads by implementing an explicit network ring buffer pattern using bufio.Reader wrapped around a channel queue. This production pattern decouples the network connection reader thread from the application processing logic:
- Avoid Garbage Collection (GC) Overhead: As shown in the Go example above, never allocate new memory (make([]byte)) inside a hot loop tracking incoming network traffic. Use sync.Pool to recycle byte arrays. Under extreme throughput, GC cycles are often what causes applications to lag behind network queues.
- Pin CPU IRQs (Interrupt Requests): For multi-core bare-metal servers, network performance degrades if packet processing bounces randomly across CPU cores. Use a tool like irqbalance or manually bind specific network queues to dedicated CPU cores via /proc/irq/ affinity masks to keep your memory caches hot.
When deploying high-throughput, standard TCP systems on the Google Cloud Platform (GCP), optimizations differ significantly from bare-metal environments.
Virtual machines (Compute Engine) run on Andromeda, Google’s software-defined network virtualized stack. Because of how GCP handles tracking connections, virtualization overhead, and resource queuing, specific infrastructure and network settings must be implemented to achieve optimal throughput.
Before modifying code or kernel profiles, your Compute Engine instance configurations must support high-speed network traffic.
- Enable Google Virtual NIC (gVNIC): Do not use the legacy virtio network driver. When creating or modifying your Compute Engine templates, select gVNIC. It dramatically lowers host CPU overhead and is required for high-bandwidth networking tiers. [2, 6]
- Enable Tier 1 Networking Performance: By default, egress bandwidth scales with vCPU count (~2 Gbps per vCPU). For maximum performance, use memory-optimized or compute-optimized machine types (like N2, C2, or C3) and explicitly enable Tier 1 Performance to uncap bandwidth limits up to 50 Gbps–100 Gbps.
- Set Jumbo Frames (MTU 8896): In a private VPC where your Go custom application communicates directly with your MySQL node, do not stick to the standard Internet MTU of 1500. Change the VPC network MTU to 8896 (Jumbo Frames). Larger frames mean fewer packets to unpack, drastically cutting CPU interruption overhead.
Google Cloud enforces a hard, 10-minute idle timeout connection tracking limit via its underlying Andromeda virtualized firewall. If a TCP stream goes quiet for 10 minutes without pushing data, GCP silently stops passing inbound packets for that connection. Your application will hang indefinitely thinking the socket is still open. To circumvent this, force the Linux kernel to send keep-alive probes long before GCP drops the connection: Add this custom profile configuration to /etc/sysctl.conf and activate via
sudo sysctl -p:
To make sure your custom Go network layer respects your underlying GCP kernel adjustments, you must explicitly enable system keep-alives and disable Nagle’s Algorithm (TCP_NODELAY). Nagle's algorithm buffers tiny chunks of outbound data to make packets larger, which introduces latency spikes in high-throughput transactional apps. Ensure your Go connection listeners explicitly modify incoming TCP connection settings:
If you are planning the production deployment, let me know:
- For example my clients be connecting from inside the same GCP VPC (like internal microservices).
- I putting a Google Cloud Load Balancer (GCP L4 NetLB) in front of these instances.
Next is to adjust the configuration to account for proxy headers or routing topologies.
Since the traffic is entirely inside the same GCP VPC and passes through a Google Cloud Load Balancer, your configuration needs a highly specialized setup. For internal Layer 4 traffic, you should use GCP's Internal Passthrough Network Load Balancer (ILB). Unlike HTTP(S) proxy load balancers, the Passthrough ILB is not a proxy. It routes connections directly to your VM instances at the routing layer, preserving the original client IP and port without terminating the TCP session. This architectural setup requires specific network adjustments to handle maximum throughput without dropping packets.
Because the Passthrough ILB delivers the packet straight to your Virtual Machine with the Load Balancer's Virtual IP (VIP) as the destination address, your VM's Linux kernel must be explicitly prepared to accept this traffic.
- Configure Symmetric Routing: GCP automatically sets up routing rules for metadata and network checks. However, when traffic volume spikes, the kernel might flag incoming ILB packets as asymmetric and drop them.
- Enable IP Forwarding: When launching your Compute Engine instances, you must explicitly toggle canIpForward = true (Enable IP Forwarding) in the network interface configuration. This permits the VM to process packets that aren't strictly addressed to its internal primary private IP.
Since your clients are other internal microservices inside the same VPC network, you do not need to worry about internet congestion or packet loss over the public web. You can safely maximize buffer queues and optimize for high-speed local VPC peering. Add these custom optimizations to /etc/sysctl.conf on your Go and MySQL application servers, then apply them using
sudo sysctl -p:
GCP's Passthrough Load Balancer determines whether to send traffic to your instance based on a GCP Health Check. If your custom Go application freezes or blocks during a massive write spike, the health check will fail, and the load balancer will stop sending traffic to that node. To prevent this, you must isolate your health check server from your high-throughput data processing loop using a dedicated port.
- VPC Firewall Rules: Ensure you have an ingress firewall rule explicitly allowing the GCP Health Check IP ranges (35.191.0.0/16 and 130.211.0.0/22) to access your dedicated health check port (8080).
- Keep-Alives: Because it's an internal passthrough load balancer, the 10-minute idle connection timeout is enforced by the VPC network itself. The Go code and kernel settings we configured in the previous step will successfully keep these internal connections alive.
- connections open permanently to keep long-lived persistent TCP
- connections periodically frequently connect, send data, and disconnect