Scaling Eureka Horizontally

[cowboy-cod3r]

Hi,

We recently had some issues with our 3 node Eureka Server cluster. We had some major scaling events which brought the number of registered instances with Eureka to about 2400. Upon reaching this number, file handles and CPU spiked. Eureka was unable to recover from this and our only recourse was to scale back our services and kill off the 2 Eureka Servers that were in distress (the 3rd was not in distress). We were running an older version of Eureka (1.6.2) via spring cloud. We have since upgraded Eureka to version 1.9.13. After our upgrade to 1.9.13 we noticed we could now register about 3500 instances before we would start seeing a significant number of socket timeouts and CPU would jitter quite a bit. We then found the following issue and supporting pull request:

• server in denial of service caused by flooding replication requests from peers #982
• prevent replication flood caused by socket read time out leads to some node CPU 100% #1031

In short, the guidance was to increase the peer node read timeout using the eureka.server.peer-node-read-timeout-ms property. We did this and doing so got rid of timeouts completely and settled CPU. Our target was to register 10,000 instances which we've now hit.

Our next test was to figure out what impact scaling the Eureka server horizontally from 3 instances to 9 instances would have. When we scaled out with 10,000 registered instances, each eureka server seemed to be giving a different picture of the world and we'd never see the same number of instances registered between our eureka server. Most of the time each server was off by the thousands. I tried scaling back down to 6 instances but that did not work either. It wasn't until I scaled back to three did things start to settle out. So here are my questions:

1. Is there any real benefit to scaling Eureka horizontally other than providing redundancy?
2. Are there specific tuning parameters that may be helpful for replication when scaling out the number of Eureka servers?
3. Regarding the eureka.server.max-elements-in-status-replication-pool property, what exactly defines an element? Does this relate to the number of registered instances?
4. Would you be able to share any numbers on the netflix side in terms of performance testing for Eureka?

To possibly help with any of my questions above, here is what our configurations look like on our Eureka servers (we use DNS resolution to find the eureka servers and I've omitted those properties from the config below). I realize some of these properties are the default but we define them anyways in our application.properties file.

# Server Specific Settings
server.port=8761
server.tomcat.max-threads=500

# Client Settings
eureka.client.register-with-eureka=true
eureka.client.fetch-registry=false
eureka.client.enabled=true
eureka.client.prefer-same-zone-eureka=false
eureka.client.on-demand-update-status-change=true

# Instance Settings
eureka.instance.app-group-name=SERVICE-DISCOVERY
eureka.instance.appname=eureka
eureka.instance.non-secure-port=${server.port}
eureka.instance.virtual-host-name=eureka

# Upon startup, the time to wait before eureka will serve its cache.  You want to wait long enough, where you
# think the cache will be populated enough to provide to backends.  You don't want to serve an empty cache.
eureka.server.wait-time-in-ms-when-sync-empty=180000

# Turn on self preservation mode and sent renewal threshold to 30%. If 30% are lost at once, stop ejecting instances.
eureka.server.enable-self-preservation=true
eureka.server.renewal-percent-threshold=0.30

# How often eureka will requery DNS for a new peer listing.
eureka.server.peer-eureka-status-refresh-time-interval-ms=60000

# Indicates whether the replication between cluster nodes should be batched for network efficiency. Default is false.
eureka.server.batch-replication=true

# The replicated data sent in the request will be always compressed. Default is false.
eureka.server.enable-replicated-request-compression=false

# Max number of replication events that can be allowed to back up in the status replication pool.  Default is 10000.
eureka.server.max-elements-in-status-replication-pool=10000

# Max number of replication events that can be allowed to back up in the replication pool.  This replication pool
# is responsible for all events except status updates.  Default is 10000.
eureka.server.max-elements-in-peer-replication-pool=10000

# The min number of threads to be used for replication. Default is 5.
eureka.server.min-threads-for-peer-replication=5

# The max number of threads to be used for replication. Default is 20.
eureka.server.max-threads-for-peer-replication=20

# Total number of HTTP connections allowed to peer eureka nodes for replication. Default is 1000.
eureka.server.peer-node-total-connections=1000

# Total number of HTTP connections allowed to a particular peer eureka node for replication. Default is 500.
eureka.server.peer-node-total-connections-per-host=500

# Replication timeout.  Default is 200
eureka.server.peer-node-read-timeout-ms=10000

# Others to consider
eureka.server.max-idle-thread-age-in-minutes-for-peer-replication=15
eureka.server.max-threads-for-status-replication=10
eureka.server.peer-node-connect-timeout-ms=1000
eureka.server.peer-node-connection-idle-timeout-seconds=30

[cowboy-cod3r]

Another thing we've noticed, is that anytime we attempt to scale beyond 3 nodes, at least one of the nodes always reports that there are unavailable replicas. Over time, the node that reports the unavailable replicas changes.

Also, we randomly get this error:
ERROR c.n.e.c.ReplicationTaskProcessor - Batch update failure with HTTP status code 500; discarding 250 replication tasks

We'll see this several times in a row and all of a sudden a significant number of registered instances will be evicted from the Eureka server.

[troshko111]

Sean, thanks for the detailed write up. I don't see anything out of ordinary in your config, it should be ok, you might want to fix renewal-percent-threshold, it's the other way around (70% threshold means under 30% of nodes can drop dead at once).

Is there any real benefit to scaling Eureka horizontally other than providing redundancy?

I can offer a personal opinion only, I don't think there's a whole lot of benefit scaling horizontally beyond 2-3 nodes for redundancy alone, heck running 1 would not be crazy. Services should be designed in a way a temporary Name Service (Eureka) unavailability is not an issue, worst case if none of the servers are up, new deployments will be slowed down and the existing instances will have a potentially stale state (unless this is naturally very volatile, couple minutes of delay is not a problem); and this is only until you restart / redeploy the name service nodes after which it all goes back to normal. This is not a service you need many nines of availability for, there's a little reason to go above 99.9% for a name service but opinions vary.

An actual benefit of scaling horizontally may be amplifying the NIC throughput in case you have a lot of readers and are close to saturating the NICs on your Eureka nodes. The number of horizontal nodes here is proportional of the total throughput you want to handle, but I would not go crazy (maybe 5 tops), as this is a full mesh replication.

Are there specific tuning parameters that may be helpful for replication when scaling out the number of Eureka servers?

Batching is probably the only important one. Try playing with these settings.

Regarding the eureka.server.max-elements-in-status-replication-pool property, what exactly defines an element? Does this relate to the number of registered instances?

eureka-core/src/main/java/com/netflix/eureka/cluster/PeerEurekaNode.java:245, this is used for ASG status replication only.

Would you be able to share any numbers on the netflix side in terms of performance testing for Eureka?

I'm unable to share the exact numbers but can say that we have significantly more instances than a couple thousands, but trying to keep the NIC saturation under 25% on the Eureka nodes. We do not run a massive number of Eureka nodes, just a couple (horizontally), most of the traffic is reads, the replication is negligible in comparison, so that's the main optimization opportunity.

Based on your write up, it seems like you may be having either networking issues or sub-optimal tomcat configuration, I suggest start with checking that, this is a bunch of blocking IO so one thing to check is whether the pools are getting full and hence requests are getting dropped due to that.

[cowboy-cod3r]

@troshko111
Thank you for taking the time to respond above. I'll have to dive into our tomcat settings. We currently deploy eureka through spring-cloud and the only thing we tune WRT to tomcat is server.tomcat.max-threads.

For now, we have decided to stick with three eureka servers. Regarding the consistency issue, I did notice that consistency seemed to get better when I changed eureka.client.fetch-registry from false to true. We ended up changing it to true based on another issue we were seeing where the renewal threshold would go down to 0 (although it wouldn't start out that way). Here is the link that gave me the answer to that:
spring-cloud/spring-cloud-netflix#380 (comment)

Thank you again. I think we can close this one out.