Clarify peer replication setup of eureka servers

[william-tran]

In order for peer replication to work properly, all eureka servers need to know about all other peers. Given servers peer1, peer2, peer3, this won't work.

---
spring.profiles: peer1
eureka.instance.hostname: peer1
eureka.client.serviceUrl.defaultZone: http://peer2/eureka/

---
spring.profiles: peer2
eureka.instance.hostname: peer2
eureka.client.serviceUrl.defaultZone: http://peer3/eureka/

---
spring.profiles: peer3
eureka.instance.hostname: peer3
eureka.client.serviceUrl.defaultZone: http://peer1/eureka/

This is because a server receiving a peer replication request will not further replicate that request to the other peers it knows about: https://github.com/Netflix/eureka/blob/v1.4.10/eureka-core/src/main/java/com/netflix/eureka/registry/PeerAwareInstanceRegistryImpl.java#L610 This can result peers having different views and leases expiring because a particular peer hasn't received the heartbeats it was expecting from a client.

This passage in the docs seems to suggest that the above setup would be sufficient though:

You can add multiple peers to a system, and as long as they are all connected to each other by at least one edge, they will synchronize the registrations amongst themselves. If the peers are physically separated (inside a data centre or between multiple data centres) then the system can in principle survive split-brain type failures.

[ryanjbaxter]

I agree with your analysis of the documentation, I am interpreting it the same way. When you implement the above configuration what is the result on each peer?

[william-tran]

If all clients have the same property value:

eureka.client.serviceUrl.defaultZone: http://peer1/eureka/,http://peer2/eureka/,http://peer3/eureka/

peer1 and peer2 will have the full registry, and peer3 will be empty.

[ryanjbaxter]

So if each server has

eureka.client.serviceUrl.defaultZone: http://peer1/eureka/,http://peer2/eureka/,http://peer3/eureka/

peer3 is empty?

What is the result when you have the configuration in your issue description above?

---
spring.profiles: peer1
eureka.instance.hostname: peer1
eureka.client.serviceUrl.defaultZone: http://peer2/eureka/

---
spring.profiles: peer2
eureka.instance.hostname: peer2
eureka.client.serviceUrl.defaultZone: http://peer3/eureka/

---
spring.profiles: peer3
eureka.instance.hostname: peer3
eureka.client.serviceUrl.defaultZone: http://peer1/eureka/

[ccit-spence]

@william-tran @ryanjbaxter I am using this approach eureka.client.serviceUrl.defaultZone: http://peer1/eureka/,http://peer2/eureka/,http://peer3/eureka/

All three are showing the same information. Is that what is expected?

[william-tran]

@ryanjbaxter maybe I wasn't clear in my messages, I was describing the setup where the servers have this configuration

---
spring.profiles: peer1
eureka.instance.hostname: peer1
eureka.client.serviceUrl.defaultZone: http://peer2/eureka/

---
spring.profiles: peer2
eureka.instance.hostname: peer2
eureka.client.serviceUrl.defaultZone: http://peer3/eureka/

---
spring.profiles: peer3
eureka.instance.hostname: peer3
eureka.client.serviceUrl.defaultZone: http://peer1/eureka/

And the clients have

eureka.client.serviceUrl.defaultZone: http://peer1/eureka/,http://peer2/eureka/,http://peer3/eureka/

The clients are only sending requests to peer1 and have no need to fall back to peer2 or peer3.
peer2 has all the registry information because peer1 is replicating to it. peer3 has no information because neither the clients are sending anything to it, nor are the other peers sending anything to it.

[william-tran]

If the servers have this configuration:

eureka.client.serviceUrl.defaultZone: http://peer1/eureka/,http://peer2/eureka/,http://peer3/eureka/

---
spring.profiles: peer1
eureka.instance.hostname: peer1

---
spring.profiles: peer2
eureka.instance.hostname: peer2

---
spring.profiles: peer3
eureka.instance.hostname: peer3

Each peer knows the complete list of all peers, and so all peers will have the same information. Any client can use any peer and get the same up to date information. The point I'm trying to make is that this requirement is not clear in the documentation.

[ryanjbaxter]

OK makes more sense now. @spencergibb asked someone from the netflix OSS team to comment on the issue so lets see what the intended behavior is and then we can adjust the docs accordingly.

[spencergibb]

netflix uses dns so their "static" list isn't truly static and can be updated eventually.

[qiangdavidliu]

From an architectural point of view, each eureka server needs to know all other peers for replication to fully work. Currently in eureka there are two methods exposed to allow for this, one via config (the method described above), and another via a txt type dns record that allows for a bit more flexibility (the txt based dns record can be changed dynamically and be picked up by the servers if configured to do so).

[william-tran]

Thanks for confirming that behaviour @qiangdavidliu.

[spencergibb]

Some of the difference is that at netflix there is one eureka per zone, so peering happens accross zones. So clients in zone1 have the zone1 eureka first, then zones 2 & 3. Clients in zone 2 have zone 2 first then 1 & 3, etc... @qiangdavidliu tell me if I'm wrong.

[qiangdavidliu]

@spencergibb you are not wrong, though we have since moved away from a strict zonal configuration since. Today we run multiple eureka server per zone, and all eurekaClients can talk to any eureka server, however this is code that ensures clients in a zone preferentially connect to (a random) server in the same zone.

[spencergibb]

@qiangdavidliu thanks for the update

[spencergibb]

https://github.com/Netflix/eureka/wiki/Deploying-Eureka-Servers-in-EC2

[ryanjbaxter]

@william-tran based on what we know if we updated the docs to state something like

You can add multiple peers to a system, and as long as they are all connected to each other, they will synchronize the registrations amongst themselves. If the peers are physically separated (inside a data centre or between multiple data centres) then the system can in principle survive split-brain type failures.

Would that be clearer? Welcome to suggestions...

[spencergibb]

Closed via 4b40a45

[bsushant-athena]

@qiangdavidliu is it possible for you share an example repo for DNS based peer awareness across zones?

[n0mer]

@bsushant-athena https://github.com/Netflix/eureka/wiki/Deploying-Eureka-Servers-in-EC2#configuring-eips-using-dns