New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eureka Clustering documentation and best practices #203

Open
aivans opened this Issue Feb 10, 2015 · 26 comments

Comments

Projects
None yet
10 participants
@aivans

aivans commented Feb 10, 2015

I understand that there is a section in the documentation about Eureka Peer Awareness, but somehow I do not find this enough.
I find it hard to understand how this would make Eureka more HA. There would be a chain server1, server2, server3 and they are cross referenced, but if one node fails the chain breaks anyway.
Would it be a good option to just specify a list containing all 3 servers separated by comma for each all of them (server1, server2, server3)?

What is your experience?

PS: did not try DNS yet

@aivans

This comment has been minimized.

aivans commented Feb 11, 2015

Here are my fist findings:

  • the HA strategy seems to be one main instance with backup eureka servers
  • the clients are provided with a list of eureka servers through config (or dns)
  • the clients connect to the first server from the list, and this is where they send heartbeats and fetch information
  • the other eureka servers are just sitting idle and have no registered application (unless eureka servers reference each other, and in this case only referenced eureka instances will show app, but no business clients)
  • in case server1 becomes unavailable, all the clients migrate to the next eureka server from the list
    and keep trying to contact server1
  • in case server1 comes back online, all the clients migrate back to server1

Remarks:

  • this seems to be sufficient to have HA considering that clients cache locally

Questions and concerns:

  • in case of a high number of clients, it may be that in case server1 fails, server2 may fail too because all clients will try to migrate to it
    Seems like they took care of this when enabling DNS for serviceURLs:
    // Rearrange the fail over server list to distribute the load
    arrangeListBasedonHostname(serviceUrls);
    This leads to another concern.
  • I tried to connect a few services to server1 and a few to server2 by specifying different orders in serviceUrls list. The problem is that in this case topology partitions are created. Even if the 2 eureka servers are cross referenced and they are aware of eachother they do not exchange their registered applications, so BusinessService1 connected to Server1 will have no knowledge about BusinessService2 connected to Server2. The only time the servers have same registry information is when a server is restarted manually. At this point DiscoveryClient.getAndStoreFullRegistry() is called at startup. But after a few minutes, some instances are expired becuase no heartbeats are received.

Remarks:

  • would be nice to be able to distribute clients to different Eureka servers if the servers would exchange their registries regularly, but maybe this was not the intention
  • Maybe it is working with DNS enabled since there exists the capability to connect to different servers in the list randomly
@aivans

This comment has been minimized.

aivans commented Feb 13, 2015

Hi again,

I am pleased to announce that the behaviour from the previous post happens because all my Eureka nodes were configured with localhost as host name.
Changing to peer1 and peer2 as in the documentation allows for the creation of a mirrored style cluster.

@radenui

This comment has been minimized.

radenui commented Mar 2, 2015

Hello,

I have a similar problem here.
I'm using AWS, and I'd like to leverage the HA provided by Netflix using DNS and the region settings (https://github.com/Netflix/eureka/wiki/Configuring-Eureka-in-AWS-Cloud#configuring-eips-using-dns).
I'm not really interested in EIP or Auto-scaling groups here, just the DNS part.

As my Eureka clients will be in different Availability zones, I will need them to contact the Eureka server in their own availability zone (as a priority, with fallback on other zones in case of a problem).
And I still will need my Eureka servers to connect each other (across multiple AZs) and share their configuration.

It seems like I cannot use the same properties defined in Netflix wiki (they are not interpreted by the client).
And the way the authentication works (in the URL !!) seems a little restrictive in this case.

Do you have any suggestion ?

Thanks a lot,

Arthur

@aivans

This comment has been minimized.

aivans commented Mar 2, 2015

eureka:
  client:
    registerWithEureka: true
    fetchRegistry: true
    useDnsForFetchingServiceUrls: true
    eurekaServerDNSName: eureka.local
    eurekaServerPort: 8761
    eurekaServerURLContext: eureka  

The settings above definitely do the trick for me.

@aivans

This comment has been minimized.

aivans commented Mar 2, 2015

I did not test multiple zones though.
Please come back and share your results for your multiple zones setup.

@radenui

This comment has been minimized.

radenui commented Mar 3, 2015

Thank you for your answers!

I started using these properties:

eureka:
  client:
    registerWithEureka: true
    fetchRegistry: true
    useDnsForFetchingServiceUrls: true
    eurekaServerDNSName: eureka.local
    eurekaServerPort: 8761
    eurekaServerURLContext: eureka
    region: us-east-1
    availabilityZones:
      us-east-1: us-east-1a,us-east-1b
    preferSameZoneEureka: true

On the DNS side, I have:

  • a TXT record for the region
  • one TXT record by zone (with "user:password@server" as a value).

Into my Eureka servers (wanted them to descover their peers with DNS) and it seems like the DNS discovery does not allow to define peers at all: I had a very strange and inconsistent behavior using the same client params. Any idea why ?

But this seem to work pretty well with the eureka clients:

For the availability zones, not sure what is happening: I have this error sometimes (and have no idea why):

No match for the zone us-east-1a in the list of available zones [us-east-1b]

And

Updating the serviceUrls as they seem to have changed from [http://user:password@server-us-east-1b:8761/eureka/] to [http://user:password@server-us-east-1a:8761/eureka/]

I will try to increase the log level to see what's happening here, and will keep you in touch.
Thanks a lot,

Arthur

@aivans

This comment has been minimized.

aivans commented Mar 3, 2015

Never tested multiple zones.
In one single zone (defaultZone) I could configure multiple servers and they saw eachother as replicas.

How are your clients configured?

@radenui

This comment has been minimized.

radenui commented Mar 3, 2015

I tried the same "client" configuration for both eureka clients and servers (see previous message).
It only works on the clients.
On the Eureka servers, they are not able to use the DNS information to define peers: I had to use "defaultZone" as well.

@aivans

This comment has been minimized.

aivans commented Mar 3, 2015

i would start with the default region(don't try to set it yet) and default zone and see if the dns config is ok.
It is ok when in the Eureka web UI you will see servers as available replicas when they are all started.
Afterwards, I would try a different zone for one server.

My dns config:
txt.us-east-1 IN TXT "defaultZone.eureka.local"
txt.defaultZone IN TXT "user:password@peer1" "user:password@peer2"

@aivans

This comment has been minimized.

aivans commented Mar 3, 2015

availabilityZones:
      us-east-1: us-east-1a,us-east-1b

My hunch is that for the client you should specify one single zone, after all the client runs in a single zone, isn't it?

@aivans

This comment has been minimized.

aivans commented Mar 3, 2015

Actually, i think your issue is that you specify availabilityZones. you should leave that to the dns.

@radenui

This comment has been minimized.

radenui commented Mar 3, 2015

Ok, I can try that.
But as I cannot set the datacenter as "Amazon", how does the eureka client know its zone ?

@aivans

This comment has been minimized.

aivans commented Mar 3, 2015

datacenter=cloud?

@radenui

This comment has been minimized.

radenui commented Mar 3, 2015

I like that!
what is the full name for this property? just "datacenter" ?

@aivans

This comment has been minimized.

aivans commented Mar 3, 2015

Arthur, maybe you need to read the docs a bit, and the source code.

@aivans

This comment has been minimized.

aivans commented Mar 3, 2015

never tried it myself on Amazon

@radenui

This comment has been minimized.

radenui commented Mar 3, 2015

Well, I'll be delighted to find some docs... Because when I believe what I see in the source code, I have this!
Thanks for your help Adrian, I'll be interested if you can find any documentation on it.

@ryanjbaxter

This comment has been minimized.

Contributor

ryanjbaxter commented Jan 7, 2016

+1 adding additional documentation around how to configure this would be helpful

@aivans

This comment has been minimized.

aivans commented Jan 7, 2016

@ryanjbaxter are you referring to Eureka Clustering in general or Amazon info?

@cforce

This comment has been minimized.

cforce commented Feb 16, 2016

Is there an exampe (including dns records setup) describing how to setup 4 eureka nodes , whre each of two are in an own zone: zone1 and zone2 beeing in the same region but having zone affinity load balancing for all zuul and feign requests?

@Shawn-Cao

This comment has been minimized.

Shawn-Cao commented Mar 18, 2016

+1 need an example with 3+ nodes.

The sample on http://projects.spring.io/spring-cloud/spring-cloud.html#_peer_awareness doesn't look right. I expect each peer to still default to it's own zone while knowing peers from other zones.

I'm playing with config like below without success:

eureka:
  client:
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
    availability-zones:
      local: http://localhost:8761/eureka/
      local2: http://localhost:8762/eureka/
      local3: http://localhost:8763/eureka/
      cloud: ...
@kennedyoliveira

This comment has been minimized.

kennedyoliveira commented Mar 18, 2016

+1, i find the @cforce comments interesting, would like to see/know how to do that.

@Shawn-Cao

This comment has been minimized.

Shawn-Cao commented Mar 24, 2016

After some trial and error. I got a configuration set which allow a few Eureka servers synchronize and share apps registered under each of them. Sharing below:

----eureka-app.yml---- (for eureka servers only)
eureka:
  client:
    prefer-same-zone-eureka: true
    use-dns-for-fetching-service-urls: false
    region: shawn
#    fetch-remote-regions-registry: 'shawn,region2'
    availability-zones:
      shawn: 'pcfShawn,localShawn'
      region2: 'pcf1,pcf2'
    serviceUrl:
      'pcfShawn': 'http://<yourCloudFoundryEurekaUrl>/eureka/'
      'localShawn': 'http://XCao:8761/eureka/'
      'pcf1': 'http://<somewhereElseEurekaUrl>/eureka/'
      'pcf2': 'http://<yetSomewhereElseEurekaUrl>/eureka/'

----application.properties---- (for all apps)
#Eureka
eureka.instance.appname=${spring.application.name}
eureka.instance.non-secure-port=${server.port}
eureka.instance.hostname=${localhost.hostname:<yourHostName>}
eureka.instance.virtualHostName=${eureka.instance.hostname}:${eureka.instance.non-secure-port}
eureka.instance.instanceId=${eureka.instance.virtualHostName}
eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka/
@cforce

This comment has been minimized.

cforce commented Mar 25, 2016

This seems incomplete to me. Where is the eureka cfg in the same zone
sycned with same zone affinity,where the eureka in the different region
configured for region failover. Where is per region the client
configuration?

@daggerok

This comment has been minimized.

daggerok commented Apr 20, 2016

check it out
1

$ cat /etc/hosts
127.0.0.1       peer1
127.0.0.1       peer2

2

$ cat configs/eureka-replica.properties
server.port=8761
eureka.datacenter=peer1
eureka.instance.hostname=peer1
eureka.client.serviceUrl.defaultZone=http://peer2:8762/eureka/

3

$ cat configs/eureka.properties
server.port=8762
eureka.datacenter=peer2
eureka.instance.hostname=peer2
eureka.client.serviceUrl.defaultZone=http://peer1:8761/eureka/

4

$ cat configs/application.properties
eureka.server.registry-sync-retry-wait-ms=500
eureka.server.a-sgcache-expiry-timeout-ms=60000
eureka.server.eviction-interval-timer-in-ms=30000
eureka.server.peer-eureka-nodes-update-interval-ms=15000
eureka.server.renewal-threshold-update-interval-ms=300000

eureka.client.healthcheck.enabled=true
eureka.client.prefer-same-zone-eureka=true
eureka.client.region=zone2
eureka.client.availability-zones.zone1='peer1,peer2'
eureka.client.availability-zones.zone2='peer2,peer1'
eureka.client.serviceUrl.peer1=http://peer1:8761/eureka/
eureka.client.serviceUrl.peer2=http://peer2:8762/eureka/
eureka.client.serviceUrl.defaultZone=http://peer2:8762/eureka/
# seems like defaultZone must be configured
# so defaultZone peer will use all clients on start up registration if it will not specify
eureka.instance.appname=${spring.application.name}
eureka.instance.instanceId=${spring.application.name}-${spring.cloud.client.ipAddress}-${INSTANCE_ID:${random.value}}
eureka.instance.statusPageUrlPath=${management.contextPath}/info
eureka.instance.healthCheckUrlPath=${management.contextPath}/health
eureka.instance.preferIpAddress=true
# and other out of topic options
management.contextPath=/admin
endpoints.health.sensitive=false
spring.cloud.config.discovery.enabled=true

when not first peer will start, you'll find logs like these:

...PeerAwareInstanceRegistryImpl    : Got <some numbers of> instances from neighboring DS node
...PeerAwareInstanceRegistryImpl    : Renew threshold is: <some other number>
...PeerAwareInstanceRegistryImpl    : Changing status to UP

when peer1 go down, you will see log like this:

ERROR ... [-target_peer1-1] ...cluster.ReplicationTaskProcessor   : Network level connection to peer peer1; retrying after delay

as soon it will restore you'll see common instance registeration log message

@asarkar

This comment has been minimized.

Contributor

asarkar commented Jan 21, 2017

I've created a blog post with the details of Eureka here, that fills in some missing detail from Spring doc or Netflix blog. It is the result of several days of debugging and digging through source code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment