Health monitoring issue with SBA Cluster for Apps registered via Simple Discovery Client

[umeshupawar]

There is an issue for monitoring applications registered via Simple Discovery Client using SBA cluster.

Only first node in cluster monitors health of application instances in the cluster. All remaining nodes doesn’t monitor health of instances and sends notifications. For all remaining nodes application instances registration status shows already registered. As a result, 2nd instance doesn’t start health monitoring automatically even after the 1st (first) instance goes down in the cluster as registration status doesn’t change and registration event doesn’t get fired.

After this even if the first instance starts again it won’t start monitoring health of applications as instance status is still already registered.
In this case, Only option is to bring down the entire SBA cluster (All nodes) and start all nodes again then one of the cluster nodes starts monitoring application health and starts sending notifications.

Need to fix this issue.

[joshiste]

Which version of SBA are you using? Can you provide a project to reproduce the issue?

[umeshupawar]

I am using spring boot admin version 2.1.6.
It's simple to reproduce. Create one or two spring boot applications.
Add client information in admin server application property file.

spring:
  cloud:
    discovery:
      client:
        simple:
          instances:
            test:
              - uri: http://instance1:8080
                metadata:
                  management.context-path: /actuator
              - uri: http://instance2:8080
                metadata:
                  management.context-path: /actuator

Create two instances of admin server in clusters by adding Hazelcast dependencies.
spring.boot.admin.hazelcast.enabled=true

Start two instances and you will notice health is getting monitored by only first instance.
Stop first admin server instance - you will notice that health is not getting monitored.
Start first admin server instance again while second instance is UP and Running. - After this as well health check will be disabled.

[joshiste]

If you want me to have a look at it please provide a project to reproduce the issue.

1. This way it I'm sure I run exact the same configuration as you are.
2. Imho the setup ain't simple and I currently don't have the time to do it myself.

[umeshupawar]

Thanks. It's not very urgent to fix it immediately. I am also currently too much busy.
The existing project can't be shared as it is since there are multiple components involved. Apps running on 3 to 4 systems with cluster environment and config. I have to perform some cleanup and make it simple so that issue can be reproduced on one system with minimum and simple config. I will work on it and will provide a project to reproduce the issue once I get time.
Till that time the Issue can be kept on hold for verification.

[jayavardhanpannem]

How to we pass additional metadata like below, in case of spring cloud Kubernetes with spring boot admin without eureka server or spring boot admin clients.

eureka.instance.status-page-url=${server.servlet.context-path}/actuator/info
eureka.instance.metadata-map.user.name=${spring.security.user.name}
eureka.instance.metadata-map.user.password=${spring.security.user.password}
eureka.instance.metadata-map.path=${server.servlet.context-path}/actuator/health
eureka.instance.metadata-map.context-path=${server.servlet.context-path}
eureka.instance.metadata-map.management.context-path=${server.servlet.context-path}/actuator

[joshiste]

@jayavardhanpannem
In future please try to avoid asking questions totally unrelated to the issue.
To answer your question nevertheless: As I expect that you are using the spring-cloud-kubernetes discovery client in the admin server, please look it up in it's documentation. But I'd guess you need to add the metadata to the k8s services' yml descriptor.

[uvwild]

i also observed a similar issue in our docker swarm application.
ANY configured endpoint in cloud.discovery.client.simple.instances.OurService.uri
shows up as an application instance in spring boot admin no matter if the node is running an instance

So the information in the spring boot admin UI is inherently wrong and confusing.

I inherited the application and there might be an errror in how its configured.
However, I could not find any further information regarding the config details in a docker swarm.
Some clarification how the actual service instances are to be correctly configured in the SBA config woud be helpful.

dependencies {
compile group: 'org.springframework.boot', name: 'spring-boot-starter-security'
compile group: 'org.springframework.boot', name: 'spring-boot-starter-mail'
compile group: 'de.codecentric', name: 'spring-boot-admin-server', version: '1.5.7'
compile group: 'de.codecentric', name: 'spring-boot-admin-server-ui', version: '1.5.7'
compile group: 'de.codecentric', name: 'spring-boot-admin-server-ui-hystrix', version: '1.5.7'
compile group: 'de.codecentric', name: 'spring-boot-admin-server-ui-login', version: '1.5.7'
compile group: 'de.codecentric', name: 'spring-boot-admin-server-ui-turbine', version: '1.5.7'
compile group: 'com.netflix.turbine', name: 'turbine-core', version: '1.0.0', {exclude (group: 'javax.servlet', module: 'servlet-api')}

[joshiste]

ANY configured endpoint in cloud.discovery.client.simple.instances.OurService.uri
shows up as an application instance in spring boot admin no matter if the node is running an instance

If you use the static discovery client to define static services, we expect a service to run on the URI. If not we report this service as offline.

[omaraligit]

okay so if I run my 2 microservice projects [api-gateway, products-service] using the docker-compose file it's okay I get the health check from actuator on SBA dashboard.
but once I run my apps on Kubernetes using the production, Service YAML files the products-service is discovered by eureka but the SBA shows the service as INSTANCES DOWN
my questions

• how did SBA see products-service if it's down
• why does it show on eureka
  you can see the projects here its for learning purposes
  https://github.com/omaraligit/jee-microservices-startup

[joshiste]

how did SBA see products-service if it's down

using the health-check url from eureka.