Provide a way to listen to specific services instead of the whole catalog

[fuleow]

spring-cloud-consul provides a Consul Catalog Watch that publishes heartbeat events on catalog changes. In an environment with many services the catalog can change rapidly (multiple times per second) causing heartbeat events to trigger for services which the application is not interested in.

For example, this mechanism is used in Spring Cloud Config Client if discovery is enabled link

In practice the config client is only interested in updates to the spring-cloud-config-server but this triggers each time the catalog updates.

The catalog services watch can already be disabled. It would be very useful if an alternative heartbeat event producer can be implemented which takes a list of relevant services and only publishes heartbeat events when those specific services are updated.

Currently any micro service in our organization will start generating many requests to consul just by adding spring-cloud-starter-consul-discovery. While the watch-delay is configurable it's less than ideal if your application is only interested in a subset of services. If the watch-delay is too high you risk not getting an immediate update when the services changes and if it's too low you get flooded with events.

[spencergibb]

The consul api we use does not provide filtering. https://www.consul.io/api/catalog.html#list-services

[fuleow]

The consul api we use does not provide filtering. https://www.consul.io/api/catalog.html#list-services

Yes it doesn't provide filtering on the entire catalog. However, long polling watches can be created for each service that the application is interested in. The mechanism can be very similar to how it's done now with a background thread for each service.

https://www.consul.io/api/catalog.html#list-nodes-for-service

[spencergibb]

Having to explicitly list every service you want to watch doesn't seem scalable to me.

[fuleow]

I think that applications which have many upstream services can continue using the existing catalog watch mechanism. However it would be nice if microservices which only have a couple of interfaces be able to watch those they are interested in instead of the entire catalog.

Here is some data on our own consul index and how quickly it changes

while true; do curl -s -v http://consul:8500/v1/catalog/services 2>&1> /dev/null | grep Index; sleep 5; done
< X-Consul-Index: xxxx59768
< X-Consul-Index: xxxx59896
< X-Consul-Index: xxxx59961
< X-Consul-Index: xxxx60067
< X-Consul-Index: xxxx60243
< X-Consul-Index: xxxx60415
< X-Consul-Index: xxxx60548
< X-Consul-Index: xxxx60743
< X-Consul-Index: xxxx60774

[spencergibb]

I think that is a rarity. As you mentioned, you can disable it and roll your own. We'll wait to see if there are more folks who want this.

[alex-dubrouski]

Good afternoon,
Not sure what do you mean by rarity. It is very common to have hundreds of microservices and use continuous deployment strategies in huge organizations. May be you are operating at a different scale, but we have multiple datacenters and Consul catalog contains thousands of tags.
To be precise we started investigating high CPU/Memory/Network usage of some idle microservices running in the cloud. I profiled couple of them and found that they are all affected by the same problem. Spring schedules a task which continuously polls Consul and fetches updates (due to high rate of changes it happens every couple of seconds) it results in heavy memory consumption by underlying "com.ecwid.consul" library. This results in frequent garbage collections and high CPU usage.

Here is JProfiler hot spot allocation report. In reality absolutely idle service continuously uses around 400MB of memory. I am sorry but I think this implementation does not scale. We are currently suspended usage of it and looking for other ways to implement service discovery.

[spencergibb]

@alex-dubrouski please open a separate issue as that does not seem related to ConsulCatalogWatch. We'd be happy to entertain ways to make things more efficient as no one has reported anything similar.

What doesn't seem to be scalable is to have to manually add a service that needs to be watched for the OPs use case.

[alex-dubrouski]

Spencer,
Please review the attached screenshot again. ConsulCatalogWatch schedules task which fetches catalog here:
https://github.com/spring-cloud/spring-cloud-consul/blob/master/spring-cloud-consul-discovery/src/main/java/org/springframework/cloud/consul/discovery/ConsulCatalogWatch.java#L129
When rate of changes of Consul catalog is high and catalog itself is huge it results in this high memory allocation / CPU usage. Memory allocated for CatalogConsulClient.getCatalogServices()on attached picture is 289MB