2.1 Elasticsearch health check failures

[wyaeld]

Seeing the following WARN appearing in logs constantly since 2.1

Elasticsearch cluster is 6.4.3-oss docker image running on GKE
Client libs are the 6.4.3 high level rest client

The actual code is driven directly via the elastic client libs of

import org.elasticsearch.client.RestClient
import org.elasticsearch.client.RestHighLevelClient

so the sudden failures of code that we haven't changed in months when we next deployed the service was surprising, but I understand actuator is now automatically running healthchecks since 2.1 autoconfiguration is finding the elastic client on the classpath.

I've disabled it for now, but it does seem something is wrong with the healthcheck itself. All our client code is still working, and the elastic cluster is functional

WARN Elasticsearch health check failed java.net.ConnectException: Connection refused 
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:943)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
at org.springframework.boot.actuate.elasticsearch.ElasticsearchRestHealthIndicator.doHealthCheck(ElasticsearchRestHealthIndicator.java:61)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84)
at org.springframework.boot.actuate.health.HealthIndicatorReactiveAdapter.invoke(HealthIndicatorReactiveAdapter.java:48)
at org.springframework.boot.actuate.health.HealthIndicatorReactiveAdapter.lambda$null$0(HealthIndicatorReactiveAdapter.java:43)
at reactor.core.scheduler.ElasticScheduler$DirectScheduleTask.run(ElasticScheduler.java:291)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ... 1 common frames omitted

[wilkinsona]

The health indicator uses each RestClient bean in the application context. It’s failing because the connection is being refused. That suggests that there is a misconfigured RestClient bean in the application context. I can’t tell anything more from the information provided thus far.

If you would like us to spend some more time trying to help you, please spend some time describing your client configuration in detail and, ideally, providing a minimal sample that reproduces the problem.

[wyaeld]

If the RestClient was misconfigured, wouldn't all other communication with the cluster fail?

Its a pretty vanilla setup, only using host and port

@Throws(UnknownHostException::class)
    private fun buildRestClient(): RestHighLevelClient {
        val restClientBuilder = RestClient.builder(HttpHost(host!!, port, "http"))

        return RestHighLevelClient(restClientBuilder)
    }

[wilkinsona]

If the RestClient was misconfigured, wouldn't all other communication with the cluster fail?

Yes, I would expect so.

Its a pretty vanilla setup, only using host and port

That's configuring a RestHighLevelClient. The health indicator uses RestClient beans.

Assuming they're on the classpath, the auto-configuration will create both a RestClient and a RestHighLevelClient. Both are created from a RestClientBuilder that is configured using the spring.elasticsearch.rest.* properties. I can't tell exactly what's happening in your case as you haven't described your configuration of the client in sufficient detail. I would guess that the auto-configured RestClient bean is still being created and it is using the default configuration which will mean that it attempts to connect to http://localhost:9200.

I would recommend using the spring.elasticsearch.rest.* properties to configure the RestClientBuilder from which the RestClient and RestHighLevelClient beans are created. If you prefer to configure things programatically, I would recommend defining your own RestClientBuilder or RestClientBuilderCustomizer bean rather than the client bean.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[liquidjoo]

uhm.. same this issue
i think..

because RestHighLevelClient another Spring boot process lock

I Solved
Another Spring boot process kill