Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate common Elasticsearch configuration properties #23106

Closed
izeye opened this issue Aug 27, 2020 · 7 comments
Closed

Consolidate common Elasticsearch configuration properties #23106

izeye opened this issue Aug 27, 2020 · 7 comments
Assignees
Labels
type: enhancement A general enhancement
Milestone

Comments

@izeye
Copy link
Contributor

izeye commented Aug 27, 2020

If imperative Spring Data Elasticsearch is being used and Spring Boot WebFlux starter happens to exist only for WebClient, the following exceptions are thrown:

2020-08-27 12:13:48.938 ERROR 14810 --- [ctor-http-nio-2] reactor.core.publisher.Operators         : Operator called default onErrorDropped

reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.data.elasticsearch.client.NoReachableHostException: Host 'localhost:9200' not reachable. Cluster state is offline.
Caused by: org.springframework.data.elasticsearch.client.NoReachableHostException: Host 'localhost:9200' not reachable. Cluster state is offline.
	at org.springframework.data.elasticsearch.client.reactive.SingleNodeHostProvider.lambda$lookupActiveHost$4(SingleNodeHostProvider.java:108) ~[spring-data-elasticsearch-4.1.0-M2.jar:4.1.0-M2]
	at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:102) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Mono.subscribe(Mono.java:3892) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2021) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:259) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:216) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:265) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:413) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:250) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:373) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.DirectProcessor.emitNext(DirectProcessor.java:180) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.SerializedManySink.emitNext(SinksSpecs.java:112) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:183) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:284) ~[reactor-netty-http-1.0.0-M2.jar:1.0.0-M2]
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:158) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:423) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$5(SimpleDequePool.java:253) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:538) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:433) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.netty.transport.TransportConnector$MonoChannelPromise$1.tryFailure(TransportConnector.java:484) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

2020-08-27 12:13:49.005  INFO 14810 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-08-27 12:13:49.053  INFO 14810 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-08-27 12:13:49.068  INFO 14810 --- [           main] com.izeye.throwaway.Application          : Started Application in 3.004 seconds (JVM running for 4.365)
2020-08-27 12:13:49.588  INFO 14810 --- [1)-192.168.0.21] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-08-27 12:13:49.588  INFO 14810 --- [1)-192.168.0.21] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2020-08-27 12:13:49.589  WARN 14810 --- [ctor-http-nio-3] a.e.ElasticsearchReactiveHealthIndicator : Elasticsearch health check failed

org.springframework.data.elasticsearch.client.NoReachableHostException: Host 'localhost:9200' not reachable. Cluster state is offline.
	at org.springframework.data.elasticsearch.client.reactive.SingleNodeHostProvider.lambda$lookupActiveHost$4(SingleNodeHostProvider.java:108) ~[spring-data-elasticsearch-4.1.0-M2.jar:4.1.0-M2]
	at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:102) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:220) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Mono.subscribe(Mono.java:3892) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2021) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:259) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:216) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:265) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:413) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:250) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:373) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.DirectProcessor.emitNext(DirectProcessor.java:180) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.SerializedManySink.emitNext(SinksSpecs.java:112) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:183) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:284) ~[reactor-netty-http-1.0.0-M2.jar:1.0.0-M2]
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:158) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:423) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$5(SimpleDequePool.java:253) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:538) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[reactor-core-3.4.0-M2.jar:na]
	at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:433) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at reactor.netty.transport.TransportConnector$MonoChannelPromise$1.tryFailure(TransportConnector.java:484) ~[reactor-netty-core-1.0.0-M2.jar:1.0.0-M2]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

I can work around it by excluding auto-configurations for reactive ones:

spring.autoconfigure.exclude=\
  org.springframework.boot.actuate.autoconfigure.elasticsearch.ElasticSearchReactiveHealthContributorAutoConfiguration,\
  org.springframework.boot.autoconfigure.data.elasticsearch.ReactiveElasticsearchRepositoriesAutoConfiguration,\
  org.springframework.boot.autoconfigure.data.elasticsearch.ReactiveElasticsearchRestClientAutoConfiguration

This is a sample project to reproduce it: https://github.com/izeye/spring-boot-throwaway-branches/tree/data-elasticsearch-and-web-client

It would be good if an option for choosing among imperative, reactive, and both was given.

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Aug 27, 2020
@wilkinsona wilkinsona added the for: team-attention An issue we'd like other members of the team to review label Oct 13, 2020
@philwebb
Copy link
Member

See also #22692

@philwebb philwebb added this to the 2.4.x milestone Oct 23, 2020
@philwebb philwebb added type: enhancement A general enhancement and removed for: team-attention An issue we'd like other members of the team to review status: waiting-for-triage An issue we've not yet triaged labels Oct 23, 2020
@bclozel
Copy link
Member

bclozel commented Oct 23, 2020

Note for the team - during our discussion we wondered what would happen if we'd configure both properties to the correct Elasticsearch endpoint. I've tried the following configuration (note that they need different values here...):

spring.elasticsearch.rest.uris=http://localhost:9201
spring.data.elasticsearch.client.reactive.endpoints=localhost:9201

The application did start correctly and Spring Data detected the repository type:

2020-10-23 18:03:32.612  INFO 55842 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Elasticsearch repositories in DEFAULT mode.
2020-10-23 18:03:32.657  INFO 55842 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 41 ms. Found 1 Elasticsearch repository interfaces.
2020-10-23 18:03:32.662  INFO 55842 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Reactive Elasticsearch repositories in DEFAULT mode.
2020-10-23 18:03:32.665  INFO 55842 --- [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 3 ms. Found 0 Reactive Elasticsearch repository interfaces.

During our team call, we discussed this issue and had a few ideas in mind.

We could move both spring.elasticsearch.rest.uris and spring.data.elasticsearch.client.reactive.endpoints to a new, shared property that would cover all cases: an Elasticsearch REST client (low or high level), Spring Data Repositories (imperative powered by the Elasticsearch REST client, or reactive powered by WebClient).

My previous comment shows that even with a classpath containing all of Spring Data Elasticsearch, the Elasticsearch REST client and WebFlux, this would still work as expected and Spring Data would make the right choice with the repositories.

Now the spring-boot-starter-data-elasticsearch brings in Spring Data Elasticsearch and the Elasticsearch REST clients. We could consider creating a different starter spring-boot-starter-data-reactive-elasticsearch that would bring WebFlux and exclude the Elasticsearch REST clients, just to split the use cases a bit more.

@wilkinsona
Copy link
Member

The reactive client wanting host:port pairs while the imperative client wants http:// or https:// URIs that may contain a username and password too. This makes a common spring.elasticsearch.uris property difficult to implement. There are separate spring.data.elasticsearch.client.reactive.username, spring.data.elasticsearch.client.reactive.password, and spring.data.elasticsearch.client.reactive.use-ssl properties to consider that may have some overlap with a URI provided in spring.elasticsearch.uris. The auto-configuration of the imperative client already deals with some of this to an extent. It considers both the spring.elasticsearch.rest.username/spring.elasticsearch.rest.password properties and the user info from the URIs. We may be able to do something similar in the reactive case but it's feeling quite complicated.

@wilkinsona
Copy link
Member

wilkinsona commented Oct 27, 2020

If we share a single uris property, we probably need to consolidate the properties for username and password as well. At the moment we've got the following:

  • spring.elasticsearch.rest.username
  • spring.elasticsearch.rest.password
  • spring.data.elasticsearch.client.reactive.username
  • spring.data.elasticsearch.client.reactive.password

There's the added complication that the imperative REST client side of things also allows credentials to be provided in the URIs. When there are multiple URIs, it's possible for each to use different credentials. Using different credentials for different endpoints isn't supported by the reactive REST client so we may need to fail if multiple URIs with different credentials are configured.

@wilkinsona
Copy link
Member

After a chat with @snicoll, we concluded that the timeout properties should be consolidated as well. At the moment, we've got the following:

  • spring.elasticsearch.rest.connection-timeout
  • spring.elasticsearch.rest.read-timeout
  • spring.data.elasticsearch.client.reactive.connection-timeout
  • spring.data.elasticsearch.client.reactive.socket-timeout

Despite its name, spring.elasticsearch.rest.read-timeout configures a socket timeout so we have two sets of properties for configuring connection and socket timeouts. They could become the following:

  • spring.elasticsearch.connection-timeout
  • spring.elasticsearch.socket-timeout

This is becoming and increasingly big change. The original problem has a workaround (excluding some auto-configuration) so our feeling is that we shouldn't try to squeeze this into 2.4. Instead, we'll defer this to 2.5 where it can be implemented early on and have longer to bake before release.

@wilkinsona wilkinsona changed the title Imperative Spring Data Elasticsearch throws exceptions with Spring Boot WebFlux starter Consolidate common Elasticsearch configuration properties Oct 27, 2020
@wilkinsona wilkinsona modified the milestones: 2.4.x, 2.5.x Oct 27, 2020
@TarekSaid
Copy link

TarekSaid commented Jan 27, 2021

hm...

I don't know if it follows Spring conventions, but what about something like this:

  • spring.elasticsearch.connection-timeout
  • spring.elasticsearch.read-timeout
  • spring.elasticsearch.uris (like the spring.datasource.url property, can accept different URI patterns, such as http://host:port, http://username:password@host:port, https://host:port, https://username:password@host:port/path/prefix, etc, instead of specifying the other properties below)
  • spring.elasticsearch.host
  • spring.elasticsearch.port
  • spring.elasticsearch.username
  • spring.elasticsearch.password
  • spring.elasticsearch.use-ssl
  • spring.elasticsearch.client-type = rest (default) / reactive OR spring.elasticsearch.reactive.enabled = true (default=false)

edit: I know this should probably be in a separate issue, but it would be nice if we could have a path-prefix property (for an Elasticsearch server behind a proxy):

  • spring.elasticsearch.path-prefix

@wilkinsona
Copy link
Member

wilkinsona commented Jan 27, 2021

Thanks for the suggestions, @TarekSaid. For once, naming things isn't the hard part in this case. The difficulty is in how to handle the cases where there are multiple properties at the moment that we'd like to combine into one. Some of these difficulties are described in the comments above. If you're interested, the beginnings of the changes that are needed are in this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: enhancement A general enhancement
Projects
None yet
Development

No branches or pull requests

6 participants