Spring data elastic search bulkUpdate issue

[naidusana]

Spring data elastic bulkUpdate giving below error

Exception: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'Builder.'

Code:

List updateQueries = new ArrayList<>();
for(Map<String, Map<String, Object>> map : dataList) {
for(Entry<String, Map<String, Object>> e : map.entrySet()) {
if(StringUtils.isNotBlank(e.getKey()) && e.getValue() != null) {
String id = String.valueOf(e.getValue().get(idFieldName));
UpdateQuery updateQuery = UpdateQuery.builder(id)
.withScript(e.getKey())
.withParams(e.getValue())
.build();
updateQueries.add(updateQuery);
}
}
}
BulkOptionsBuilder bulkOptionsBuilder = BulkOptions.builder();
bulkOptionsBuilder.withTimeout(Duration.ofMillis(5000));
if(isRefresh) {
bulkOptionsBuilder.withRefreshPolicy(RefreshPolicy.IMMEDIATE);
}
BulkOptions bulkOptions = bulkOptionsBuilder.build();

		for (List<UpdateQuery> queryList : splitList) {
			esOps.bulkUpdate(updateQueries, bulkOptions, IndexCoordinates.of(indexName));
		}

SpringBoor: 3.1.10
data elastic search : 5.1.10

[sothawo]

Please provide a minimal compilable and runnable example that reproduces this error and a complete stacktrace.

[naidusana]

Hi, Please find below details

Earlier:

Springboot: 2.7.1
Spring Data elastic search: 4.4.1
Rest High Level Client : 7.17.5

Elastic Cluster: 8.10.3

Elastic Configuration:

@configuration
public class ElasticsearchConfig extends AbstractElasticsearchConfiguration{

@Bean
@Override
public RestHighLevelClient elasticsearchClient() {
	ElasticsearchProperty esProp = elasticsearchProperty();
	HttpHeaders compatibilityHeaders = new HttpHeaders();
	compatibilityHeaders.add(DataFlowConstants.ACCEPT, "application/vnd.elasticsearch+json;compatible-with=7);
	compatibilityHeaders.add(DataFlowConstants.CONTENT_TYPE, "application/vnd.elasticsearch+json;compatible-with=7);
	ClientConfiguration clientConfiguration = ClientConfiguration.builder()
			.connectedTo("url")
			.withBasicAuth("username", "password")
			.withDefaultHeaders(compatibilityHeaders)
			.withConnectTimeout(5000)
			.withSocketTimeout(5000)
			.build();
	
	RestHighLevelClient restClient = RestClients.create(clientConfiguration).rest();
	return restClient;
}

}

After Upgrade:

SpringBoot: 3.1.10
Spring Data Elastic search: 5.1.10

Elastic Cluster: 8.10.3

Elastic Configuration:

@configuration
public class ElasticsearchConfig extends ElasticsearchConfiguration {

@NotNull
public ClientConfiguration clientConfiguration() {
	ElasticsearchProperty esProp = elasticsearchProperty();
	HttpHeaders compatibilityHeaders = new HttpHeaders();
	return ClientConfiguration.builder()
			.connectedTo("url")
			.withBasicAuth("username", "password")
			.withDefaultHeaders(compatibilityHeaders)
			.withConnectTimeout(5000)
			.withSocketTimeout(5000)
			.build();
}	

}

Update documents Logic:

import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.elasticsearch.core.ElasticsearchOperations;
import org.springframework.data.elasticsearch.core.RefreshPolicy;
import org.springframework.data.elasticsearch.core.mapping.IndexCoordinates;
import org.springframework.data.elasticsearch.core.query.BulkOptions;
import org.springframework.data.elasticsearch.core.query.UpdateQuery;

import java.time.Duration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class Test {
@Autowired
private static ElasticsearchOperations esOps;

public static void main(String[] args) {
    List<Map<String, Map<String, Object>>> dataList = new ArrayList<>();
	
    Map<String, Map<String, Object>> scriptMap = new HashMap<>();
	
    Map<String, Object> map = new HashMap<>();
    map.put("ID", "ID1");
    map.put("NAME", "NAME1");
    scriptMap.put("ctx._source.NAME=params.NAME;", map);
	
    Map<String, Object> map1 = new HashMap<>();
    map1.put("ID", "ID2");
    map1.put("NAME", "NAME2");
    scriptMap.put("ctx._source.NAME=params.NAME;", map1);
	
    dataList.add(scriptMap);
    updateAllByScript("index", dataList, "ID", false);
}

public static void updateAllByScript(String indexName, List<Map<String, Map<String, Object>>> dataList, String idFieldName, boolean isRefresh) {

    List<UpdateQuery> updateQueries = new ArrayList<>();
    for (Map<String, Map<String, Object>> map : dataList) {
        for (Map.Entry<String, Map<String, Object>> e : map.entrySet()) {
            if (StringUtils.isNotBlank(e.getKey()) && e.getValue() != null) {
                String id = String.valueOf(e.getValue().get(idFieldName));
                UpdateQuery updateQuery = UpdateQuery.builder(id)
                        .withScript(e.getKey())
                        .withParams(e.getValue())
                        .build();
                updateQueries.add(updateQuery);
            }
        }
        BulkOptions.BulkOptionsBuilder bulkOptionsBuilder = BulkOptions.builder();
        bulkOptionsBuilder.withTimeout(Duration.ofMillis(5000));
        if (isRefresh) {
            bulkOptionsBuilder.withRefreshPolicy(RefreshPolicy.IMMEDIATE);
        }
        BulkOptions bulkOptions = bulkOptionsBuilder.build();
        esOps.bulkUpdate(updateQueries, bulkOptions, IndexCoordinates.of(indexName));
    }
}

}

Above example working fine in before upgrade and after upgrade getting below exception

Exception:

co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'Builder.'
at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76) ~[elasticsearch-java-8.7.1.jar:?]
at co.elastic.clients.elasticsearch._types.Script.(Script.java:78) ~[elasticsearch-java-8.7.1.jar:?]
at co.elastic.clients.elasticsearch._types.Script.(Script.java:51) ~[elasticsearch-java-8.7.1.jar:?]
at co.elastic.clients.elasticsearch._types.Script$Builder.build(Script.java:160) ~[elasticsearch-java-8.7.1.jar:?]
at co.elastic.clients.elasticsearch._types.Script$Builder.build(Script.java:134) ~[elasticsearch-java-8.7.1.jar:?]
at co.elastic.clients.elasticsearch._types.Script.of(Script.java:84) ~[elasticsearch-java-8.7.1.jar:?]
at org.springframework.data.elasticsearch.client.elc.RequestConverter.getScript(RequestConverter.java:716) ~[spring-data-elasticsearch-5.1.10.jar:5.1.10]
at org.springframework.data.elasticsearch.client.elc.RequestConverter.lambda$bulkUpdateOperation$26(RequestConverter.java:670) ~[spring-data-elasticsearch-5.1.10.jar:5.1.10]
at co.elastic.clients.elasticsearch.core.bulk.UpdateOperation$Builder.action(UpdateOperation.java:195) ~[elasticsearch-java-8.7.1.jar:?]
at org.springframework.data.elasticsearch.client.elc.RequestConverter.bulkUpdateOperation(RequestConverter.java:668) ~[spring-data-elasticsearch-5.1.10.jar:5.1.10]
at org.springframework.data.elasticsearch.client.elc.RequestConverter.lambda$documentBulkRequest$33(RequestConverter.java:768) ~[spring-data-elasticsearch-5.1.10.jar:5.1.10]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1242) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.springframework.data.elasticsearch.client.elc.RequestConverter.documentBulkRequest(RequestConverter.java:771) ~[spring-data-elasticsearch-5.1.10.jar:5.1.10]
at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.doBulkOperation(ElasticsearchTemplate.java:285) ~[spring-data-elasticsearch-5.1.10.jar:5.1.10]
at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.bulkUpdate(ElasticsearchTemplate.java:173) ~[spring-data-elasticsearch-5.1.10.jar:5.1.10]
at com.cisco.ccrc.ccrcdatapipeline.elastic.ElasticClient.updateAllByScript(ElasticClient.java:308) ~[classes/:?]
at com.cisco.ccrc.ccrcdatapipeline.controller.PingController.pingRes(PingController.java:45) ~[classes/:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-6.0.18.jar:6.0.18]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:798) ~[spring-webmvc-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081) ~[spring-webmvc-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974) ~[spring-webmvc-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.0.18.jar:6.0.18]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.0.18.jar:6.0.18]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.19.jar:6.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.0.18.jar:6.0.18]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.19.jar:6.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.AbstractRequestLoggingFilter.doFilterInternal(AbstractRequestLoggingFilter.java:289) ~[spring-web-6.0.18.jar:6.0.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.18.jar:6.0.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.0.18.jar:6.0.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.18.jar:6.0.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.0.18.jar:6.0.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.18.jar:6.0.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109) ~[spring-web-6.0.18.jar:6.0.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.18.jar:6.0.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.0.18.jar:6.0.18]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.18.jar:6.0.18]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) ~[tomcat-embed-core-10.1.19.jar:10.1.19]
at java.lang.Thread.run(Thread.java:842) ~[?:?]

[sothawo]

As I wrote:

Please provide a minimal compilable and runnable example that reproduces this error

I do not have the time to create a project to add this code to to be able to reproduce or debug this error.

And notice that the version you use (5.1.x) is outdated and not maintained anymore. So please first check if this error still occurs with the current 5.3 release of Spring Data Elasticsearch.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.