Add BulkRequest support to High Level Rest client #23312

tlrx · 2017-02-22T20:16:30Z

This commit adds support for BulkRequest execution in the High Level Rest client.

javanna

left a few comments

javanna · 2017-02-22T20:20:40Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+        parameters.withRefreshPolicy(bulkRequest.getRefreshPolicy());
+
+        // Bulk API only supports newline delimited JSON
+        XContentType xContentType = XContentType.JSON;


it also supports smile. should we validate that the format of all the index/update requests is the same and use that one? I wouldn't want to make things too complicated though given that most likely nobody uses smile here. People may also theoretically be using cbor or yaml in the single index requests now using the index api, as they don't have to go through REST.

javanna · 2017-02-22T20:25:37Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+            if (opType == DocWriteRequest.OpType.INDEX || opType == DocWriteRequest.OpType.CREATE) {
+                IndexRequest indexRequest = (IndexRequest) request;
+                BytesReference indexSource = indexRequest.source();
+                XContentType indexXContentType = indexRequest.getContentType();


interesting idea, to convert all the index requests to the target format, that way we can choose to always use either json or smile, without validating anything. Maybe that's better than what I suggested above. Although maybe tricky cause if a user sends cbor or smile, maybe they really expect cbor or smile to be stored in lucene rather than the json obtained from the conversion, but that is not possible when going through REST. I am on the fence. Maybe we should rather do some validation before we run the request, make sure that all of the requests have the same content-type and that can only be either json or smile. That way if something is not expected users are notified with an error. Thoughts?

I always forgot about Smile... I think it makes sense to check that all requests have the same content type and that it's either JSON or Smile. I updated the code and added tests for that, the first content type found dictates what's expected for the other requests.

javanna · 2017-02-22T20:46:48Z

client/rest-high-level/src/test/java/org/elasticsearch/client/CrudIT.java

+                assertEquals(errors[i] ? RestStatus.INTERNAL_SERVER_ERROR : RestStatus.CREATED, bulkItemResponse.status());
+            } else if (requestOpType == DocWriteRequest.OpType.UPDATE) {
+                assertEquals(errors[i], bulkItemResponse.isFailed());
+                assertEquals(errors[i] ? RestStatus.INTERNAL_SERVER_ERROR : RestStatus.OK, bulkItemResponse.status());


are these internal server errors ok? this seems fishy.

They are ElasticsearchException and ElasticsearchException almost always return INTERNAL_SERVER_ERROR as the status.

They come from shard info failures for the bulk item request, but the type is lost because the bulk response (and associated bulk item responses) are parsed back by the high level rest client.

I checked the status, I'm not sure that the test should check more than that.

ok let's try and fix this separately.

tlrx · 2017-02-23T10:22:40Z

@javanna Round 2, ready

javanna

thanks @tlrx left a couple more comments. it's close though.

javanna · 2017-02-23T10:46:49Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+                if (bulkContentType == null) {
+                    bulkContentType = requestContentType;
+                } else {
+                    match = (requestContentType == bulkContentType);


I wish we could throw exception here already rather than setting a flag. maybe add a static method that throws the exception so we don't worry about copying the message around?

javanna · 2017-02-23T10:48:03Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+        // Bulk API only supports newline delimited JSON or Smile. Before executing
+        // the bulk, we need to check that all requests have the same content-type
+        // and this content-type is supported by the Bulk API.
+        List<XContentType> allowedContentTypes = Arrays.asList(XContentType.JSON, XContentType.SMILE);


add a short method that checks the content type and get rid of the list? If this class becomes too big I am good with getting the bulk part out. some of this methods are going to be useful for msearch too at some point.

javanna · 2017-02-23T10:52:38Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+                        bulkContentType = requestContentType;
+                    } else {
+                        match = (requestContentType == bulkContentType);
+                    }


this code block is repeated and always deals with index requests I think. We can probably factor it out to a method.

tlrx · 2017-02-23T12:46:26Z

@javanna Thanks again. Would you like to have another look, please?

javanna

left a few minors, LGTM otherwise

javanna · 2017-02-23T13:05:44Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+            return requestContentType;
+        }
+        if (requestContentType != xContentType) {
+            throw new IllegalStateException("Mismatching content-type found for request with content-type [" + requestContentType


maybe we should throw illegal argument rather than illegal state.

javanna · 2017-02-23T13:06:48Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+
+    /**
+     * Ensure that the {@link IndexRequest}'s content type is supported by the Bulk API and that it conforms
+     * to the current {@link BulkRequest}'s content type (if it's known at the time of this method get called).


mention the return type in the docs?

javanna · 2017-02-23T13:09:25Z

client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java

+     * Ensure that the {@link IndexRequest}'s content type is supported by the Bulk API and that it conforms
+     * to the current {@link BulkRequest}'s content type (if it's known at the time of this method get called).
+     */
+    static XContentType ensureBulkContentType(IndexRequest indexRequest, @Nullable XContentType xContentType) {


enforceSameContentType or enforceContentType ?

javanna · 2017-02-23T13:23:41Z

client/rest-high-level/src/test/java/org/elasticsearch/client/CrudIT.java

+                assertEquals(errors[i] ? RestStatus.INTERNAL_SERVER_ERROR : RestStatus.CREATED, bulkItemResponse.status());
+            } else if (requestOpType == DocWriteRequest.OpType.UPDATE) {
+                assertEquals(errors[i], bulkItemResponse.isFailed());
+                assertEquals(errors[i] ? RestStatus.INTERNAL_SERVER_ERROR : RestStatus.OK, bulkItemResponse.status());


ok let's try and fix this separately.

This commit adds support for BulkRequest execution in the High Level Rest client.

tlrx · 2017-02-23T15:37:37Z

Thanks @javanna !

* master: (54 commits) Keep the pipeline handler queue small initially Do not create String instances in 'Strings' methods accepting StringBuilder (elastic#22907) Tests: fix AwsS3ServiceImplTests Remove abstract InternalMetricsAggregation class (elastic#23326) Add BulkRequest support to High Level Rest client (elastic#23312) Wrap getCredentials() in a doPrivileged() block (elastic#23297) Respect promises on pipelined responses Align REST specs for HEAD requests Remove unnecessary result sorting in SearchPhaseController (elastic#23321) Fix SamplerAggregatorTests to have stable and predictable docIds Tests: Ensure multi node integ tests wait on first node Relocate a comment in HttpPipeliningHandler Add comments to HttpPipeliningHandler [TEST] Fix incorrect test cluster name in cluster health doc tests Build: Change location in zip of license and notice inclusion for plugins (elastic#23316) Script: Fix value of `ctx._now` to be current epoch time in milliseconds (elastic#23175) Build: Rework integ test setup and shutdown to ensure stop runs when desired (elastic#23304) Handle long overflow when adding paths' totals Don't set local node on cluster state used for node join validation (elastic#23311) Ensure that releasing listener is called ...

tlrx added :Java High Level REST Client >enhancement v6.0.0-alpha1 labels Feb 22, 2017

tlrx requested review from javanna and cbuescher February 22, 2017 20:16

javanna requested changes Feb 22, 2017

View reviewed changes

javanna requested changes Feb 23, 2017

View reviewed changes

javanna approved these changes Feb 23, 2017

View reviewed changes

tlrx added 4 commits February 23, 2017 16:33

Add BulkRequest support to High Level Rest client

3104453

This commit adds support for BulkRequest execution in the High Level Rest client.

Update after Luca review

121553e

Update after Luca (2)

bce46bf

Rename method and use IllegalArgumentException instead

d823ca7

tlrx force-pushed the add-bulk-request-to-high-level-rest-client branch from 2e2dd2b to d823ca7 Compare February 23, 2017 15:35

tlrx merged commit 7e3c06c into elastic:master Feb 23, 2017

tlrx deleted the add-bulk-request-to-high-level-rest-client branch February 23, 2017 15:37

javanna mentioned this pull request Jun 6, 2017

Backport high level REST client #25086

Merged

javanna mentioned this pull request Nov 1, 2017

Java high-level REST client completeness #27205

Closed

80 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BulkRequest support to High Level Rest client #23312

Add BulkRequest support to High Level Rest client #23312

tlrx commented Feb 22, 2017

javanna left a comment

javanna Feb 22, 2017

javanna Feb 22, 2017

tlrx Feb 23, 2017

javanna Feb 22, 2017

tlrx Feb 23, 2017

javanna Feb 23, 2017

tlrx commented Feb 23, 2017

javanna left a comment

javanna Feb 23, 2017

javanna Feb 23, 2017

javanna Feb 23, 2017

tlrx commented Feb 23, 2017

javanna left a comment

javanna Feb 23, 2017

javanna Feb 23, 2017

javanna Feb 23, 2017

javanna Feb 23, 2017

tlrx commented Feb 23, 2017

Add BulkRequest support to High Level Rest client #23312

Add BulkRequest support to High Level Rest client #23312

Conversation

tlrx commented Feb 22, 2017

javanna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlrx commented Feb 23, 2017

javanna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlrx commented Feb 23, 2017

javanna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tlrx commented Feb 23, 2017