Add multi get api to the high level rest client #27337

martijnvg · 2017-11-10T06:22:44Z

Relates to #27205

vshank77 · 2017-11-14T17:24:27Z

Can we contribute to this one?

javanna · 2017-11-14T19:23:00Z

hi @vshank77 this is almost done already, it only needs review, which I am going to do in the coming days. Maybe you can pick another API from the list in #27205 ?

javanna

thanks a lot @martijnvg sorry it took me ages to review this. I left some comments and also asked @cbuescher to have a look too.

javanna · 2017-12-01T19:26:56Z

client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java

@@ -283,6 +285,25 @@ public void getAsync(GetRequest getRequest, ActionListener<GetResponse> listener
        performRequestAsyncAndParseEntity(getRequest, Request::get, GetResponse::fromXContent, listener, singleton(404), headers);
    }

+    /**
+     * Retrieves multi documents by id using the Multi Get API


s/multi/multiple ?

javanna · 2017-12-01T19:27:02Z

client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java

+    }
+
+    /**
+     * Asynchronously retrieves multi documents by id using the Multi Get API


s/multi/multiple ?

javanna · 2017-12-01T19:31:33Z

client/rest-high-level/src/test/java/org/elasticsearch/client/RequestTests.java

+            }
+        }
+
+        int numberOfRequests = randomIntBetween(0, 32);


what happens when we add no items?

In this test it is fine, but on the server side we do fail if no items have been specified (see MultiGetRequest#validate(...)).

javanna · 2017-12-01T19:33:49Z

client/rest-high-level/src/test/java/org/elasticsearch/client/RequestTests.java

+                } else {
+                    fetchSourceContext = new FetchSourceContext(false);
+                }
+                item.fetchSourceContext(fetchSourceContext);


maybe you can use randomizeFetchSourceContextParams ?

javanna · 2017-12-01T19:38:24Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+                        break;
+                    case START_ARRAY:
+                        if (Fields.DOCS.equals(currentFieldName) == false) {
+                            throw new ElasticsearchParseException("Unexpected field [" + currentFieldName + "]");


I don't think that throwing exceptions here is a good idea. We should rather be lenient to ensure forward compatibility. Say we add another array at the same level of docs (not likely, but still...) we should not throw error if your own client receives such a response. Rather ignore it.

javanna · 2017-12-01T19:40:11Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+                                    items.add(new MultiGetItemResponse(getResponse, null));
+                                }
+                            }  else {
+                                throw new ElasticsearchParseException("Unexpected token [" + innerToken + "]");


here too, throwing is a problem.

javanna · 2017-12-01T19:48:17Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+                                // then creating a parser from that is a bit inefficient, but makes the parsing code less
+                                // complex. I think this is the right trade off. Unless we introduce a new xcontent format
+                                // in mget response for the hl rest client that is enabled via a parameter that renders
+                                // an unambiguous format.


It makes me a bit sad to have to go with maps here, yet I see why it is hard to do it differently.
@cbuescher any idea on making this work without maps? Here is an example response for clarify:

{ "docs" : [ { "_index" : "my_index", "_type" : "my_type", "_id" : "1", "_version" : 1, "found" : true, "_source" : { "user" : [ { "field1" : 1, "field2" : 2 }, { "field1" : 3, "field2" : 4 } ] } }, { "_index" : "test", "_type" : "type", "_id" : "2", "error" : { "root_cause" : [ { "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_expression", "resource.id" : "test", "index_uuid" : "_na_", "index" : "test" } ], "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_expression", "resource.id" : "test", "index_uuid" : "_na_", "index" : "test" } } ] }

As far as I understand after a first glace the difference here is that the objects in docs can eithe be failures or sth. that can already be parsed by GetResult#fromXContent(). The only thing the failure items have is an error field. Ideally we could reuse the GetResult parser so it optionally accepts an error field. Unfortunaltely the current GetResult parser isn't easily extendable, but could we add a flag to it that allows optionally parsing the error field and have GetResult contain an optional Failure?
Another option would be to rewrite GetResult#fromXContent to use Object parser, then we could have a common "declareFields(Parser ...)" method that sets up the parser with the parts that GetResult needs and reuse that setup method plus a parser for the "error" field for a specialized MultiGetResult or something.
I think that way we could avoid parsing to a generic map here, which I would also appreciate.

@javanna @cbuescher I'll look into changing the GetResult#fromXContent(...) method, so that we don't have to parse into a generic map here.

javanna · 2017-12-01T19:49:14Z

core/src/main/java/org/elasticsearch/index/VersionType.java

+            case EXTERNAL_GTE:
+                return "external_gte";
+            case FORCE:
+                return "force";


can't we just do versionType.toLowercase(Locale.ROOT) ?

javanna · 2017-12-01T19:50:11Z

core/src/test/java/org/elasticsearch/action/get/MultiGetResponseTests.java

+            MultiGetResponse expected = createTestInstance();
+            XContentType xContentType = randomFrom(XContentType.values());
+            BytesReference shuffled = toShuffledXContent(expected, xContentType, ToXContent.EMPTY_PARAMS, false);
+


I think that inserting random fields here would reveal problems on the parsing side with the current code.

So I tried inserting random fields, but then the tests fails, because in GetResult#fromXContentEmbedded(...) in line 294 adds any unknow json field as field to retrieve, this causes the expected response item to be not equal with the actial response item.

javanna · 2017-12-01T19:51:24Z

test/framework/src/main/java/org/elasticsearch/test/ESTestCase.java

+            } else {
+                throw new IllegalArgumentException("unsupported token [" + token + "]");
+            }
+            return xContentBuilder;


ops, I wonder why we are finding this only now

I added this because otherwise FetchSourceContextTests test fails. I guess we never shuffled xcontent that had only a top level value (which is valid).

I still wonder why this would be a problem. At least as far as I understand this method should only take whole xContent Objects or Arrays, and they should all parse to a map fine (even with just one field). Or are you talking about things like { "foo" }. Is that valid at all? I would like to take a look at the failing test before adding this here, I think this method should rather check that the start token is either Array- or Object-Start and otherwise reject.

Or are you talking about things like { "foo" }. Is that valid at all?

Yes, that is valid.

I noticed that we didn't have a isolated unit test for FetchSourceContext. This class serializes a bool value directly without adding an enclosing json object. That is because fetch source context is part of a search request under a json field. This causes FetchSourceContextTests to fail, because shuffleXContent(...) method returned an empty XContentBuilder. This made me change the shuffleXContent(...) method.

I think I can add an assert here to check whether a value is top level and otherwise fail?

I took a look at FetchSourceContext and I think its toXContent() method is kind of problematic in a sense that it sometimes renders a complete object (with start/end Object) and sometimes only a value. Given that FetchSourceContext extends ToXContentObject, I think the later is a case that shouldn't happen. Should we move this out of this PR or is FetchSourceContextTests needed? I don't think I would like to special-case the shuffle methods for this.

No, this test is not needed for adding mget api to high level rest client. So 👍 to move this test out and its related changes out of this pr.

cbuescher

@martijnvg @javanna I took a first look regarding the parsing to map and left a first idea, I will also take a look at the rest of the PR later today.

Relates to elastic#27337

martijnvg · 2017-12-08T11:27:04Z

@javanna @cbuescher I've changed the MultiGetResponse#fromXContent(...) method to be completely streamable and not use a map to temporarily store all the key/value pairs for each response item.

cbuescher

@martijnvg thanks, I did another round of reviews, I still need to take a short deeper look at the tests and will add those comments later. But the rest looks good already to me minus a few small things.

cbuescher · 2017-12-12T11:18:36Z

core/src/main/java/org/elasticsearch/action/get/MultiGetRequest.java

@@ -123,6 +129,11 @@ public String id() {
            return this.id;
        }

+        public Item id(String id) {


Where is this setter needed? I experimentally removed it and everything seems fine, in which case I would like to remove it.

Yes, this can be removed. This was a left over from the parser.

cbuescher · 2017-12-12T11:26:52Z

core/src/main/java/org/elasticsearch/action/get/MultiGetRequest.java

@@ -371,9 +403,9 @@ public MultiGetRequest add(@Nullable String defaultIndex, @Nullable String defau

    public static void parseDocuments(XContentParser parser, List<Item> items, @Nullable String defaultIndex, @Nullable String defaultType, @Nullable String[] defaultFields, @Nullable FetchSourceContext defaultFetchSource, @Nullable String defaultRouting, boolean allowExplicitIndex) throws IOException {


Can this be private? Seems only be used from "add" and the other parseDocuments() method.

cbuescher · 2017-12-12T11:28:25Z

core/src/main/java/org/elasticsearch/action/get/MultiGetRequest.java

@@ -493,8 +525,8 @@ public static void parseDocuments(XContentParser parser, List<Item> items) throw
    }


Where do we need public static void parseDocuments(XContentParser parser, List<Item> items) in the code? I think we can remove this and only leave "add(...)" as an entry point to parsing.

The add(...) method is pretty large already, I like not to make it larger than it already is by adding the that is now in parseDocuments(...) to it.

cbuescher · 2017-12-12T11:31:42Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java


 public class MultiGetResponse extends ActionResponse implements Iterable<MultiGetItemResponse>, ToXContentObject {

+    private static final ParseField INDEX = new ParseField(Fields._INDEX);


nit: can we remove the Fields constants and use ParseField.getPreferredName() where we currently use the Strings (like in toXContent())

cbuescher · 2017-12-12T11:35:24Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+                                String id = null;
+                                ElasticsearchException exception = null;
+                                GetResult getResult = null;
+                                item: for (token = parser.nextToken(); token != Token.END_OBJECT; token = parser.nextToken()) {


Interesting, haven't seen breaks with targets in a while. This might be a bit too sneaky, I think I would like pulling out the item parser for better readability.

cbuescher · 2017-12-12T11:36:12Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+        String currentFieldName = null;
+        List<MultiGetItemResponse> items = new ArrayList<>();
+        for (Token token = parser.nextToken(); token != Token.END_OBJECT; token = parser.nextToken()) {
+            switch (token) {


Does this switch have a default case?

No, there is no default case. Normally I would add a default case that throws an exception, but we should be lenient here.

I see, can you still add an empty default case, maybe with a comment that we ignore those cases? I get a warning here that could be avoided.

cbuescher · 2017-12-12T11:36:56Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+                                ElasticsearchException exception = null;
+                                GetResult getResult = null;
+                                item: for (token = parser.nextToken(); token != Token.END_OBJECT; token = parser.nextToken()) {
+                                    switch (token) {


Same here, maybe have a default case? Not sure if it needs to do anything (I think parsing should be lenient for the response parsing)

No, there is no default case. Normally I would add a default case that throws an exception, but we should be lenient here.

cbuescher · 2017-12-12T13:52:58Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+                                            break;
+                                        case START_OBJECT:
+                                            if (ERROR.match(currentFieldName)) {
+                                                exception = ElasticsearchException.fromXContent(parser);


I'm wondering if it would be possible/worthwhile to augment GetResult (and its parsing method) with an optional exception field and then only use GetResult.fromXContentEmbedded() for both cases. That would add something to GetResult that is unused in the normal Get-API, but it would simplify things here. I'm on the fence about this idea, maybe @javanna can comment on this too.

martijnvg · 2017-12-15T14:59:13Z

@cbuescher Thanks for the review! I've updated the PR.

cbuescher

@martijnvg Hi, thanks for the updates, I left some super minor nits but other than that looks good to me. I don't know if @javanna wants to give this another look though.

cbuescher · 2017-12-18T11:27:22Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+        String id = null;
+        ElasticsearchException exception = null;
+        GetResult getResult = null;
+        item: for (Token token = parser.nextToken(); token != Token.END_OBJECT; token = parser.nextToken()) {


nit: no need for the loop label anymore

cbuescher · 2017-12-18T11:28:47Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+        String currentFieldName = null;
+        List<MultiGetItemResponse> items = new ArrayList<>();
+        for (Token token = parser.nextToken(); token != Token.END_OBJECT; token = parser.nextToken()) {
+            switch (token) {


I see, can you still add an empty default case, maybe with a comment that we ignore those cases? I get a warning here that could be avoided.

cbuescher · 2017-12-18T11:29:16Z

core/src/main/java/org/elasticsearch/action/get/MultiGetResponse.java

+        ElasticsearchException exception = null;
+        GetResult getResult = null;
+        item: for (Token token = parser.nextToken(); token != Token.END_OBJECT; token = parser.nextToken()) {
+            switch (token) {


nit: maybe empty default wt. comment here as well.

javanna · 2018-01-16T14:04:04Z

no need to wait for my final LGTM here, I trust you @cbuescher @martijnvg ! please merge it in, although it has conflicts now unfortunately.

Relates to elastic#27205

shabtaisharon · 2018-04-12T18:49:34Z

Is this in a released version? if yes, can you please point me to the documentation?

cbuescher · 2018-04-12T18:59:18Z

@shabtaisharon according to the labels on this issue this has been merged to 6.2. However, it seems the documentation was only added in 6.x so far: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/6.x/java-rest-high-document-multi-get.html, but I think that should reflect what is there in 6.2 already.

shabtaisharon · 2018-04-12T19:13:15Z

@cbuescher thank you!

martijnvg added :Java High Level REST Client >enhancement v6.1.0 v7.0.0 labels Nov 10, 2017

martijnvg requested a review from javanna November 10, 2017 06:22

javanna mentioned this pull request Nov 10, 2017

Java high-level REST client completeness #27205

Closed

80 tasks

javanna self-assigned this Nov 13, 2017

martijnvg added v6.2.0 and removed v6.1.0 labels Nov 22, 2017

javanna requested changes Dec 1, 2017

View reviewed changes

cbuescher self-requested a review December 1, 2017 20:09

cbuescher reviewed Dec 4, 2017

View reviewed changes

martijnvg force-pushed the hl_client_mget_api branch 2 times, most recently from 17484eb to a21b3ec Compare December 5, 2017 09:27

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Dec 7, 2017

Added test for FetchSourceContext

32b0bc9

Relates to elastic#27337

cbuescher reviewed Dec 12, 2017

View reviewed changes

martijnvg force-pushed the hl_client_mget_api branch from 2ae4a63 to 67bebfa Compare December 15, 2017 14:44

cbuescher approved these changes Dec 18, 2017

View reviewed changes

javanna approved these changes Jan 16, 2018

View reviewed changes

martijnvg force-pushed the hl_client_mget_api branch 2 times, most recently from ec3a2ee to 2b8c1d6 Compare January 16, 2018 14:18

Added multi get api to the high level rest client.

853f7e8

Relates to elastic#27205

martijnvg force-pushed the hl_client_mget_api branch from 2b8c1d6 to 853f7e8 Compare January 16, 2018 16:27

martijnvg merged commit 853f7e8 into elastic:master Jan 16, 2018

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

		@@ -371,9 +403,9 @@ public MultiGetRequest add(@Nullable String defaultIndex, @Nullable String defau

		public static void parseDocuments(XContentParser parser, List<Item> items, @Nullable String defaultIndex, @Nullable String defaultType, @Nullable String[] defaultFields, @Nullable FetchSourceContext defaultFetchSource, @Nullable String defaultRouting, boolean allowExplicitIndex) throws IOException {

		@@ -493,8 +525,8 @@ public static void parseDocuments(XContentParser parser, List<Item> items) throw
		}


		public class MultiGetResponse extends ActionResponse implements Iterable<MultiGetItemResponse>, ToXContentObject {

		private static final ParseField INDEX = new ParseField(Fields._INDEX);

Add multi get api to the high level rest client #27337

Add multi get api to the high level rest client #27337

Conversation

martijnvg commented Nov 10, 2017

vshank77 commented Nov 14, 2017

javanna commented Nov 14, 2017 • edited

javanna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbuescher left a comment

Choose a reason for hiding this comment

martijnvg commented Dec 8, 2017

cbuescher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martijnvg commented Dec 15, 2017

cbuescher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javanna commented Jan 16, 2018

shabtaisharon commented Apr 12, 2018

cbuescher commented Apr 12, 2018

shabtaisharon commented Apr 12, 2018

javanna commented Nov 14, 2017 •

edited