REST high-level client: add reindex API #32679

sohaibiftikhar · 2018-08-07T12:43:41Z

Relates to #27205

sohaibiftikhar · 2018-08-07T12:48:45Z

client/rest-high-level/src/test/java/org/elasticsearch/client/CrudIT.java

+            );
+        }
+        {
+            TimeUnit.SECONDS.sleep(1);


@nik9000 I know this is incorrect. But I couldn't make the test pass without it. Will firing a refresh request help?

It worked. Sorry for the noise.

nik9000 · 2018-08-07T13:41:52Z

@elasticmachine, white list this please

nik9000 · 2018-08-07T13:42:24Z

whitelist this please
open sesame

javanna · 2018-08-07T13:45:20Z

add to whitelist

LOL

elasticmachine · 2018-08-07T16:35:59Z

Pinging @elastic/es-core-infra

nik9000

I left a few questions, the biggest being, maybe a mutable builder object that knows how to build a status or the error might make this less terrible to parse. Maybe. Could you check?

nik9000 · 2018-08-07T21:01:21Z

...est-high-level/src/test/java/org/elasticsearch/client/documentation/CRUDDocumentationIT.java

+            request.setScript(
+                new Script(
+                    ScriptType.INLINE, "painless",
+                    "if (ctx._source.foo == 'bar') {ctx._version++; ctx._source.remove('foo')}",


I think messing with the _version is a fairly rare thing. I think It'd be more normal to, say, add split a field on a regex and stick it into two fields. Or something like that.

Ah, I copied this from the other doc. It is to show (I believe) that you can mess with it during reindex but not during update_by_query. I can change it.

I think it is worth changing. Probably in both places to be honest, but just in this place in this PR.

nik9000 · 2018-08-07T21:03:02Z

docs/java-rest/high-level/document/reindex.asciidoc

+--------------------------------------------------
+<1> Set the versionType to `EXTERNAL`
+
+Settings `opType` to `create` will cause `_reindex` to only create missing documents in the target index. All existing


/Settings/Setting/

nik9000 · 2018-08-07T21:05:30Z

docs/java-rest/high-level/document/reindex.asciidoc

+<1> `setScript` to bump the version of the source document
+
+`ReindexRequest` supports reindexing from a remote Elasticsearch cluster. When using a remote cluster the query should be
+specified inside the `RemoteInfo` object and not using `setSourceQuery`.


I'd explicitly say that the query set by setSourceQuery is ignored. It might also be nice to explain why: the remote Elasticsearch may not understand queries built by the modern query builders. This works all the way back to Elasticsearch 0.90 and the query language has drifted a bit since then. When you reach to old versions, it is better to write the query by hand in json.

nik9000 · 2018-08-07T21:08:34Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollResponse.java

+                List<Failure> bulkFailures = new ArrayList<>();
+                List<SearchFailure> searchFailures = new ArrayList<>();
+                for (Object object: failures) {
+                    if (object instanceof Failure) {


I regret building this in this way. It is indeed messy.

nik9000 · 2018-08-07T21:09:48Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollResponse.java

+                   case SearchFailure.SHARD_FIELD:
+                       shardId = parser.intValue();
+                       break;
+                   default:


nik9000 · 2018-08-07T21:12:47Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollTask.java

+            Float requestsPerSecond = (Float) a[startIndex + 10];
+            requestsPerSecond = requestsPerSecond == -1 ? Float.POSITIVE_INFINITY : requestsPerSecond;
+            String reasonCancelled = (String) a[startIndex + 11];
+            TimeValue throttledUntil =


I'd probably extract them all from the array first, casting them, and then manipulating them.

Actually, I wonder if a builder object would be better than this object array stuff. It might be cleaner. Not nice, but the way I laid out the json makes nice kind of impossible.

I see what you mean. But it might not be super helpful. There are two constructors. The way it works right now is you call one or the other. So you calculate stuff from sliceStatuses or you specify everything and sliceStatuses is an empty list. Not sure how that would translate here. Also, personally I like immutable objects. Validation is consolidated in the constructor. For example, fields that must be set. Plus it wouldn't change much in terms of lines of code I think.
But if you think it's important I can change it.

By mutable builder I mean making a mutable object that you just use as the target for the parsing logic and then throw away afterwords. It just has setters and then a method to build status or error depending on which fields are set. The mutable object never escapes parsing. It might not be less code but I think it'd be easier to understand.

Ah, I see. Okay, thanks for explaining! I can make that change.

nik9000 · 2018-08-07T21:26:56Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollTask.java

+        }
+
+        public static Status innerFromXContent(XContentParser parser) throws IOException {
+            Token token = parser.currentToken();


I wonder if you can avoid all of this with a builder object that knows how to build either status or error. Then you can use regular ObjectParser with it.

I believe I cannot parse the error myself (as we already discussed)? I need to peek at a field name to realise what I need to parse which doesn't work well with an ObjectParser. I could declare all fields in the ObjectParser and then try to build an object using that but for that, I would need to write the parse for the error as well. Which I really do not want to do... Or are you suggesting something else?

Could you add a comment here explaining why you need to parse this manually? I'm 100% sure when I read this again in 6 months I'll waste an hour figuring out why you did it this way. A comment will save future me some time.

nik9000

This looks good to me! I left about 7 requests for more comments, mostly because I know that I'll read this again in six months and won't know why you made the choices that you did. Comments explaining those choices would be super nice. And some Javadoc on the new public methods would be wonderful!

nik9000 · 2018-08-09T18:42:59Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollTask.java

@@ -189,6 +202,115 @@ public boolean shouldCancelChildrenOnCancellation() {
        return true;
    }

+    public static class StatusBuilder {


Could you add javadoc for this?

nik9000 · 2018-08-09T18:43:20Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollResponseBuilder.java

+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+public class BulkByScrollResponseBuilder extends StatusBuilder {


Could you add javadoc for this too?

Also, should it be public? What about package scope?

nik9000 · 2018-08-09T18:44:42Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollTask.java

+        }
+
+        public static Status innerFromXContent(XContentParser parser) throws IOException {
+            Token token = parser.currentToken();


Could you add a comment here explaining why you need to parse this manually? I'm 100% sure when I read this again in 6 months I'll waste an hour figuring out why you did it this way. A comment will save future me some time.

nik9000 · 2018-08-09T18:46:07Z

server/src/main/java/org/elasticsearch/index/reindex/BulkByScrollTask.java

@@ -610,6 +961,41 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
            return builder;
        }

+        public static StatusOrException fromXContent(XContentParser parser) throws IOException {


Can you leave a comment about why you do it this way? The reasoning is fairly involved and a comment will totally help future-me when I reread this six months from now.

nik9000 · 2018-08-09T18:46:40Z

server/src/main/java/org/elasticsearch/index/reindex/ReindexRequest.java

@@ -39,7 +44,8 @@
 * of reasons, not least of which that scripts are allowed to change the destination request in drastic ways, including changing the index
 * to which documents are written.
 */
-public class ReindexRequest extends AbstractBulkIndexByScrollRequest<ReindexRequest> implements CompositeIndicesRequest {
+public class ReindexRequest extends AbstractBulkIndexByScrollRequest<ReindexRequest>
+    implements CompositeIndicesRequest, ToXContentObject {


Could you indent this one more level? I think this kind of indenting makes it look like implements is part of the class body.

nik9000 · 2018-08-09T18:47:10Z

server/src/main/java/org/elasticsearch/index/reindex/ReindexRequest.java

+        return this;
+    }
+
+    public ReindexRequest setDestIndex(String destIndex) {


Could you add javadoc for these? They'd be nice because these are part of the public API.

I know we weren't good about this in the past but we're trying to be better lately.

nik9000

I merged master and pushed a tiny cleanup. I'm going to let CI chew on it.

* master: Painless: Add Bindings (#33042) Update version after client credentials backport Fix forbidden apis on FIPS (#33202) Remote 6.x transport BWC Layer for `_shrink` (#33236) Test fix - Graph HLRC tests needed another field adding to randomisation exception list HLRC: Add ML Get Records API (#33085) [ML] Fix character set finder bug with unencodable charsets (#33234) TESTS: Fix overly long lines (#33240) Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic Remove unsupported group_shard_failures parameter (#33208) Update BucketUtils#suggestShardSideQueueSize signature (#33210) Parse PEM Key files leniantly (#33173) INGEST: Add Pipeline Processor (#32473) Core: Add java time xcontent serializers (#33120) Consider multi release jars when running third party audit (#33206) Update MSI documentation (#31950) HLRC: create base timed request class (#33216) [DOCS] Fixes command page titles HLRC: Move ML protocol classes into client ml package (#33203) Scroll queries asking for rescore are considered invalid (#32918) Painless: Fix Semicolon Regression (#33212) ingest: minor - update test to include dissect (#33211) Switch remaining LLREST usage to new style Requests (#33171) HLREST: add reindex API (#32679)

Adds the reindex API to the high level REST client.

* 6.x: HLRC: ML Flush job (#33187) Switch more LLREST usage to new style Requests (#33171) HLRC: Adding ML Job stats (#33183) HLREST: add reindex API (#32679) Mute testSyncerOnClosingShard [DOCS] Moves machine learning APIs to docs folder (#31118)

sohaibiftikhar commented Aug 7, 2018

View reviewed changes

sohaibiftikhar force-pushed the reindex branch from d0739e2 to c7ef420 Compare August 7, 2018 13:23

dliappis added the :Core/Java High Level REST Client label Aug 7, 2018

nik9000 reviewed Aug 7, 2018

View reviewed changes

nik9000 requested changes Aug 9, 2018

View reviewed changes

sohaibiftikhar mentioned this pull request Aug 9, 2018

REST high-level client: add update by query API #32760

Merged

1 task

sohaibiftikhar force-pushed the reindex branch 2 times, most recently from d3d7f9f to 34cb5ed Compare August 9, 2018 23:45

sohaibiftikhar added 6 commits August 10, 2018 15:52

added reindex API

8874e7d

Fixed CrudIT to not use sleep

7c09364

Added statusbuilder object

a42a419

Changed to ObjectParser from ConstructingObjectParser

40b108f

changed documentation

3346b40

Added javadoc

34cbed6

sohaibiftikhar force-pushed the reindex branch 2 times, most recently from 657e26e to 4490216 Compare August 10, 2018 14:39

Added options for request converters

4fa987c

sohaibiftikhar force-pushed the reindex branch from 4490216 to 4fa987c Compare August 10, 2018 14:50

sohaibiftikhar mentioned this pull request Aug 10, 2018

REST high-level client: add delete by query API #32782

Merged

2 tasks

Merge branch 'master' into pr/32679

9a8bbb8

nik9000 approved these changes Aug 24, 2018

View reviewed changes

nik9000 merged commit 7f5e29d into elastic:master Aug 28, 2018

nik9000 added >enhancement v7.0.0 backport pending v6.5.0 labels Aug 28, 2018

nik9000 removed the backport pending label Sep 1, 2018

nik9000 pushed a commit that referenced this pull request Sep 1, 2018

HLREST: add reindex API (#32679)

4dc899c

Adds the reindex API to the high level REST client.

sohaibiftikhar deleted the reindex branch September 5, 2018 08:33

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REST high-level client: add reindex API #32679

REST high-level client: add reindex API #32679

sohaibiftikhar commented Aug 7, 2018

sohaibiftikhar Aug 7, 2018

sohaibiftikhar Aug 7, 2018

nik9000 commented Aug 7, 2018

nik9000 commented Aug 7, 2018

javanna commented Aug 7, 2018

elasticmachine commented Aug 7, 2018

nik9000 left a comment

nik9000 Aug 7, 2018

sohaibiftikhar Aug 7, 2018

nik9000 Aug 7, 2018

nik9000 Aug 7, 2018

nik9000 Aug 7, 2018

nik9000 Aug 7, 2018

nik9000 Aug 7, 2018

nik9000 Aug 7, 2018

sohaibiftikhar Aug 7, 2018

nik9000 Aug 7, 2018

sohaibiftikhar Aug 7, 2018

nik9000 Aug 7, 2018

sohaibiftikhar Aug 7, 2018 •

edited

Loading

nik9000 Aug 9, 2018

nik9000 left a comment

nik9000 Aug 9, 2018

nik9000 Aug 9, 2018

nik9000 Aug 9, 2018

nik9000 Aug 9, 2018

nik9000 Aug 9, 2018

nik9000 Aug 9, 2018

nik9000 Aug 9, 2018

nik9000 left a comment

REST high-level client: add reindex API #32679

REST high-level client: add reindex API #32679

Conversation

sohaibiftikhar commented Aug 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Aug 7, 2018

nik9000 commented Aug 7, 2018

javanna commented Aug 7, 2018

elasticmachine commented Aug 7, 2018

nik9000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sohaibiftikhar Aug 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 left a comment

Choose a reason for hiding this comment

sohaibiftikhar Aug 7, 2018 •

edited

Loading