Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic reindex implementation #15125

Merged
merged 129 commits into from Jan 13, 2016
Merged
Changes from 123 commits
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
839bb21
Basic index-by-search
nik9000 Nov 25, 2015
e918bc1
Fix vagrant tests
nik9000 Nov 25, 2015
39e228e
Docs
nik9000 Nov 25, 2015
7501c00
Basic REST
nik9000 Nov 30, 2015
eee9583
Remove leftovers
nik9000 Nov 30, 2015
f4ec860
tabs!
nik9000 Nov 30, 2015
542c656
More docs
nik9000 Nov 30, 2015
d02f277
Copy routing
nik9000 Nov 30, 2015
e5ae49f
Add support for preserving external versions
nik9000 Nov 30, 2015
169e45e
Be more careful with the search request
nik9000 Dec 1, 2015
6db33aa
Clean up rest
nik9000 Dec 1, 2015
81430f9
More docs and tests
nik9000 Dec 1, 2015
f81e4e4
Fix more validation
nik9000 Dec 1, 2015
dd8fa09
Add support for routing
nik9000 Dec 1, 2015
222c986
Remove file that is no longer needed
nik9000 Dec 2, 2015
f9a3ae0
Add test for updating in place
nik9000 Dec 2, 2015
d40767e
Add support for op_type
nik9000 Dec 2, 2015
85ca3be
Move size override out of builder
nik9000 Dec 3, 2015
e562f35
Lots of version related things
nik9000 Dec 4, 2015
b96b218
More versioning work
nik9000 Dec 4, 2015
0be4b29
Preserve version by default reindexing
nik9000 Dec 4, 2015
6e7650c
Docs
nik9000 Dec 4, 2015
a3cc171
Start fixing response format
nik9000 Dec 4, 2015
6f3e563
Return failures
nik9000 Dec 7, 2015
f622e36
Rest tests for failures and batches
nik9000 Dec 7, 2015
630f26c
Squash index-by-search into on package
nik9000 Dec 7, 2015
79ee748
Add warning about atomicity to docs
nik9000 Dec 7, 2015
012336e
Rename version type and fix typo
nik9000 Dec 10, 2015
4e016fb
More rename version type
nik9000 Dec 10, 2015
3f67488
Simplify failure handling
nik9000 Dec 10, 2015
ee981a4
Start building ReindexInPlace
nik9000 Dec 10, 2015
c3a3db8
Fixup reindex and get tests passing
nik9000 Dec 10, 2015
f9d4e40
More reindex support
nik9000 Dec 11, 2015
e3e5d82
More reindex tests
nik9000 Dec 11, 2015
4b285ff
Hack to support body content
nik9000 Dec 11, 2015
04506e1
Remove parsers we can't use
nik9000 Dec 11, 2015
6f438b4
Add support for update scripts
nik9000 Dec 14, 2015
ab19a8a
Script support fixes
nik9000 Dec 14, 2015
4f8f14a
Get run picking up groovy
nik9000 Dec 14, 2015
7fc9f2d
_index_by_search's op_type parameter
nik9000 Dec 14, 2015
dee79c5
Revive update while indexing test
nik9000 Dec 14, 2015
0360bca
index-by-search can't write where it reads
nik9000 Dec 15, 2015
f86cd42
Rename index -> dest
nik9000 Dec 15, 2015
d6050d4
Add convenience destination methods to build
nik9000 Dec 15, 2015
8e5a3a6
More convenience methods in builder
nik9000 Dec 15, 2015
4bb66c2
Rename "search" to "source"
nik9000 Dec 16, 2015
df0ee51
Switch implementation to client
nik9000 Dec 16, 2015
3977010
Share script with reindex
nik9000 Dec 16, 2015
ae59719
Start adding metadata to scripts
nik9000 Dec 16, 2015
c2947a2
Add support for changing routing and parent
nik9000 Dec 16, 2015
5288788
Test changing parent and routing
nik9000 Dec 16, 2015
73075e6
Change definition of batches
nik9000 Dec 16, 2015
c4836ac
More cleanup in preparation for BulkProcessor
nik9000 Dec 16, 2015
865d694
Fix batches better this time
nik9000 Dec 16, 2015
1c68e61
Cleanup
nik9000 Dec 16, 2015
2b924eb
Cleanup
nik9000 Dec 16, 2015
5c9911e
Rename plugin to reindex
nik9000 Dec 17, 2015
a9cc567
Rename reindex->update-by-query
nik9000 Dec 17, 2015
4924475
Rename index-by-search->reindex
nik9000 Dec 17, 2015
b676ea8
Finish rename
nik9000 Dec 17, 2015
5b27eef
Drop version_type from update-by-query
nik9000 Dec 17, 2015
b113f0a
Rename docs
nik9000 Dec 17, 2015
82e2ead
Rewrite docs for the new world
nik9000 Dec 18, 2015
3459b28
Add more strict checks after scripts
nik9000 Dec 18, 2015
4140b00
Support reindex scripts changing metadata
nik9000 Dec 18, 2015
e4947f0
More renames
nik9000 Dec 18, 2015
c977217
Remove op_type from reindex
nik9000 Dec 21, 2015
14b7560
Fix funky layout in reindex source
nik9000 Dec 21, 2015
6504b40
docs
nik9000 Dec 21, 2015
e26d501
Add reindex parameter
nik9000 Dec 21, 2015
927b0d4
Consistency and timeout parameters
nik9000 Dec 21, 2015
aff9dfc
[docs] Add routing example
nik9000 Dec 21, 2015
3d26594
Fix some nocommits
nik9000 Dec 21, 2015
b1a2a96
Remove another nocommit
nik9000 Dec 21, 2015
36beb8d
Remove another nocommit
nik9000 Dec 21, 2015
96f9a86
Add round trip tests
nik9000 Dec 22, 2015
8759aed
Remove leftovers
nik9000 Dec 22, 2015
98dd95a
Revert new VersionType
nik9000 Dec 22, 2015
626e074
Revert test for new VersionType
nik9000 Dec 22, 2015
e396e6f
more docs
nik9000 Dec 22, 2015
7869a99
Move script tests into a qa module
nik9000 Dec 22, 2015
ba4e73e
Move building the bulk onto generic thread pool
nik9000 Dec 22, 2015
9c2b508
Fix out of date comments
nik9000 Dec 22, 2015
6107250
Return an error when the request fails
nik9000 Dec 23, 2015
12d7c80
Add missing license headers
nik9000 Dec 28, 2015
b5bce59
Docs fixes
nik9000 Dec 28, 2015
2e038ba
Implement BulkShardRequest.toString
nik9000 Dec 28, 2015
3c87503
Status names -> status codes
nik9000 Dec 28, 2015
cb98b2d
Cleanups from comments
nik9000 Dec 28, 2015
06de720
Use FieldMappers field names
nik9000 Dec 28, 2015
04d2912
Fixes from comments
nik9000 Dec 28, 2015
c4e7007
Replace -1 with a constant
nik9000 Dec 28, 2015
da38415
Refactor to remove duplication
nik9000 Dec 28, 2015
d43a968
Document source/src alias
nik9000 Dec 28, 2015
0461940
Document dest/destination alias
nik9000 Dec 28, 2015
4d4d93f
Updates from comments
nik9000 Dec 28, 2015
be501c7
unmodifiable list
nik9000 Dec 28, 2015
184123f
Let IOException bubble
nik9000 Dec 28, 2015
0c3f1b0
Rename constant
nik9000 Dec 28, 2015
23aaa61
Allow user to override scroll_size
nik9000 Dec 28, 2015
fe1d36d
More renames
nik9000 Dec 28, 2015
82c318a
Remove ?
nik9000 Dec 28, 2015
906f455
Add to the list of official plugins
nik9000 Dec 28, 2015
088a978
Changes from comments
nik9000 Dec 29, 2015
022cddb
OptionalStreamable!
nik9000 Jan 5, 2016
5c1e6da
Docs updates
nik9000 Jan 5, 2016
c0bc003
Help should include reindex
nik9000 Jan 5, 2016
e076531
Some fixes from review
nik9000 Jan 6, 2016
f4fd2f6
Remove duplicate check
nik9000 Jan 6, 2016
a73c252
Switch to getters in response
nik9000 Jan 6, 2016
384a0cb
Remove {}
nik9000 Jan 6, 2016
fa72146
Remove superfluous continue
nik9000 Jan 6, 2016
fb3d418
Validate timestamp and ttl aren't set
nik9000 Jan 6, 2016
9582876
Updates from comments
nik9000 Jan 7, 2016
354f7e0
Getters
nik9000 Jan 7, 2016
6831b4f
Tighter try
nik9000 Jan 7, 2016
6c295d2
Setters
nik9000 Jan 7, 2016
ffb6ed1
Remove useless continue
nik9000 Jan 7, 2016
d1363ef
Test that thread pool rejections are reported
nik9000 Jan 7, 2016
aac3ddd
Properly fail on shard failures
nik9000 Jan 7, 2016
1dc46e9
Start moving scripting tests to unit tests
nik9000 Jan 7, 2016
00947a8
Move script tests to unit tests
nik9000 Jan 11, 2016
8e49b35
Move alias checking testing into unit tests
nik9000 Jan 11, 2016
c0f3635
[docs] Make a list of metadata
nik9000 Jan 12, 2016
8e19e4f
[docs] Describe timeout
nik9000 Jan 12, 2016
6b85c37
[doc] Add more callouts
nik9000 Jan 12, 2016
e07a49b
Default conflicts to ignore
nik9000 Jan 12, 2016
ae8dbe2
Remove src and destination
nik9000 Jan 12, 2016
a8a3467
Move metadata tests to unit tests
nik9000 Jan 13, 2016
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -19,6 +19,7 @@

package org.elasticsearch.action.bulk;

import org.elasticsearch.ElasticsearchException;
import org.elasticsearch.ExceptionsHelper;
import org.elasticsearch.action.ActionWriteResponse;
import org.elasticsearch.action.delete.DeleteResponse;
@@ -27,6 +28,9 @@
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;
import org.elasticsearch.common.io.stream.Streamable;
import org.elasticsearch.common.io.stream.Writeable;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.rest.RestStatus;

import java.io.IOException;
@@ -40,7 +44,15 @@
/**
* Represents a failure.
*/
public static class Failure {
public static class Failure implements Writeable<Failure>, ToXContent {
static final String INDEX_FIELD = "index";
static final String TYPE_FIELD = "type";
static final String ID_FIELD = "id";
static final String CAUSE_FIELD = "cause";
static final String STATUS_FIELD = "status";

public static final Failure PROTOTYPE = new Failure(null, null, null, null);

private final String index;
private final String type;
private final String id;
@@ -90,9 +102,39 @@ public RestStatus getStatus() {
return this.status;
}

/**
* The actual cause of the failure.
*/
public Throwable getCause() {
return cause;
}

@Override
public Failure readFrom(StreamInput in) throws IOException {
return new Failure(in.readString(), in.readString(), in.readOptionalString(), in.readThrowable());
}

@Override
public void writeTo(StreamOutput out) throws IOException {
out.writeString(getIndex());
out.writeString(getType());
out.writeOptionalString(getId());
out.writeThrowable(getCause());
}

@Override
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
builder.field(INDEX_FIELD, index);
builder.field(TYPE_FIELD, type);
if (id != null) {
builder.field(ID_FIELD, id);
}
builder.startObject(CAUSE_FIELD);
ElasticsearchException.toXContent(builder, params, cause);
builder.endObject();
builder.field(STATUS_FIELD, status.getStatus());
return builder;
}
}

private int id;
@@ -257,11 +299,7 @@ public void readFrom(StreamInput in) throws IOException {
}

if (in.readBoolean()) {
String fIndex = in.readString();
String fType = in.readString();
String fId = in.readOptionalString();
Throwable throwable = in.readThrowable();
failure = new Failure(fIndex, fType, fId, throwable);
failure = Failure.PROTOTYPE.readFrom(in);
}
}

@@ -286,10 +324,7 @@ public void writeTo(StreamOutput out) throws IOException {
out.writeBoolean(false);
} else {
out.writeBoolean(true);
out.writeString(failure.getIndex());
out.writeString(failure.getType());
out.writeOptionalString(failure.getId());
out.writeThrowable(failure.getCause());
failure.writeTo(out);
}
}
}
@@ -56,6 +56,17 @@ boolean refresh() {
return items;
}

@Override
public String toString() {
// This is included in error messages so we'll try to make it somewhat user friendly.
StringBuilder b = new StringBuilder("BulkShardRequest to [");
b.append(index).append("] containing [").append(items.length).append("] requests");
if (refresh) {
b.append(" and a refresh");
}
return b.toString();
}

@Override
public String[] indices() {
List<String> indices = new ArrayList<>();
@@ -22,7 +22,11 @@
import org.elasticsearch.ElasticsearchGenerationException;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.Version;
import org.elasticsearch.action.*;
import org.elasticsearch.action.ActionRequest;
import org.elasticsearch.action.ActionRequestValidationException;
import org.elasticsearch.action.DocumentRequest;
import org.elasticsearch.action.RoutingMissingException;
import org.elasticsearch.action.TimestampParsingException;
import org.elasticsearch.action.support.replication.ReplicationRequest;
import org.elasticsearch.client.Requests;
import org.elasticsearch.cluster.metadata.IndexMetaData;
@@ -35,7 +39,11 @@
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.io.stream.StreamOutput;
import org.elasticsearch.common.lucene.uid.Versions;
import org.elasticsearch.common.xcontent.*;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentHelper;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.IndexNotFoundException;
import org.elasticsearch.index.VersionType;
import org.elasticsearch.index.mapper.MapperParsingException;
@@ -660,7 +668,7 @@ public void process(MetaData metaData, @Nullable MappingMetaData mappingMd, bool
@Override
public void readFrom(StreamInput in) throws IOException {
super.readFrom(in);
type = in.readString();
type = in.readOptionalString();

This comment has been minimized.

Copy link
@ywelsch

ywelsch Jan 5, 2016

Contributor

breaks compatibility with old clients. In case of backport, we should be careful here.

This comment has been minimized.

Copy link
@nik9000

nik9000 Jan 5, 2016

Author Contributor

Indeed. And I will backport it. But I think its just a think to keep in mind for then.

This comment has been minimized.

Copy link
@s1monw

s1monw Jan 7, 2016

Contributor

can we put //NORELEASE in there?

This comment has been minimized.

Copy link
@nik9000

nik9000 Jan 12, 2016

Author Contributor

I'm not sure if there is any need to have that for master though.

id = in.readOptionalString();
routing = in.readOptionalString();
parent = in.readOptionalString();
@@ -677,7 +685,7 @@ public void readFrom(StreamInput in) throws IOException {
@Override
public void writeTo(StreamOutput out) throws IOException {
super.writeTo(out);
out.writeString(type);
out.writeOptionalString(type);
out.writeOptionalString(id);
out.writeOptionalString(routing);
out.writeOptionalString(parent);
@@ -20,7 +20,11 @@
package org.elasticsearch.plugins;

import org.apache.lucene.util.IOUtils;
import org.elasticsearch.*;
import org.elasticsearch.Build;
import org.elasticsearch.ElasticsearchCorruptionException;
import org.elasticsearch.ElasticsearchTimeoutException;
import org.elasticsearch.ExceptionsHelper;
import org.elasticsearch.Version;
import org.elasticsearch.bootstrap.JarHell;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.cli.Terminal;
@@ -35,9 +39,24 @@
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.file.*;
import java.nio.file.attribute.*;
import java.util.*;
import java.nio.file.DirectoryStream;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.nio.file.attribute.GroupPrincipal;
import java.nio.file.attribute.PosixFileAttributeView;
import java.nio.file.attribute.PosixFileAttributes;
import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.UserPrincipal;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Random;
import java.util.Set;
import java.util.stream.StreamSupport;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
@@ -85,6 +104,7 @@
"mapper-attachments",
"mapper-murmur3",
"mapper-size",
"reindex",
"repository-azure",
"repository-s3",
"store-smb"));
@@ -72,7 +72,8 @@ public void handleRequest(final RestRequest request, final RestChannel channel,
PutWarmerRequest putWarmerRequest = new PutWarmerRequest(request.param("name"));

BytesReference sourceBytes = RestActions.getRestContent(request);
SearchSourceBuilder source = RestActions.getRestSearchSource(sourceBytes, queryRegistry, parseFieldMatcher);
SearchSourceBuilder source = new SearchSourceBuilder();
RestActions.parseRestSearchSource(source, sourceBytes, queryRegistry, parseFieldMatcher);
SearchRequest searchRequest = new SearchRequest(Strings.splitStringByCommaToArray(request.param("index")))
.types(Strings.splitStringByCommaToArray(request.param("type")))
.requestCache(request.paramAsBoolean("request_cache", null)).source(source);
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.