Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic reindex implementation #15125

Merged
merged 129 commits into from Jan 13, 2016
Merged
Changes from 1 commit
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
839bb21
Basic index-by-search
nik9000 Nov 25, 2015
e918bc1
Fix vagrant tests
nik9000 Nov 25, 2015
39e228e
Docs
nik9000 Nov 25, 2015
7501c00
Basic REST
nik9000 Nov 30, 2015
eee9583
Remove leftovers
nik9000 Nov 30, 2015
f4ec860
tabs!
nik9000 Nov 30, 2015
542c656
More docs
nik9000 Nov 30, 2015
d02f277
Copy routing
nik9000 Nov 30, 2015
e5ae49f
Add support for preserving external versions
nik9000 Nov 30, 2015
169e45e
Be more careful with the search request
nik9000 Dec 1, 2015
6db33aa
Clean up rest
nik9000 Dec 1, 2015
81430f9
More docs and tests
nik9000 Dec 1, 2015
f81e4e4
Fix more validation
nik9000 Dec 1, 2015
dd8fa09
Add support for routing
nik9000 Dec 1, 2015
222c986
Remove file that is no longer needed
nik9000 Dec 2, 2015
f9a3ae0
Add test for updating in place
nik9000 Dec 2, 2015
d40767e
Add support for op_type
nik9000 Dec 2, 2015
85ca3be
Move size override out of builder
nik9000 Dec 3, 2015
e562f35
Lots of version related things
nik9000 Dec 4, 2015
b96b218
More versioning work
nik9000 Dec 4, 2015
0be4b29
Preserve version by default reindexing
nik9000 Dec 4, 2015
6e7650c
Docs
nik9000 Dec 4, 2015
a3cc171
Start fixing response format
nik9000 Dec 4, 2015
6f3e563
Return failures
nik9000 Dec 7, 2015
f622e36
Rest tests for failures and batches
nik9000 Dec 7, 2015
630f26c
Squash index-by-search into on package
nik9000 Dec 7, 2015
79ee748
Add warning about atomicity to docs
nik9000 Dec 7, 2015
012336e
Rename version type and fix typo
nik9000 Dec 10, 2015
4e016fb
More rename version type
nik9000 Dec 10, 2015
3f67488
Simplify failure handling
nik9000 Dec 10, 2015
ee981a4
Start building ReindexInPlace
nik9000 Dec 10, 2015
c3a3db8
Fixup reindex and get tests passing
nik9000 Dec 10, 2015
f9d4e40
More reindex support
nik9000 Dec 11, 2015
e3e5d82
More reindex tests
nik9000 Dec 11, 2015
4b285ff
Hack to support body content
nik9000 Dec 11, 2015
04506e1
Remove parsers we can't use
nik9000 Dec 11, 2015
6f438b4
Add support for update scripts
nik9000 Dec 14, 2015
ab19a8a
Script support fixes
nik9000 Dec 14, 2015
4f8f14a
Get run picking up groovy
nik9000 Dec 14, 2015
7fc9f2d
_index_by_search's op_type parameter
nik9000 Dec 14, 2015
dee79c5
Revive update while indexing test
nik9000 Dec 14, 2015
0360bca
index-by-search can't write where it reads
nik9000 Dec 15, 2015
f86cd42
Rename index -> dest
nik9000 Dec 15, 2015
d6050d4
Add convenience destination methods to build
nik9000 Dec 15, 2015
8e5a3a6
More convenience methods in builder
nik9000 Dec 15, 2015
4bb66c2
Rename "search" to "source"
nik9000 Dec 16, 2015
df0ee51
Switch implementation to client
nik9000 Dec 16, 2015
3977010
Share script with reindex
nik9000 Dec 16, 2015
ae59719
Start adding metadata to scripts
nik9000 Dec 16, 2015
c2947a2
Add support for changing routing and parent
nik9000 Dec 16, 2015
5288788
Test changing parent and routing
nik9000 Dec 16, 2015
73075e6
Change definition of batches
nik9000 Dec 16, 2015
c4836ac
More cleanup in preparation for BulkProcessor
nik9000 Dec 16, 2015
865d694
Fix batches better this time
nik9000 Dec 16, 2015
1c68e61
Cleanup
nik9000 Dec 16, 2015
2b924eb
Cleanup
nik9000 Dec 16, 2015
5c9911e
Rename plugin to reindex
nik9000 Dec 17, 2015
a9cc567
Rename reindex->update-by-query
nik9000 Dec 17, 2015
4924475
Rename index-by-search->reindex
nik9000 Dec 17, 2015
b676ea8
Finish rename
nik9000 Dec 17, 2015
5b27eef
Drop version_type from update-by-query
nik9000 Dec 17, 2015
b113f0a
Rename docs
nik9000 Dec 17, 2015
82e2ead
Rewrite docs for the new world
nik9000 Dec 18, 2015
3459b28
Add more strict checks after scripts
nik9000 Dec 18, 2015
4140b00
Support reindex scripts changing metadata
nik9000 Dec 18, 2015
e4947f0
More renames
nik9000 Dec 18, 2015
c977217
Remove op_type from reindex
nik9000 Dec 21, 2015
14b7560
Fix funky layout in reindex source
nik9000 Dec 21, 2015
6504b40
docs
nik9000 Dec 21, 2015
e26d501
Add reindex parameter
nik9000 Dec 21, 2015
927b0d4
Consistency and timeout parameters
nik9000 Dec 21, 2015
aff9dfc
[docs] Add routing example
nik9000 Dec 21, 2015
3d26594
Fix some nocommits
nik9000 Dec 21, 2015
b1a2a96
Remove another nocommit
nik9000 Dec 21, 2015
36beb8d
Remove another nocommit
nik9000 Dec 21, 2015
96f9a86
Add round trip tests
nik9000 Dec 22, 2015
8759aed
Remove leftovers
nik9000 Dec 22, 2015
98dd95a
Revert new VersionType
nik9000 Dec 22, 2015
626e074
Revert test for new VersionType
nik9000 Dec 22, 2015
e396e6f
more docs
nik9000 Dec 22, 2015
7869a99
Move script tests into a qa module
nik9000 Dec 22, 2015
ba4e73e
Move building the bulk onto generic thread pool
nik9000 Dec 22, 2015
9c2b508
Fix out of date comments
nik9000 Dec 22, 2015
6107250
Return an error when the request fails
nik9000 Dec 23, 2015
12d7c80
Add missing license headers
nik9000 Dec 28, 2015
b5bce59
Docs fixes
nik9000 Dec 28, 2015
2e038ba
Implement BulkShardRequest.toString
nik9000 Dec 28, 2015
3c87503
Status names -> status codes
nik9000 Dec 28, 2015
cb98b2d
Cleanups from comments
nik9000 Dec 28, 2015
06de720
Use FieldMappers field names
nik9000 Dec 28, 2015
04d2912
Fixes from comments
nik9000 Dec 28, 2015
c4e7007
Replace -1 with a constant
nik9000 Dec 28, 2015
da38415
Refactor to remove duplication
nik9000 Dec 28, 2015
d43a968
Document source/src alias
nik9000 Dec 28, 2015
0461940
Document dest/destination alias
nik9000 Dec 28, 2015
4d4d93f
Updates from comments
nik9000 Dec 28, 2015
be501c7
unmodifiable list
nik9000 Dec 28, 2015
184123f
Let IOException bubble
nik9000 Dec 28, 2015
0c3f1b0
Rename constant
nik9000 Dec 28, 2015
23aaa61
Allow user to override scroll_size
nik9000 Dec 28, 2015
fe1d36d
More renames
nik9000 Dec 28, 2015
82c318a
Remove ?
nik9000 Dec 28, 2015
906f455
Add to the list of official plugins
nik9000 Dec 28, 2015
088a978
Changes from comments
nik9000 Dec 29, 2015
022cddb
OptionalStreamable!
nik9000 Jan 5, 2016
5c1e6da
Docs updates
nik9000 Jan 5, 2016
c0bc003
Help should include reindex
nik9000 Jan 5, 2016
e076531
Some fixes from review
nik9000 Jan 6, 2016
f4fd2f6
Remove duplicate check
nik9000 Jan 6, 2016
a73c252
Switch to getters in response
nik9000 Jan 6, 2016
384a0cb
Remove {}
nik9000 Jan 6, 2016
fa72146
Remove superfluous continue
nik9000 Jan 6, 2016
fb3d418
Validate timestamp and ttl aren't set
nik9000 Jan 6, 2016
9582876
Updates from comments
nik9000 Jan 7, 2016
354f7e0
Getters
nik9000 Jan 7, 2016
6831b4f
Tighter try
nik9000 Jan 7, 2016
6c295d2
Setters
nik9000 Jan 7, 2016
ffb6ed1
Remove useless continue
nik9000 Jan 7, 2016
d1363ef
Test that thread pool rejections are reported
nik9000 Jan 7, 2016
aac3ddd
Properly fail on shard failures
nik9000 Jan 7, 2016
1dc46e9
Start moving scripting tests to unit tests
nik9000 Jan 7, 2016
00947a8
Move script tests to unit tests
nik9000 Jan 11, 2016
8e49b35
Move alias checking testing into unit tests
nik9000 Jan 11, 2016
c0f3635
[docs] Make a list of metadata
nik9000 Jan 12, 2016
8e19e4f
[docs] Describe timeout
nik9000 Jan 12, 2016
6b85c37
[doc] Add more callouts
nik9000 Jan 12, 2016
e07a49b
Default conflicts to ignore
nik9000 Jan 12, 2016
ae8dbe2
Remove src and destination
nik9000 Jan 12, 2016
a8a3467
Move metadata tests to unit tests
nik9000 Jan 13, 2016
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

Add support for preserving external versions

  • Loading branch information...
nik9000 committed Nov 30, 2015
commit e5ae49f7d19cd161ecca65831c4da52e1a34d3b8
@@ -137,6 +137,35 @@ When using the `q` parameter, the following additional parameters are
supported (as explained in {ref}/search-uri-request.html[URI search request]):
`df`, `analyzer`, `default_operator`.

[float]
=== Preserving external versioning

If the documents were built with external version numbers
(`version_type=external`) then you can preserve them during the index like so:
[source,js]
--------------------------------------------------
POST /source/_index_by_search
{
"index": {
"index": "dest",
"version_type": "external"
}
}
--------------------------------------------------
// AUTOSENSE

or

[source,js]
--------------------------------------------------
POST /source/_index_by_search?version_type=external
{
"index": {
"index": "dest"
}
}
--------------------------------------------------
// AUTOSENSE

[float]
=== Response body
@@ -19,6 +19,8 @@

package org.elasticsearch.action.indexbysearch;

import static org.elasticsearch.search.sort.SortBuilders.fieldSort;

import java.io.IOException;

import org.elasticsearch.action.ActionRequest;
@@ -33,6 +35,7 @@

public class IndexBySearchRequest extends ActionRequest<IndexBySearchRequest> {
private static final TimeValue DEFAULT_SCROLL_TIMEOUT = TimeValue.timeValueMinutes(5);
private static final int DEFAULT_SIZE = 100;

/**
* The search to be executed.
@@ -59,6 +62,10 @@ public IndexBySearchRequest(SearchRequest search, IndexRequest index) {
this.index = index;

search.scroll(DEFAULT_SCROLL_TIMEOUT);
search.source(new SearchSourceBuilder());
search.source().version(true);
search.source().sort(fieldSort("_doc"));
search.source().size(DEFAULT_SIZE);
}

@Override
@@ -87,6 +94,7 @@ public IndexRequest index() {
* Set the search source (used by ObjectParser).
*/
public void searchSource(SearchSourceBuilder sourceBuilder) {
// NOCOMMIT - this is totally broken.
search().source(sourceBuilder);
}

@@ -19,6 +19,8 @@

package org.elasticsearch.action.indexbysearch;

import static java.lang.Math.min;

import org.elasticsearch.action.Action;
import org.elasticsearch.action.ActionRequestBuilder;
import org.elasticsearch.action.index.IndexAction;
@@ -27,9 +29,6 @@
import org.elasticsearch.action.search.SearchRequestBuilder;
import org.elasticsearch.client.ElasticsearchClient;

import static java.lang.Math.min;
import static org.elasticsearch.search.sort.SortBuilders.fieldSort;

public class IndexBySearchRequestBuilder
extends ActionRequestBuilder<IndexBySearchRequest, IndexBySearchResponse, IndexBySearchRequestBuilder> {
private final SearchRequestBuilder search;
@@ -47,9 +46,6 @@ private IndexBySearchRequestBuilder(ElasticsearchClient client,
super(client, action, new IndexBySearchRequest(search.request(), index.request()));
this.search = search;
this.index = index;

search.addSort(fieldSort("_doc"));
search.setSize(100); // Default to large-ish bulks
}

public SearchRequestBuilder search() {
@@ -19,6 +19,8 @@

package org.elasticsearch.action.indexbysearch;

import static org.elasticsearch.index.VersionType.EXTERNAL;

import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.TransportBulkAction;
@@ -87,10 +89,10 @@ protected BulkRequest buildBulk(Iterable<SearchHit> docs) {
*/
index.type(doc.type());
}
// TODO routing needs to be copied
index.source(doc.sourceRef());
// TODO version - preserve external versioning optionally - if
// it is external than we should always?
if (index.versionType() == EXTERNAL) {
index.version(doc.version());
}
// TODO if writing back to the same index we can check the
// version numbers

@@ -41,6 +41,7 @@
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.index.VersionType;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryParseContext;
import org.elasticsearch.indices.query.IndicesQueriesRegistry;
@@ -73,6 +74,7 @@
ObjectParser<IndexRequest, Void> indexParser = new ObjectParser<>("index");
indexParser.declareString(IndexRequest::index, new ParseField("index"));
indexParser.declareString(IndexRequest::type, new ParseField("type"));
indexParser.declareString((i, v) -> i.versionType(VersionType.fromString(v)), new ParseField("version_type"));
PARSER.declareField((p, v, c) -> indexParser.parse(p, v.index(), null), new ParseField("index"), ValueType.OBJECT);
}

@@ -102,7 +104,7 @@ public void handleRequest(RestRequest request, RestChannel channel, Client clien
internalRequest.search().indicesOptions(IndicesOptions.fromRequest(request, internalRequest.search().indicesOptions()));
internalRequest.search().routing(request.param("routing"));

// TODO routing for index?
internalRequest.index().versionType(VersionType.fromString(request.param("version_type"), internalRequest.index().versionType()));

try (XContentParser xcontent = XContentFactory.xContent(request.content()).createParser(request.content())) {
PARSER.parse(xcontent, internalRequest, new QueryParseContext(indicesQueriesRegistry));
@@ -19,7 +19,32 @@

package org.elasticsearch.plugin.indexbysearch;

import static org.elasticsearch.index.VersionType.EXTERNAL;
import static org.elasticsearch.index.VersionType.INTERNAL;

import org.elasticsearch.action.indexbysearch.IndexBySearchRequestBuilder;

public class IndexBySearchVersionTests extends IndexBySearchTestCase {
// NOCOMMIT add tests for not clobbering versions on update
// NOCOMMIT add tests for preserving external versioning

public void testExternalVersioning() throws Exception {
indexRandom(true, client().prepareIndex("test", "source", "test")
.setVersionType(EXTERNAL)
.setVersion(4).setSource("foo", "bar"));

assertEquals(4, client().prepareGet("test", "source", "test").get().getVersion());

// If the copy request
IndexBySearchRequestBuilder copy = newIndexBySearch();
copy.index().setIndex("test").setType("dest_internal").setVersionType(INTERNAL);
assertResponse(copy.get(), 1, 0);
refresh();
assertEquals(1, client().prepareGet("test", "dest_internal", "test").get().getVersion());

copy = newIndexBySearch();
copy.index().setIndex("test").setType("dest_external").setVersionType(EXTERNAL);
assertResponse(copy.get(), 1, 0);
refresh();
assertEquals(4, client().prepareGet("test", "dest_external", "test").get().getVersion());
}
}
@@ -20,6 +20,11 @@
"q": {
"type" : "string",
"description" : "Query in the Lucene query string syntax"
},
"version_type": {
"type" : "enum",
"options" : ["internal", "external", "external_gte", "force"],
"description" : "Specific version type"
}
}
},
@@ -0,0 +1,76 @@
---
"Preserve external version if set on url":
- do:
index:
index: test
type: test
id: 1
version: 4
version_type: external
body: { "text": "test" }
- do:
indices.refresh: {}

- do:
get:
index: test
type: test
id: 1
- match: { _version: 4 }

- do:
index_by_search:
index: test
version_type: external
body:
index:
index: target
- do:
indices.refresh: {}

- do:
get:
index: target
type: test
id: 1
- match: { _version: 4 }

---
"Preserve external version if set on the index request":
- do:
index:
index: test
type: test
id: 1
version: 4
version_type: external
body: { "text": "test" }
- do:
indices.refresh: {}

- do:
get:
index: test
type: test
id: 1
- match: { _version: 4 }

- do:
index_by_search:
index: test
body:
# search: NOCOMMIT setting these shouldn't break anything but it does!
# query:
# match_all: {}
index:
index: target
version_type: external
- do:
indices.refresh: {}

- do:
get:
index: target
type: test
id: 1
- match: { _version: 4 }
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.