Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JAMES-3492 #328

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
6719dcc
JAMES-3492 Adapt ElasticSearchIntegrationTest
LanKhuat Mar 2, 2021
451af03
JAMES-3492 Replace deprecated CreateIndexRequest
LanKhuat Mar 2, 2021
172e271
JAMES-3492 Remove mapping check
LanKhuat Mar 2, 2021
bffd4c4
JAMES-3492 Migrate to Java time date format
LanKhuat Mar 4, 2021
79f7181
JAMES-3492 Search no longer need to fetch individual fields
LanKhuat Mar 3, 2021
6821807
JAMES-3492 Adapt other implementations of AbstractMessageSearchIndexTest
LanKhuat Mar 3, 2021
4fdf070
JAMES-3492 Adapt ElasticSearchSearcherTest
LanKhuat Mar 3, 2021
3a9ab02
JAMES-3492 Mapping should be created at the same time as index
LanKhuat Mar 3, 2021
65f2a12
JAMES-3492 Check Mapping existence by GetMappingsAPI
LanKhuat Mar 3, 2021
99be71d
JAMES-3492 Add missing dependency
LanKhuat Mar 4, 2021
f8757dd
JAMES-3492 Adapt ElasticSearchQuotaMailboxListenerTest
LanKhuat Mar 4, 2021
63d9303
JAMES-3492 Use assertion with Awaitility in QuotaSearcherContract
LanKhuat Mar 4, 2021
55d0299
JAMES-3492 Remove NodeMappingFactory & tests
LanKhuat Mar 8, 2021
20ffa84
JAMES-3492, switch to elasticsearch 7 in guice modules
Mar 9, 2021
0ee7303
Merge remote-tracking branch 'lankhuat/elastic-search-7-mailbox-test'…
Mar 9, 2021
047455f
JAMES-3492, switch to elasticsearch 7 in poms
Mar 10, 2021
3a22278
JAMES-3492, disable failing tests
Mar 10, 2021
e984390
JAMES-3492, change ElasticSearchStartUpCheck.java to use v7
Mar 10, 2021
ca156cf
JAMES-3492, fix tests, wait for elasticsearch flushes to complete for…
Mar 10, 2021
ba2ada9
JAMES-3492, applied sortpom:sort
Mar 10, 2021
0b043da
JAMES-3492, remove mistakingly added dependency
Mar 11, 2021
b559e88
JAMES-3502, was using old elastic module
Mar 11, 2021
a3b02a7
JAMES-3502, remove elastic six modules
Mar 11, 2021
db3ea0d
JAMES-3502, fix indent in tests
Mar 11, 2021
0bf612e
JAMES-3492, pass JamesWithNonCompatibleElasticSearchServerTest
Mar 11, 2021
9152b37
JAMES-3492, add upgrade instructions
Mar 11, 2021
3291869
JAMES-3492, add upgrade instructions
Mar 12, 2021
b158ed2
JAMES-3492, upgrade documentation
Mar 12, 2021
2c2aa68
JAMES-3492, add note about need of separate Elasticsearch 6 instance …
Mar 12, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ You need a running *s3* compatible objectstorage in docker. To achieve this run:

You need a running *ElasticSearch* in docker. To achieve this run:

$ docker run -d --name=elasticsearch --env 'discovery.type=single-node' docker.elastic.co/elasticsearch/elasticsearch:6.3.2
$ docker run -d --name=elasticsearch --env 'discovery.type=single-node' docker.elastic.co/elasticsearch/elasticsearch:7.10.2

If you want to handle attachment text extraction before indexing in ElasticSearch (this makes attachments searchable)
and if you want to use all the JMAP search capabilities, you also need to start *Tika*.
Expand Down Expand Up @@ -237,7 +237,7 @@ You need a running *cassandra* in docker. To achieve this run:

You need a running *ElasticSearch* in docker. To achieve this run:

$ docker run -d --name=elasticsearch --env 'discovery.type=single-node' docker.elastic.co/elasticsearch/elasticsearch:6.3.2
$ docker run -d --name=elasticsearch --env 'discovery.type=single-node' docker.elastic.co/elasticsearch/elasticsearch:7.10.2

If you want to handle attachment text extraction before indexing in ElasticSearch (this makes attachments searchable)
and if you want to use all the JMAP search capabilities, you also need to start *Tika*.
Expand Down
2 changes: 1 addition & 1 deletion backends-common/cassandra/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@
<artifactId>commons-beanutils</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
</dependency>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,15 @@
import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;

import java.io.IOException;
import java.util.Optional;

import javax.inject.Inject;

import org.elasticsearch.ElasticsearchStatusException;
import org.elasticsearch.action.admin.indices.alias.IndicesAliasesRequest;
import org.elasticsearch.action.admin.indices.alias.get.GetAliasesRequest;
import org.elasticsearch.action.admin.indices.create.CreateIndexRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand Down Expand Up @@ -63,7 +64,11 @@ public AliasSpecificationStep addAlias(AliasName aliasName) {
}

public ReactorElasticSearchClient createIndexAndAliases(ReactorElasticSearchClient client) {
return new IndexCreationPerformer(nbShards, nbReplica, waitForActiveShards, indexName, aliases.build()).createIndexAndAliases(client);
return new IndexCreationPerformer(nbShards, nbReplica, waitForActiveShards, indexName, aliases.build()).createIndexAndAliases(client, Optional.empty());
}

public ReactorElasticSearchClient createIndexAndAliases(ReactorElasticSearchClient client, XContentBuilder mappingContent) {
return new IndexCreationPerformer(nbShards, nbReplica, waitForActiveShards, indexName, aliases.build()).createIndexAndAliases(client, Optional.of(mappingContent));
}
}

Expand All @@ -82,10 +87,10 @@ public IndexCreationPerformer(int nbShards, int nbReplica, int waitForActiveShar
this.aliases = aliases;
}

public ReactorElasticSearchClient createIndexAndAliases(ReactorElasticSearchClient client) {
public ReactorElasticSearchClient createIndexAndAliases(ReactorElasticSearchClient client, Optional<XContentBuilder> mappingContent) {
Preconditions.checkNotNull(indexName);
try {
createIndexIfNeeded(client, indexName, generateSetting(nbShards, nbReplica, waitForActiveShards));
createIndexIfNeeded(client, indexName, generateSetting(nbShards, nbReplica, waitForActiveShards), mappingContent);
aliases.forEach(Throwing.<AliasName>consumer(alias -> createAliasIfNeeded(client, indexName, alias))
.sneakyThrow());
} catch (IOException e) {
Expand All @@ -111,12 +116,13 @@ private boolean aliasExist(ReactorElasticSearchClient client, AliasName aliasNam
.existsAlias(new GetAliasesRequest().aliases(aliasName.getValue()), RequestOptions.DEFAULT);
}

private void createIndexIfNeeded(ReactorElasticSearchClient client, IndexName indexName, XContentBuilder settings) throws IOException {
private void createIndexIfNeeded(ReactorElasticSearchClient client, IndexName indexName, XContentBuilder settings, Optional<XContentBuilder> mappingContent) throws IOException {
try {
client.indices()
.create(
new CreateIndexRequest(indexName.getValue())
.source(settings), RequestOptions.DEFAULT);
CreateIndexRequest request = new CreateIndexRequest(indexName.getValue()).source(settings);
mappingContent.ifPresent(request::mapping);
client.indices().create(
request,
RequestOptions.DEFAULT);
} catch (ElasticsearchStatusException exception) {
if (exception.getMessage().contains(INDEX_ALREADY_EXISTS_EXCEPTION_MESSAGE)) {
LOGGER.info("Index [{}] already exists", indexName.getValue());
Expand Down Expand Up @@ -185,6 +191,22 @@ private XContentBuilder generateSetting(int nbShards, int nbReplica, int waitFor
public static final String KEEP_MAIL_AND_URL = "keep_mail_and_url";
public static final String SNOWBALL_KEEP_MAIL_AND_URL = "snowball_keep_mail_and_token";
public static final String ENGLISH_SNOWBALL = "english_snowball";
public static final String BOOLEAN = "boolean";
public static final String TYPE = "type";
public static final String LONG = "long";
public static final String DOUBLE = "double";
public static final String KEYWORD = "keyword";
public static final String PROPERTIES = "properties";
public static final String ROUTING = "_routing";
public static final String REQUIRED = "required";
public static final String DATE = "date";
public static final String FORMAT = "format";
public static final String NESTED = "nested";
public static final String FIELDS = "fields";
public static final String RAW = "raw";
public static final String ANALYZER = "analyzer";
public static final String NORMALIZER = "normalizer";
public static final String SEARCH_ANALYZER = "search_analyzer";

@Inject
public IndexCreationFactory(ElasticSearchConfiguration configuration) {
Expand Down

This file was deleted.

This file was deleted.

Loading