Skip to content

NIFI-9678 update Elasticsearch REST API processor integration-tests for Elasticsearch 8.x #5822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ChrisSamo632 wants to merge 1 commit into from
Closed

NIFI-9678 update Elasticsearch REST API processor integration-tests for Elasticsearch 8.x #5822

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ...t-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,14 @@ public interface ElasticSearchClientService extends ControllerService {
*/
UpdateOperationResponse updateByQuery(String query, String index, String type, Map<String, String> requestParameters);

/**
* Refresh index/indices.
*
* @param index The index to target, if omitted then all indices will be updated.
* @param requestParameters A collection of URL request parameters. Optional.
*/
void refresh(final String index, final Map<String, String> requestParameters);

/**
* Get a document by ID.
*
Expand Down
18 changes: 17 additions & 1 deletion ...r-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,18 +43,34 @@ mvn -P integration-tests,elasticsearch-oss,elasticsearch-6 clean verify

### Elasticsearch 7

Test integration with Elasticsearch 7.x:
[elasticsearch-oss](https://www.elastic.co/downloads/past-releases#elasticsearch-oss) was discontinued after `7.10.2`,
so the use of `elasticsearch-oss` is unnecessary for newer versions.

For 7.x, we have two separate profiles:

1. `elasticsearch-7` that can be used with `oss` (no X-Pack) and `default` (with X-Pack) flavours
2. `elasticsearch-7-no-oss` that can only be used with the `default` flavour (using a newer version of [elasticsearch](https://www.elastic.co/downloads/past-releases#elasticsearch))

#### With X-Pack

Allows for testing of some X-Pack only features such as "Point in Time" querying:

```bash
mvn -P integration-tests,elasticsearch-default,elasticsearch-7 clean verify
sleep 2
mvn -P integration-tests,elasticsearch-default,elasticsearch-7-no-oss clean verify
```

#### Without X-Pack

```bash
mvn -P integration-tests,elasticsearch-oss,elasticsearch-7 clean verify
```

### Elasticsearch 8

Test integration with Elasticsearch 8.x (with X-Pack):

```bash
mvn -P integration-tests,elasticsearch-default,elasticsearch-8 clean verify
```
48 changes: 38 additions & 10 deletions nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
<properties>
<!-- use with integration-tests only -->
<es.int.version>5.6.16</es.int.version>
<alexcojocaru.plugin.version>6.19</alexcojocaru.plugin.version>
<es.int.script.name>setup-5.script</es.int.script.name>
<es.int.type.name>faketype</es.int.type.name>
<es.int.path.conf>src/test/resources/conf-5/</es.int.path.conf>
Expand Down Expand Up @@ -214,11 +215,6 @@
<!-- use 3.0.0-M3 due to a classpath/class loader issue in -M5, expected to be fixed in M6+ -->
<version>3.0.0-M3</version>
</plugin>
<plugin>
<groupId>com.github.alexcojocaru</groupId>
<artifactId>elasticsearch-maven-plugin</artifactId>
<version>6.19</version>
</plugin>
</plugins>
</pluginManagement>
</build>
Expand All @@ -231,7 +227,8 @@
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<es.int.version>6.8.19</es.int.version>
<es.int.version>6.8.23</es.int.version>
<alexcojocaru.plugin.version>6.19</alexcojocaru.plugin.version>
<es.int.type.name>_doc</es.int.type.name>
<es.int.script.name>setup-6.script</es.int.script.name>
<es.int.path.conf />
Expand All @@ -245,14 +242,43 @@
</activation>
<properties>
<es.int.version>7.10.2</es.int.version>
<alexcojocaru.plugin.version>6.19</alexcojocaru.plugin.version>
<es.int.script.name>setup-7.script</es.int.script.name>
<es.int.type.name />
<es.int.path.conf />
</properties>
</profile>
<profile>
<!-- use with elasticsearch-default -->
<id>elasticsearch-7-no-oss</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<es.int.version>7.17.1</es.int.version>
<alexcojocaru.plugin.version>6.19</alexcojocaru.plugin.version>
<es.int.script.name>setup-7.script</es.int.script.name>
<es.int.type.name />
<es.int.path.conf />
</properties>
</profile>
<profile>
<!-- use with elasticsearch-default -->
<id>elasticsearch-8</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<es.int.version>8.0.1</es.int.version>
<alexcojocaru.plugin.version>6.20</alexcojocaru.plugin.version>
<es.int.script.name>setup-8.script</es.int.script.name>
<es.int.type.name />
<es.int.path.conf />
</properties>
</profile>

<profile>
<!-- OSS Elasticsearch (no XPack features); required for ES 5.x or < 6.3-->
<!-- OSS Elasticsearch (no XPack features); required for ES 5.x or < 6.3 and option for 7.x up to 7.10.2 -->
<id>elasticsearch-oss</id>
<activation>
<activeByDefault>false</activeByDefault>
Expand All @@ -273,14 +299,15 @@
<plugin>
<groupId>com.github.alexcojocaru</groupId>
<artifactId>elasticsearch-maven-plugin</artifactId>
<version>${alexcojocaru.plugin.version}</version>
<configuration>
<clusterName>${es.int.clusterName}</clusterName>
<transportPort>${es.int.transportPort}</transportPort>
<httpPort>${es.int.httpPort}</httpPort>
<version>${es.int.version}</version>
<timeout>${es.int.timeout}</timeout>
<logLevel>${es.int.logLevel}</logLevel>
<pathInitScript>${project.basedir}/src/test/resources/${es.int.script.name}</pathInitScript>
<pathInitScript>${project.basedir}/src/test/resources/${es.int.script.name}</pathInitScript>
<keepExistingData>false</keepExistingData>
<pathConf>${es.int.path.conf}</pathConf>
</configuration>
Expand All @@ -306,7 +333,7 @@
</profile>

<profile>
<!-- Elasticsearch (default) with XPack (only for ES 6.3+ although there are XPack permission problems in 6.x startup) -->
<!-- Elasticsearch (default) with XPack (only for ES 6.3+ & 7.x although there are XPack permission problems in 6.x startup) -->
<id>elasticsearch-default</id>
<activation>
<activeByDefault>false</activeByDefault>
Expand All @@ -327,6 +354,7 @@
<plugin>
<groupId>com.github.alexcojocaru</groupId>
<artifactId>elasticsearch-maven-plugin</artifactId>
<version>${alexcojocaru.plugin.version}</version>
<configuration>
<flavour>default</flavour>
<clusterName>${es.int.clusterName}</clusterName>
Expand All @@ -335,7 +363,7 @@
<version>${es.int.version}</version>
<timeout>${es.int.timeout}</timeout>
<logLevel>${es.int.logLevel}</logLevel>
<pathInitScript>${project.basedir}/src/test/resources/${es.int.script.name}</pathInitScript>
<pathInitScript>${project.basedir}/src/test/resources/${es.int.script.name}</pathInitScript>
<keepExistingData>false</keepExistingData>
</configuration>
<executions>
Expand Down
17 changes: 16 additions & 1 deletion ...t-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,7 @@ public DeleteOperationResponse deleteByQuery(final String query, final String in
return new DeleteOperationResponse(watch.getDuration(TimeUnit.MILLISECONDS));
}


@Override
public UpdateOperationResponse updateByQuery(final String query, final String index, final String type, final Map<String, String> requestParameters) {
final long start = System.currentTimeMillis();
final Response response = runQuery("_update_by_query", query, index, type, requestParameters);
Expand All @@ -401,6 +401,21 @@ public UpdateOperationResponse updateByQuery(final String query, final String in
return new UpdateOperationResponse(end - start);
}

@Override
public void refresh(final String index, final Map<String, String> requestParameters) {
try {
final StringBuilder endpoint = new StringBuilder();
if (StringUtils.isNotBlank(index) && !"/".equals(index)) {
endpoint.append(index);
}
endpoint.append("/_refresh");
final Response response = performRequest("POST", endpoint.toString(), requestParameters, null);
parseResponseWarningHeaders(response);
} catch (final Exception ex) {
throw new ElasticsearchException(ex);
}
}

@SuppressWarnings("unchecked")
@Override
public Map<String, Object> get(final String index, final String type, final String id, final Map<String, String> requestParameters) {
Expand Down
9 changes: 8 additions & 1 deletion ...est/groovy/org/apache/nifi/elasticsearch/integration/ElasticSearchClientService_IT.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ class ElasticSearchClientService_IT {

static final ComparableVersion VERSION = new ComparableVersion(System.getProperty("es_version", "0.0.0"))
static final ComparableVersion ES_7_10 = new ComparableVersion("7.10")
static final ComparableVersion ES_8_0 = new ComparableVersion("8.0")

static final String FLAVOUR = System.getProperty("es_flavour")
static final String DEFAULT = "default"
Expand Down Expand Up @@ -117,6 +118,8 @@ class ElasticSearchClientService_IT {
ex.printStackTrace()
throw ex
}

service.refresh(null, null);
}

@After
Expand Down Expand Up @@ -223,9 +226,13 @@ class ElasticSearchClientService_IT {

@Test
void testSearchWarnings() {
Assume.assumeTrue("Requires version <8.0 (no search API deprecations yet for 8.x)", VERSION < ES_8_0)

String query
String type = TYPE
if (VERSION.toString().startsWith("7.")) {
if (VERSION.toString().startsWith("8.")) {
// TODO: something that's deprecated when the 8.x branch progresses to include search-API deprecations
Copy link
Copy Markdown
Contributor

@gresockj gresockj Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What shall we do with this here?

Copy link
Copy Markdown
Contributor Author

@ChrisSamo632 ChrisSamo632 Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm generally not a fan of TODOs in code, but there's literally no deprecated Search API funcitons in Elasticsearch 8.x yet (it's too new for them to have deprecated anything, having just removed all previously deprecated stuff from 7.x).

I've Skipped the test for 8.x using the Assume but I think we should look to update this test to prove the WARNING funcitnoality for 8.x once Search API deprecations appear in Elasticsearch (so that would be a case of removing the Assume and implementing some sort of query here that caused a WARNING).

We could just remove this TODO and if branch, but I thought the TODO here would hopefully serve as a reminder to do this in future. I'd normally raise a Jira ticket to cover a TODO, but as there's no realistic way of currently implementing such a Jira ticket, it seemed like that wouldn't be sensible either.

So I think the options are:

  1. Leave as-is and pick this up for a future version of Elasticsearch 8.x
  2. Remove the TODO and hope someone picks this up in future
  3. Raise a Jira ticket to pick this up in future, noting that it can't currently be implemented and someone will need to watch out for an appropriate Search API deprecation in order to achieve the test

Copy link
Copy Markdown
Contributor

@gresockj gresockj Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the elaboration -- in light of that, I think we should keep the TODO.

} else if (VERSION.toString().startsWith("7.")) {
// querying with _type in ES 7.x is deprecated
query = prettyPrint(toJson([size: 1, query: [match_all: [:]]]))
type = "a-type"
Expand Down
4 changes: 4 additions & 0 deletions ...st/groovy/org/apache/nifi/elasticsearch/integration/TestElasticSearchClientService.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ class TestElasticSearchClientService extends AbstractControllerService implement
return null
}

@Override
void refresh(final String index, final Map<String, String> requestParameters) {
}

@Override
Map<String, Object> get(String index, String type, String id, Map<String, String> requestParameters) {
return data
Expand Down
2 changes: 0 additions & 2 deletions ...-elasticsearch-bundle/nifi-elasticsearch-client-service/src/test/resources/setup-5.script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,3 @@ PUT:nested/faketype/1:{"msg":"Hello, world","subField":{"longField":150000,"date
PUT:nested/faketype/2:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Hello, world!","deepest":{"super_secret":"I could tell, but then I would have to kill you"}}}}
PUT:nested/faketype/3:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Buongiorno, mondo!!","deepest":{"super_secret":"The sky is blue"}}}}

# refresh all indices before testing
POST:_refresh:{}
2 changes: 0 additions & 2 deletions ...-elasticsearch-bundle/nifi-elasticsearch-client-service/src/test/resources/setup-6.script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,3 @@ PUT:nested/_doc/1:{"msg":"Hello, world","subField":{"longField":150000,"dateFiel
PUT:nested/_doc/2:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Hello, world!","deepest":{"super_secret":"I could tell, but then I would have to kill you"}}}}
PUT:nested/_doc/3:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Buongiorno, mondo!!","deepest":{"super_secret":"The sky is blue"}}}}

# refresh all indices before testing
POST:_refresh:{}
2 changes: 0 additions & 2 deletions ...-elasticsearch-bundle/nifi-elasticsearch-client-service/src/test/resources/setup-7.script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,3 @@ POST:nested/_doc/1:{"msg":"Hello, world","subField":{"longField":150000,"dateFie
POST:nested/_doc/2:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Hello, world!","deepest":{"super_secret":"I could tell, but then I would have to kill you"}}}}
POST:nested/_doc/3:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Buongiorno, mondo!!","deepest":{"super_secret":"The sky is blue"}}}}

# refresh all indices before testing
POST:_refresh:{}
48 changes: 48 additions & 0 deletions ...-elasticsearch-bundle/nifi-elasticsearch-client-service/src/test/resources/setup-8.script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#create mapping
PUT:user_details/:{ "mappings":{ "properties":{ "email":{"type":"keyword"},"phone":{"type": "keyword"},"accessKey":{"type": "keyword"}}}}
PUT:messages/:{ "mappings":{ "properties":{ "msg":{"type":"keyword"}}}}
PUT:complex/:{"mappings":{ "properties":{"msg":{"type":"keyword"},"subField":{"type":"nested","properties":{"longField":{"type":"long"},"dateField":{"type":"date"}}}}}}
PUT:nested/:{"mappings":{ "properties":{"msg":{"type":"keyword"},"subField":{"type":"nested","properties":{"longField":{"type":"long"},"dateField":{"type":"date"},"deeper":{"type":"nested","properties":{"secretz":{"type":"keyword"},"deepest":{"type":"nested","properties":{"super_secret":{"type":"keyword"}}}}}}}}}}
PUT:bulk_a/:{ "mappings":{ "properties":{ "msg":{"type":"keyword"}}}}
PUT:bulk_b/:{ "mappings":{ "properties":{ "msg":{"type":"keyword"}}}}
PUT:bulk_c/:{ "mappings":{ "properties":{ "msg":{"type":"keyword"}}}}
PUT:error_handler:{ "mappings": { "properties": { "msg": { "type": "keyword" }, "intField": { "type": "integer" }}}}

#add document
POST:messages/_doc/1:{ "msg":"one" }
POST:messages/_doc/2:{ "msg":"two" }
POST:messages/_doc/3:{ "msg":"two" }
POST:messages/_doc/4:{ "msg":"three" }
POST:messages/_doc/5:{ "msg":"three" }
POST:messages/_doc/6:{ "msg":"three" }
POST:messages/_doc/7:{ "msg":"four" }
POST:messages/_doc/8:{ "msg":"four" }
POST:messages/_doc/9:{ "msg":"four" }
POST:messages/_doc/10:{ "msg":"four" }
POST:messages/_doc/11:{ "msg":"five" }
POST:messages/_doc/12:{ "msg":"five" }
POST:messages/_doc/13:{ "msg":"five" }
POST:messages/_doc/14:{ "msg":"five" }
POST:messages/_doc/15:{ "msg":"five" }
POST:complex/_doc/1:{"msg":"Hello, world","subField":{"longField":100000,"dateField":"2018-04-10T12:18:05Z"}}
POST:user_details/_doc/1:{ "email": "john.smith@company.com", "phone": "123-456-7890", "accessKey": "ABCDE"}
POST:user_details/_doc/2:{ "email": "jane.doe@company.com", "phone": "098-765-4321", "accessKey": "GHIJK"}
POST:nested/_doc/1:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"No one should see this!","deepest":{"super_secret":"Got nothin to hide"}}}}
POST:nested/_doc/2:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Hello, world!","deepest":{"super_secret":"I could tell, but then I would have to kill you"}}}}
POST:nested/_doc/3:{"msg":"Hello, world","subField":{"longField":150000,"dateField":"2018-08-14T10:08:00Z","deeper":{"secretz":"Buongiorno, mondo!!","deepest":{"super_secret":"The sky is blue"}}}}

4 changes: 4 additions & 0 deletions ...est/groovy/org/apache/nifi/processors/elasticsearch/TestElasticsearchClientService.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,10 @@ class TestElasticsearchClientService extends AbstractControllerService implement
return new UpdateOperationResponse(100L)
}

@Override
void refresh(final String index, final Map<String, String> requestParameters) {
}

@Override
Map<String, Object> get(String index, String type, String id, Map<String, String> requestParameters) {
common(throwErrorInGet || throwNotFoundInGet, requestParameters)
Expand Down
4 changes: 4 additions & 0 deletions ...oovy/org/apache/nifi/processors/elasticsearch/mock/AbstractMockElasticsearchClient.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ class AbstractMockElasticsearchClient extends AbstractControllerService implemen
return null
}

@Override
void refresh(final String index, final Map<String, String> requestParameters) {
}

@Override
Map<String, Object> get(String index, String type, String id, Map<String, String> requestParameters) {
return null
Expand Down