Skip to content

NIFI-10437 Converted the Elasticsearch client service tests from Groovy to Java and added Testcontainers integration #6487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MikeThomsen wants to merge 6 commits into from
Closed

NIFI-10437 Converted the Elasticsearch client service tests from Groovy to Java and added Testcontainers integration #6487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
65a6d76
NIFI-10437 Converted SearchResponseTest from Groovy to Java.
MikeThomsen Sep 30, 2022
e6d2f27
NIFI-10437 Finished feedback from a PR; made testcontainers a sane de…
MikeThomsen Oct 6, 2022
55ccc10
NIFI-10437 Updated testcontainers version.
MikeThomsen Oct 6, 2022
fe1f647
NIFI-10437 Made requested changes.
MikeThomsen Oct 6, 2022
927e4e4
NIFI-10437 Made MapBuilder behave like Map.of in Java 11+.
MikeThomsen Oct 6, 2022
eb0250c
NIFI-10437 Finished documentation update.
MikeThomsen Oct 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 17 additions & 45 deletions ...r-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,60 +17,32 @@

## Integration Tests

The `nifi-elasticsearch-client-service` component build allows for optional Integration Tests to be executed to verify
additional functionality.
### Overview

The Integration Tests create an in-memory instance of Elasticsearch, populate it with known data, perform operations
upon the instance and verify the results.
The integration tests use [Testcontainers](https://www.testcontainers.org/) to provide a sane default for developers who have installed Docker. Testcontainers support can be disabled by setting the system property `elasticsearch.testcontainers.enabled` to something other than `true`. If Testcontainers are disabled, the endpoint will need to be configured. It can be set manually with the system property `elasticsearch.endpoint`. The default value is `http://localhost:9200`.

These can be activated by running the following build commands:
### Maven Profiles

### Elasticsearch 5
* `integration-tests`
* `elasticsearch6`
* `elasticsearch7`

Test integration with Elasticsearch 5.x:
### Configurable System Properties

```bash
mvn -P integration-tests,elasticsearch-oss clean verify
```
* `elasticsearch.endpoint` - Manually configure the endpoint root for a non-Docker version of Elasticsearch,
* `elasticsearch.testcontainers.enabled` - Set to anything other than `true` to disable Testcontainers and use a non-Docker version of Elasticsearch.
* `elasticsearch.elastic_user.password` - Set the Elasticsearch `elastic` user's password. When Testcontainers are enabled, this sets up the Docker container and the rest clients for accessing it within the tests. When Testcontainers are disabled, it needs to be set to whatever password is used on the external Elasticsearch node or cluster.

### Elasticsearch 6
### Maven Run Examples

Test integration with Elasticsearch 6.x:
Elasticsearch 8.X is the current default version of Elasticsearch when Testcontainers are used. An example run of the integration tests with Elasticsearch 7 support would be like this:

```bash
mvn -P integration-tests,elasticsearch-oss,elasticsearch-6 clean verify
```
`mvn clean install -Pintegration-tests,elasticsearch7`

### Elasticsearch 7
An example using a non-Docker version of Elasticsearch:

[elasticsearch-oss](https://www.elastic.co/downloads/past-releases#elasticsearch-oss) was discontinued after `7.10.2`,
so the use of `elasticsearch-oss` is unnecessary for newer versions.
`mvn clean install -Pintegration-tests -Delasticsearch.testcontainers.enabled=false -Delasticsearch.elastic_user.password=s3cret1234`

For 7.x, we have two separate profiles:
### Misc

1. `elasticsearch-7` that can be used with `oss` (no X-Pack) and `default` (with X-Pack) flavours
2. `elasticsearch-7-no-oss` that can only be used with the `default` flavour (using a newer version of [elasticsearch](https://www.elastic.co/downloads/past-releases#elasticsearch))

#### With X-Pack

Allows for testing of some X-Pack only features such as "Point in Time" querying:

```bash
mvn -P integration-tests,elasticsearch-default,elasticsearch-7 clean verify
sleep 2
mvn -P integration-tests,elasticsearch-default,elasticsearch-7-no-oss clean verify
```

#### Without X-Pack

```bash
mvn -P integration-tests,elasticsearch-oss,elasticsearch-7 clean verify
```

### Elasticsearch 8

Test integration with Elasticsearch 8.x (with X-Pack):

```bash
mvn -P integration-tests,elasticsearch-default,elasticsearch-8 clean verify
```
The Testcontainers support currently only supports the x64 release of Dockerized Elasticsearch. ARM64 support may be added later.
198 changes: 27 additions & 171 deletions nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,20 +25,6 @@
<artifactId>nifi-elasticsearch-client-service</artifactId>
<packaging>jar</packaging>

<properties>
<!-- use with integration-tests only -->
<es.int.version>5.6.16</es.int.version>
<alexcojocaru.plugin.version>6.19</alexcojocaru.plugin.version>
<es.int.script.name>setup-5.script</es.int.script.name>
<es.int.type.name>faketype</es.int.type.name>
<es.int.path.conf>src/test/resources/conf-5/</es.int.path.conf>
<es.int.clusterName>testCluster</es.int.clusterName>
<es.int.transportPort>9500</es.int.transportPort>
<es.int.httpPort>9400</es.int.httpPort>
<es.int.timeout>90</es.int.timeout>
<es.int.logLevel>ERROR</es.int.logLevel>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
Expand Down Expand Up @@ -196,6 +182,17 @@
<version>${nifi.groovy.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>elasticsearch</artifactId>
<version>${testcontainers.version}</version>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${testcontainers.version}</version>
</dependency>
</dependencies>

<profiles>
Expand All @@ -204,181 +201,40 @@
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<elasticsearch_docker_image>8.4.3</elasticsearch_docker_image>
<elasticsearch.elastic.password>s3cret</elasticsearch.elastic.password>
</properties>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<elasticsearch.endpoint>http://localhost:9200</elasticsearch.endpoint>
<elasticsearch.testcontainers.enabled>true</elasticsearch.testcontainers.enabled>
<elasticsearch.docker.image>docker.elastic.co/elasticsearch/elasticsearch:${elasticsearch_docker_image}</elasticsearch.docker.image>
<elasticsearch.elastic_user.password>${elasticsearch.elastic.password}</elasticsearch.elastic_user.password>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</profile>

<profile>
<!-- use with elasticsearch-oss only (-default can be used if x-pack-ml permissions fixed) -->
<id>elasticsearch-6</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<id>elasticsearch6</id>
<properties>
<es.int.version>6.8.23</es.int.version>
<es.int.type.name>_doc</es.int.type.name>
<es.int.script.name>setup-6.script</es.int.script.name>
<es.int.path.conf />
<elasticsearch_docker_image>6.8.23</elasticsearch_docker_image>
</properties>
</profile>
<profile>
<!-- use with elasticsearch-oss or elasticsearch-default -->
<id>elasticsearch-7</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<id>elasticsearch7</id>
<properties>
<es.int.version>7.10.2</es.int.version>
<es.int.script.name>setup-7.script</es.int.script.name>
<es.int.type.name />
<es.int.path.conf />
<elasticsearch_docker_image>7.17.6</elasticsearch_docker_image>
</properties>
</profile>
<profile>
<!-- use with elasticsearch-default -->
<id>elasticsearch-7-no-oss</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<es.int.version>7.17.3</es.int.version>
<es.int.script.name>setup-7.script</es.int.script.name>
<es.int.type.name />
<es.int.path.conf />
</properties>
</profile>
<profile>
<!-- use with elasticsearch-default -->
<id>elasticsearch-8</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<es.int.version>8.2.0</es.int.version>
<!-- elasticsearch-maven-plugin version 6.20+ required for Elasticsearch 8.x+ -->
<alexcojocaru.plugin.version>6.22</alexcojocaru.plugin.version>
<es.int.script.name>setup-8.script</es.int.script.name>
<es.int.type.name />
<es.int.path.conf />
</properties>
</profile>

<profile>
<!-- OSS Elasticsearch (no XPack features); required for ES 5.x or < 6.3 and option for 7.x up to 7.10.2 -->
<id>elasticsearch-oss</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<type_name>${es.int.type.name}</type_name>
<es_version>${es.int.version}</es_version>
<es_flavour>oss</es_flavour>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>com.github.alexcojocaru</groupId>
<artifactId>elasticsearch-maven-plugin</artifactId>
<version>${alexcojocaru.plugin.version}</version>
<configuration>
<clusterName>${es.int.clusterName}</clusterName>
<transportPort>${es.int.transportPort}</transportPort>
<httpPort>${es.int.httpPort}</httpPort>
<version>${es.int.version}</version>
<timeout>${es.int.timeout}</timeout>
<logLevel>${es.int.logLevel}</logLevel>
<pathInitScript>${project.basedir}/src/test/resources/${es.int.script.name}</pathInitScript>
<keepExistingData>false</keepExistingData>
<pathConf>${es.int.path.conf}</pathConf>
</configuration>
<executions>
<execution>
<id>start-elasticsearch</id>
<phase>pre-integration-test</phase>
<goals>
<goal>runforked</goal>
</goals>
</execution>
<execution>
<id>stop-elasticsearch</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

<profile>
<!-- Elasticsearch (default) with XPack (only for ES 6.3+ & 7.x although there are XPack permission problems in 6.x startup) -->
<id>elasticsearch-default</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<type_name>${es.int.type.name}</type_name>
<es_version>${es.int.version}</es_version>
<es_flavour>default</es_flavour>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>com.github.alexcojocaru</groupId>
<artifactId>elasticsearch-maven-plugin</artifactId>
<version>${alexcojocaru.plugin.version}</version>
<configuration>
<flavour>default</flavour>
<clusterName>${es.int.clusterName}</clusterName>
<transportPort>${es.int.transportPort}</transportPort>
<httpPort>${es.int.httpPort}</httpPort>
<version>${es.int.version}</version>
<timeout>${es.int.timeout}</timeout>
<logLevel>${es.int.logLevel}</logLevel>
<pathInitScript>${project.basedir}/src/test/resources/${es.int.script.name}</pathInitScript>
<keepExistingData>false</keepExistingData>
</configuration>
<executions>
<execution>
<id>start-elasticsearch</id>
<phase>pre-integration-test</phase>
<goals>
<goal>runforked</goal>
</goals>
</execution>
<execution>
<id>stop-elasticsearch</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
57 changes: 0 additions & 57 deletions ...ch-client-service/src/test/groovy/org/apache/nifi/elasticsearch/SearchResponseTest.groovy
View file
Open in desktop

This file was deleted.

Loading