Integration Test Spring Boot Application on Kubernetes/OpenShift

A demo application that will show how to do Integration Test for Spring Boot application(s) deployed inside Kubernetes/OpenShift using Arquillian CUBE

Setup

• You might need an access to Kubernetes Cluster to play with this application. The easiest way to get local Kuberentes cluster up and running is using minikube

• Install Spring Boot CLI

• Untar/Unzip Spring Boot CLI 1.5.7.RELEASE and add it your path

Note	The rest of the document assumes you have minikube up and running

Build and Deploy

Create project

For the demo purpose we will be creating a simple Spring Boot project,

spring init --artifactId=spring-boot-arq-demo \
   --name="Kubernetes:: Spring Boot:: Arquillian :: Integration Testing Demo"  \
   --groupId="com.redhat.developers" \
   --package-name="com.redhat.developers" \
   --dependencies=web,actuator,lombok \
   --extract spring-boot-arq-demo (1)

cd spring-boot-arq-demo

./mvnw io.fabric8:fabric8-maven-plugin:3.5.30:setup (2)

1. Creates a Spring Boot project with web and actuator dependencies

2. Adds the fabric8-maven-plugin to the project that will help with deploying the application on Kubernetes

The complete demo sources is available here

The directory spring-boot-arq-demo will be referred to as $PROJECT_HOME throughout this document.

Add a simple Greeting REST API

package com.redhat.developers;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
public class GreeterController {


    @GetMapping("/greet")
    public String greet() {
        return "Hello";
    }
}

Unit Test

The unit test is done in typical Spring Boot Unit testing strategy.

Update pom.xml

Update the pom.xml to add some extra dependencies that we will be using during testing

...
<dependency> (1)
  <groupId>org.assertj</groupId>
  <artifactId>assertj-core</artifactId>
  <scope>test</scope>
</dependency>
<dependency> (2)
  <groupId>com.jayway.restassured</groupId>
  <artifactId>rest-assured</artifactId>
  <version>2.9.0</version>
  <scope>test</scope>
</dependency>
...

1. provides API for assertions

2. provides API for making rest calls and asserting various part of the response like status code, body etc.,

Add Common test cases

Since we will be using same test case for both UT and IT, lets abstract it in to separate class,

package com.redhat.developers;

import org.junit.Test;

import static com.jayway.restassured.RestAssured.when;
import static org.hamcrest.CoreMatchers.is;

public class AbstractGreeterTests {


    @Test
    public void greet_should_return_hello_with_status_http_OK() throws Exception {
        when().get() (1)
                .then()
                .statusCode(200) (2)
                .header("Content-Type", "text/plain;charset=UTF-8") (3)
                .body(is("Hello")); (4)
    }
}

1. Do a HTTP GET to the RestAssured.baseURI which will be set by each Test subclass

2. Ensure to get HTTP Status code as 200

3. Ensure that the Content-Type is text/plain;charset=UTF-8

4. Ensure that response body has content called Hello

Add Greeter Test

package com.redhat.developers;

import com.jayway.restassured.RestAssured;
import org.junit.Before;
import org.junit.runner.RunWith;
import org.springframework.boot.context.embedded.LocalServerPort;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class GreeterTest extends AbstractGreeterTests {

    @LocalServerPort
    int port; (1)

    @Before
    public void setup() {
        RestAssured.baseURI = String.format("http://localhost:%d/api/greet", port); (2)
    }

}

1. gives us the random server port on which Spring Boot is started

2. sets the URI which will be used to make the REST calls

Run Test

Lets do the unit test now and make sure to see all our test cases pass

./mvnw clean test

Integration Test

The integration testing of the application on Kubernetes/OpenShift will use Arquillian CUBE framework.

Update pom.xml

To use Arquillian CUBE framework, lets add its related dependencies.

The updated pom.xml is shown below,

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.redhat.developers</groupId>
  <artifactId>spring-boot-arq-demo</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>Kubernetes:: Spring Boot:: Arquillian :: Integration Testing Demo</name>
  <description>Demo project for Spring Boot</description>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.7.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>

  <dependencies>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <optional>true</optional>
    </dependency>

    <!-- Testing -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.jayway.restassured</groupId>
      <artifactId>rest-assured</artifactId>
      <version>2.9.0</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.jboss.arquillian.junit</groupId>
      <artifactId>arquillian-junit-container</artifactId> (1)
      <version>1.1.12.Final</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.arquillian.cube</groupId>
      <artifactId>arquillian-cube-requirement</artifactId> (2)
      <version>1.9.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.arquillian.cube</groupId>
      <artifactId>arquillian-cube-kubernetes</artifactId> (3)
      <version>1.9.0</version>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupId>io.undertow</groupId>
          <artifactId>undertow-core</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>org.arquillian.cube</groupId>
      <artifactId>arquillian-cube-openshift</artifactId> (4)
      <version>1.9.0</version>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupId>io.undertow</groupId>
          <artifactId>undertow-core</artifactId> (5)
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>io.fabric8</groupId>
      <artifactId>kubernetes-api</artifactId>
      <version>2.3.5</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.fabric8</groupId>
      <artifactId>kubernetes-client</artifactId> (6)
      <version>2.6.3</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>io.fabric8</groupId>
      <artifactId>openshift-client</artifactId> (7)
      <version>2.6.3</version>
      <scope>test</scope>
    </dependency>

  </dependencies>

  <build>
    <testResources>
      <testResource>
        <directory>src/test/resources</directory>
        <filtering>true</filtering>
      </testResource>
    </testResources>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
      <plugin>
        <groupId>io.fabric8</groupId>
        <artifactId>fabric8-maven-plugin</artifactId>
        <version>3.5.30</version>
        <executions>
          <execution>
            <id>fmp</id>
            <phase>package</phase>
            <goals>
              <goal>resource</goal>
              <goal>build</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <additionalClasspathElements>
            <additionalClasspathElement>${project.build.outputDirectory}/META-INF/fabric8
            </additionalClasspathElement> (8)
          </additionalClasspathElements>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

1. The Arquillian JUnit Container where tests will be run

2. ??

3. The Arquillian extension to support Arquillian Tests inside Kubernetes

4. The Arquillian extension to support Arquillian Tests inside OpenShift

5. Undertow needs to be excluded as it might cause classpath corruption for spring-web which uses tomcat

6. Kubernetes Client API

7. OpenShift Client API

8. Ensuring that the generated Kubernetes/OpenShift manifests are included in the classpath while running IT tests

Arquillian Configuration

An arquillian configuration file called arquillian.xml needs to be created in $PROJECT_HOME/src/test/resources with the following content,

<?xml version="1.0"?>
<arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xmlns="http://jboss.org/schema/arquillian"
            xsi:schemaLocation="http://jboss.org/schema/arquillian
    http://jboss.org/schema/arquillian/arquillian_1_0.xsd">

  <extension qualifier="@arq.qualifier@"> (1)
    <!-- This is needed to apply the right resources manifest either OpenShift/Kubernetes-->
    <property name="env.config.resource.name">@arq.qualifier@.yml</property> (2)
  </extension>
</arquillian>

1. Whether its kubernetes or openshift extension

2. Apply the right config based on the arq.qualifier

Logging Configuration

To enable debug of Arquillian CUBE and supress unwanted logs, you can add the following content to a file called logback-test.xml inside $PROJECT_HOME/src/test/resources

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <!-- encoders are assigned the type
             ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <include resource="org/springframework/boot/logging/logback/base.xml" />
    <logger name="org.hibernate.validator" level="info" /> <!-- Validator prints a lot of debug messages for Arquillian Cube  -->
    <root level="info">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

Add Greeter Integration Test

package com.redhat.developers;

import com.jayway.restassured.RestAssured;
import io.fabric8.kubernetes.api.KubernetesHelper;
import io.fabric8.kubernetes.api.model.Service;
import io.fabric8.kubernetes.api.model.ServicePort;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.openshift.api.model.Route;
import io.fabric8.openshift.client.OpenShiftClient;
import lombok.extern.slf4j.Slf4j;
import org.arquillian.cube.kubernetes.api.Session;
import org.jboss.arquillian.junit.Arquillian;
import org.jboss.arquillian.test.api.ArquillianResource;
import org.junit.Before;
import org.junit.runner.RunWith;

import java.util.Objects;
import java.util.Optional;

import static org.assertj.core.api.Assertions.assertThat;

@RunWith(Arquillian.class)
@Slf4j
public class GreeterIT extends AbstractGreeterTests {

    @ArquillianResource
    KubernetesClient client;

    @ArquillianResource
    Session session;


    //both application name and service name are same, if you use different modify it
    private String applicationName = "spring-boot-arq-demo";

    @Before
    public void setup() {

        String baseURI;

        /**
         * Checking whether the host is OpenShift Cluster or Raw Kubernetes Cluster and
         * based on that we can use {@link Route} or NodePort
         */
        if (KubernetesHelper.isOpenShift(client)) {
            final Route route = this.client.adapt(OpenShiftClient.class).routes()
                .inNamespace(this.session.getNamespace()).withName(applicationName).get();
            assertThat(route).isNotNull();
            baseURI = String.format("http://%s/api/greet", Objects.requireNonNull(route).getSpec().getHost());

        } else {

            //Construct NodePortIP url
            final String nodeIp = client.getMasterUrl().getHost(); (5)

            final Service service = client.services()
                .inNamespace(this.session.getNamespace())
                .withName(applicationName)
                .get();

            Optional<ServicePort> servicePort = service.getSpec().getPorts().stream()
                .filter(sp -> sp.getPort() == 8080)
                .findFirst();

            assertThat(servicePort.isPresent()).isTrue();

            int nodePort = servicePort.get().getNodePort(); (6)
            assertThat(nodePort).isBetween(30000, 32767); // verify if it is NodePort Range

            baseURI = String.format("http://%s:%d/api/greet", Objects.requireNonNull(nodeIp), Objects.requireNonNull(nodePort));
        }

        log.info("Using {} service  URL:{}", applicationName, baseURI);
        RestAssured.baseURI = baseURI;
    }
}

1. Make the test run with Arquillian Class

2. The Kubernetes/OpenShift deployment name, defaults to maven artifactId

3. Get the handle to OpenShift route that is deployed

4. Set the RestAssured.baseURI to host name of the route

5. Get the Kubernetes/OpenShift cluster IP which is also called NodeIp

6. Get the NodePort mapping to the default applicaiton port of 8080

Run Integration Tests

Kubernetes

./mvnw -Pkubernetes clean verify

OpenShift

./mvnw -Popenshift -Denv.init.enabled=true -DenableImageStreamDetection=true clean verify  (1)

1. Make the image stream available to all namespaces

Deploy

To deploy the application execute the command ./mvnw clean fabric8:deploy. The application deployment status can be checked using the command kubectl get pods -w

Access the Application

To access and test the application execute the following command,

curl $(minikube service spring-boot-arq-demo --url)/api/greet; echo "";

The above command should display a message like Hello

Note	minikube service spring-boot-arq-demo --url is used to get the service url and port via which we can access the application

Cleanup

To clean the deployments from $PROJECT_HOME execute ./mvnw fabric8:undeploy

--END--