Skip to content

Latest commit

 

History

History
152 lines (104 loc) · 17.8 KB

automation.md

File metadata and controls

152 lines (104 loc) · 17.8 KB
Raw

Table of Contents

Introduction

This document describes the overall automation strategy for the Quarkus superheroes services and how all the automation behind it works. There is a lot of automation and resource generation that happens "behind the scenes" when changes are pushed to the GitHub repo.

GitHub action automation

There are 3 GitHub action workflows that run when code is pushed: Basic building and testing, Build and push container images, and Create deploy resources.

Basic building and testing workflow

The Basic building and testing workflow is a "sanity check". It is required to pass before pull requests can be merged.

It runs whenever code is pushed to the main branch as well as upon any pull requests.

The workflow can also be triggered manually.

It runs ./mvnw clean verify on the event-statistics, rest-fights, rest-heroes, and rest-villains applications on both Java 11 and 17.

Build and push container images workflow

The Build and push container images workflow does pretty much what it sounds like: builds and pushes container images.

It only runs on pushes to the main branch after successful completion of the above Basic building and testing workflow.

The workflow can also be triggered manually.

It consists of 5 jobs:

If any step in any of the jobs fail then the entire workflow fails.

This image is a visual of what the workflow consists of:

build-push-images-workflow

Build and test JVM container images job

  1. Builds JVM container images for the event-statistics, rest-fights, rest-heroes, and rest-villains applications on both Java 11 and 17 using the Quarkus Docker container image extension.

    • Each container image created has 2 tags: {{app-version}}-quarkus-{{quarkus-version}}-java{{java-version}} and java{{java-version}}-latest.
      • Replace {{app-version}} with the application version (i.e. 1.0).
      • Replace {{quarkus-version}} with Quarkus version the application uses (i.e. 2.6.2.Final).
      • Replace {{java-version}} with the Java version the application was built with (i.e. 11 or 17).

  2. Once each JVM container image is built (4 applications x 2 JVM versions = 8 total JVM container images), each JVM container is launched and the integration tests are run against the image.

Build and test native container images job

Runs in parallel with the Build and test JVM container images and Build UI image jobs.

  1. Builds native executables for the event-statistics, rest-fights, rest-heroes, and rest-villains applications on both Java 11 and 17 using Mandrel.

  2. Once each native executable is created (4 applications x 2 JVM versions = 8 total native executables), each native executable is launched and the integration tests are run against the executable.

  3. Once tested, a container image is created from each native executable using the Quarkus Docker container image extension.

    NOTE: The native executable is not re-created. It is re-used from the previous step.

    • Each container image created has 2 tags: {{app-version}}-quarkus-{{quarkus-version}}-native-java{{java-version}} and native-java{{java-version}}-latest.
      • Replace {{app-version}} with the application version (i.e. 1.0).
      • Replace {{quarkus-version}} with Quarkus version the application uses (i.e. 2.6.2.Final).
      • Replace {{java-version}} with the Java version the application was built with (i.e. 11 or 17).

  4. Once each native executable container image is built (8 total native executable container images), each container is launched and the integration tests are run against the image.

Build UI image job

Runs in parallel with the Build and test JVM container images and Build and test native container images jobs.

Builds the ui-super-heroes container image with the following 2 tags: {{app-version}} and latest.

  • Replace {{app-version}} with the application version (i.e. 1.0).

Push application container images job

Runs after successful completion of the Build and test JVM container image, Build and test native container image, and Build UI image jobs and in parallel with the Push UI image job.

All of the container images created in the Build and test JVM container image and Build and test native container image jobs (16 total container images/32 tags) are pushed to https://quay.io/quarkus-super-heroes.

Push UI image job

Runs after successful completion of the Build and test JVM container image, Build and test native container image, and Build UI image jobs and in parallel with the Push application container images job.

Pushes the ui-super-heroes container image with the following 2 tags: {{app-version}} and latest.

  • Replace {{app-version}} with the application version (i.e. 1.0).

Create deploy resources workflow

The Create deploy resources workflow is responsible for generating all of the application resources, described in a later section of this document.

It only runs on pushes to the main branch after successful completion of the Build and push container images workflow.

The workflow can also be triggered manually.

All generated resources are subsequently pushed back into the repo by the action in a single commit.

Application resource generation

The resources and descriptors in the root deploy directory as well as in each individual project's deploy directory (event-statistics, rest-fights, rest-heroes, rest-villains, and ui-super-heroes) are used for deploying the entire system or subsets of it into various environments (i.e. Docker compose, OpenShift, Minikube, Kubernetes, etc).

Resources in these directories are generated by the Create deploy resources workflow mentioned in the previous section. Any manual changes made to anything in any of those directories will be overwritten by the workflow upon its next execution.

Kubernetes (and variants) resource generation

Kubernetes resources are generated into a deploy/k8s directory, either in the project root directory or in each individual project's directory.

Quarkus projects

Each Quarkus project (event-statistics, rest-fights, rest-heroes, rest-villains) uses the Quarkus Kubernetes extension to generate Kubernetes and KNative manifests, the Quarkus Minikube extension to generate Minikube manifests, and the Quarkus OpenShift extension to generate OpenShift manifests.

These extensions generate the manifests needed for the application itself but not for any other services. These extensions can also incorporate additional resources by placing additional resources in each project's src/main/kubernetes directory.

The generate-k8s-resources.sh script loops through all versions of each application (Java versions 11 & 17, both JVM and native - 16 total versions) and merges the contents of files these extensions generate and places them into each project's deploy/k8s directory as well as the respective files in the root deploy/k8s directory.

The generate-k8s-resources.sh script additionally creates the Prometheus descriptors within the root deploy/k8s directory for each Kubernetes variant platform.

In the rest-fights project, the generate-k8s-resources.sh script additionally copies in generated resources from the rest-heroes and rest-villains projects into the all-downstream.yml files in the deploy/k8s directory of the rest-fights project.

UI Project

Since the ui-super-heroes project isn't a Quarkus project it therefore doesn't have a way to generate the Kubernetes manifests.

The generate-k8s-resources.sh script implements the same pattern for this project. The ui-super-heroes project has a src/main/kubernetes directory containing the Kubernetes manifests.

The generate-k8s-resources.sh script merges the contents of these files into the files in the ui-super-heroes project's deploy/k8s directory as well as all the files in the root deploy/k8s directory.

Docker compose resource generation

Docker compose resource generation follows a similar pattern as the Kubernetes resource generation.

Docker compose resources are generated into a deploy/docker-compose directory, either in the project root directory or in each individual project's directory.

Quarkus projects

Each Quarkus project (event-statistics, rest-fights, rest-heroes, rest-villains) contains a src/main/docker-compose directory.

Inside this directory are a set of yaml files with a particular naming convention: infra.yml, java{{java-version}}.yml, and native-java{{java-version}}.yml. Each of these files contains what we are calling Docker compose snippets. These snippets aren't a complete Docker compose file on their own. Instead, they contain service definitions that will ultimately end up inside the services block in a Docker compose file.

NOTE: The rest-fights project has an additional infra-downstream.yml file.

This table describes the different files that can be found inside a project's src/main/docker-compose directory.

File name Description
infra.yml Any infrastructure definitions that are needed by the application. Definitions in here a re-used for each version of the application (i.e. JVM 11, JVM 17, Native Java 11, Native Java 17).
infra-downstream.yml Only used in the rest-fights project. Contains infrastructure definitions needed by the rest-fights application as well as all of its downstream applications (rest-heroes and rest-villains). In this configuration, only a single database container is used containing multiple databases.
java{{java-version}}.yml Definition for the JVM version of application itself for a particular java version, denoted by {{java-version}}.
native-java{{java-version}}.yml Definition for the native image version of the application itself, built with a particular java version, denoted by {{java-version}}.

The generate-docker-compose-resources.sh script loops through all versions of each application (Java versions 11 & 17, both JVM and native - 16 total versions) and merges contents of these files from each project's src/main/docker-compose directory into each project's deploy/docker-compose directory as well as the respective files in the root deploy/docker-compose directory.

The generate-docker-compose-resources.sh script additionally creates the Prometheus compose file (prometheus.yml) within the root deploy/docker-compose directory.

UI Project

Like the Quarkus projects, the ui-super-heroes project also has a src/main/docker-compose directory. There is only a single app.yml file containing the Docker compose snippet.

The generate-docker-compose-resources.sh script merges the contents of this file into the ui-super-heroes project's deploy/docker-compose/app.yml file as well as all the files in the root deploy/docker-compose directory.