TDC Pipeline

Sample application for JavaEE Pipeline as code using Jenkins, Sonar and Docker presentation. Slides available here.

The pipeline code can be found here.

💡	For pipeline DSL syntax highlight I recomment sublime text and this package: https://github.com/bjarneo/jenkinsfile-syntax. To use it just clone the above git repository into .config/sublime-text-3/Packages/ and restart sublime. OR use IntelliJ with GDSL.

Table of Contents

• 1. Running the application
• 2. Running tests
  • 2.1. Integration tests
  • 2.2. Functional tests
  • 2.3. Performance tests
• 3. Running the pipeline
  • 3.1. Jenkins
  • 3.2. Shared library
  • 3.3. Docker
  • 3.4. Sonar
  • 3.5. Quality Gates
  • 3.6. Configure slack
  • 3.7. Create the pipeline job
  • 3.8. Jenkins on docker
  • 3.9. After pipeline execution
  • 3.10. Demo

1. Running the application

❗	You need Docker, Maven and Java installed on your machine.

All commands below must be executed in project root folder.

1. Create a (filesystem) database using Flyway:

   mvn flyway:migrate -P migrations -D db.path=~/db

   After that, a file named cars.h2.db will be created on your user home/db folder.

   ℹ️	your user must have write permissions in ~/db directory.

   💡	If you are on a non unix OS like Windows you must change db.path to an existing directory e.g: db.path=C:/db

2. Create application binary with:

   mvn clean package

   ❗	If you have changed db.path in migrations command you must build the application with the same param: mvn clean package -D db.path=C:/db

3. Now create a docker image using:

   docker build -t tdc-pipeline .

4. And finally run the application:

   docker run -it --name tdc-pipeline -p 8080:8080 -v ~/db:/opt/jboss/db tdc-pipeline

   ℹ️	-v ~/db:/opt/jboss/db maps our host filesystem database (the one we created with Flyway) to docker container home dir so we can exit the container without losing data.

   ℹ️	/opt/jboss/ is the home dir (~/) in wildfly docker container.

5. Access the application at http://localhost:8080/tdc-pipeline

2. Running tests

2.1. Integration tests

Tests in this application use Arquillian, see this guide if you never heard about it.

1. Initialize database

    mvn flyway:clean flyway:migrate -Pmigrations -Ddb.name=cars-test

2. Run the tests

   mvn clean test -Pit-tests

   💡	If you want to run the test using an already running server activate wildfly-remote profile: mvn clean test -Pit-tests -Pwildfly-remote

💡	For it tests the database migration can be run only once to create tables. DataSets are prepared by Arquillian Persistence.

2.2. Functional tests

1. Initialize database

    mvn flyway:clean flyway:migrate -Pmigrations -Ddb.name=cars-ft-test

2. Run the tests

   mvn clean test -Pft-tests

💡	For ft tests the database migration is required to be run every time the tests are run because Arquillian Persistence is not enabled in functional tests.

2.3. Performance tests

Perf tests use Gatling tool.

1. Initialize database

    mvn flyway:clean flyway:migrate -Pmigrations -Ddb.name=cars

2. Deploy the application

3. Run the tests

   mvn clean gatling:execute -Pperf

   💡	By default perf tests run against http://localhost:8080/tdc-pipeline/ but you can change it by passing APP_CONTEXT system property: mvn clean gatling:execute -Pperf -DAPP_CONTEXT=http://localhost:8181/tdc-pipeline/

3. Running the pipeline

This application comes with a Jenkins declarative pipeline. Below are the steps to run the pipeline locally.

⚠️	At least 8GB of RAM is needed on the host machine, 16GB is recommended.

3.1. Jenkins

First thing that is needed is a Jenkins instance, download Jenkins here and start it on port 8080 using:

java -jar jenkins.war

ℹ️

It was tested with Jenkins 2.73.2 with following plugins installed: Pipeline model api (v1.2.2 or greater) [Already comes with Jenkins, you may have to upgrade] Pipeline model definition (v1.2.2 or greater) [Already comes with Jenkins, you may have to upgrade] Last Changes (v1.0.11 or greater) Cucumber living documentation (v1.0.12 or greater) Gatling (v1.2.2 or greater) Slack (v2.3 or greater) Sonar (v2.6.1 or greater) Quality gates (v2.5 or greater)

3.2. Shared library

This pipeline have an example of shared library. Configure https://github.com/rmpestano/tdc-pipeline as a shared lib on Jenkins > Configure system > Global Pipeline Libraries like in image below:

ℹ️	The name must be notify, it’s the name of the function we invoke in pipeline here.

More details on pipeline shared libs, look here.

3.3. Docker

This pipeline depends on Docker, install it according to your operating system as described here.

💡	look for Docker CE (community edition).

💡	To run docker without sudo look here.

3.4. Sonar

The pipeline depends on Sonar, you need to have a Sonar instance running on http://localhost:9000.

An easy way to start Sonar locally is just running it’s docker container:

docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 sonarqube:6.6

Configure Sonar on Jenkins (http://localhost:8080/configure):

3.5. Quality Gates

Quality gates can fail a pipeline if it doesn’t reach the Sonar quality gates conditions, for more details see this post.

Following is Jenkins configuration for quality gates:

3.6. Configure slack

This pipeline is integrated with slack. You’ll need to configure your Jenkins instance to work with the Slack plugin.

In Jenkins configuration search for Global Slack Notifier Settings and set Integration Token with SyQ9NWKGoEorB1g9h2h5xUuy and Base URL with value https://tdc-java.slack.com/services/hooks/jenkins-ci/.

The configuration above will configure jenkins to send messages to the following slack channel: https://tdc-java.slack.com/messages/C7L0N9V0B

ℹ️	Use this invitation link to join the channel above.

For more details on how to integrate your own slack channel and Jenkins see this simple (5 steps) tutorial.

3.7. Create the pipeline job

1. Create a new job;

2. Chose pipeline and give tdc-pipeline as name;

3. On job config check Do not allow concurrent builds option on General section;

4. On Build Triggers section Check pool scm and use * * * * * as value;

5. Finally on Pipeline section select option Pipeline script from scm, chose GIT and use this url: http://github.com/rmpestano/tdc-pipeline;

After saving job configuration the pipeline should run on next minutes.

ℹ️	There is an (manual) approval step before "going to production".

3.8. Jenkins on docker

There is also a jenkins docker container with above plugins installed, here is how to run it:

docker run -it --name jenkins -p 8080:8080 -v "$HOME/.m2":/root/.m2 -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):/usr/bin/docker --net="host" -v ~/.jenkins:/var/jenkins_home rmpestano/jenkins

❗	Docker, Sonar, Quality Gates and Slack setup described above is still needed.

⚠️	When running the pipeline on docker ft-tests stage will fail because it needs chrome on docker, to make it pass you need to run ft-tests with phantomjs: mvn test -Pft-tests -Darquillian.port-offset=120 -Darquillian.port=10110 -Darquillian.browser=phantomjs

3.9. After pipeline execution

The pipeline will generate a lot of evidences about the quality of the build:

1. Sonar analysis along with code coverage can be viewed on http://localhost:9000/dashboard/index/com.github.rmpestano:tdc-pipeline

   

2. Each build VCS diff can be browsed on http://localhost:8080/job/tdc-pipeline/last-changes

   

3. Cucumber living documentation at http://localhost:8080/job/tdc-pipeline/cucumber-living-documentation

   

4. Performance tests report http://localhost:8080/job/tdc-pipeline/gatling

   

ℹ️	Last changes, Living docs and Gatling reports will be available at job level (http://localhost:8080/job/tdc-pipeline) if at least one pipeline succeeds. For failed pipelines you need to access build level, e.g: http://localhost:8080/job/tdc-pipeline/17/last-changes/ (where 17 is the build number).

Two docker containers will be started during the pipeline, one representing the application deployment on QA environment and another which is production:

For QA the app should be available at http://localhost:8282/tdc-pipeline. In Prod env the application is available on port http://localhost:8181/tdc-pipeline.

3.10. Demo

Following is a demo video showing this pipeline: https://www.youtube.com/watch?v=xUlTyzsMPes