This repo contains all the scripts and code (as submodules) needed to create a demo that supports this walkthrough that compares and contrasts traditional Jenkins pipelines to Tekton. Concepts covered include:
| Jenkins | OpenShift Pipelines |
|---|---|
Blue Ocean |
Pipeline Console |
OpenShift Plugin / Kubernetes PodTemplate |
|
Webhooks |
Tekton Events |
S2I binary integration |
S2I binary integration |
|
Tip
|
There is a .devcontainer in this repo so if you have VisualStudio Code with remote extensions enabled, you can open repo folder in a container that has all the tools necessary to run this demo already installed. |
Before installing the demo you need to create a Slack webhook for Tekton to post notifications to.
-
If you’ve done this before and have a webhook URL, you can skip to the demo installation section with that URL
-
If you’ve not done this already, see the next section for details on how to create one
The CI/CD toolchain that is setup by the create demo script will prompt to slack when a deployment to dev is completed. In order for this to work, you need to:
-
setup a slack app (bot) and obtain a bot specific webhook to be able to post.
-
Then associate that bot with a channel so that you can see the posts
-
Setup interactivity in the bot so that it can post back to your cluster when you click the button on the message
Apps can be created from here
You will need a webhook in order to call your app/bot from outside. This can be generated by going to the "Incoming Webhooks" part of the app
It is this highlighted hook that you pass into the create-demo.sh
The Tekton pipeline will be setup to call out to your bot. Assuming your bot is associated with a channel, you’ll see something like the following posted:
In order to get the highlighted button to do something you must update the Request URL untio the bot’s Interactivity and Shortcuts to point to the external route for the slack-webhook.
You can get the route to the webhook by issuing this command:
echo "http://$(oc get route webhook-slack -n ${PROJECT_PREFIX}-cicd -o jsonpath='{.spec.host}')/"
Then turn interactivity on and paste that route in the slot indicated
|
Note
|
The script referenced is run during the Walkthrough. This just gives a little more info if needed for troubleshooting |
A good example on how to get this running with SpringBoot is here. This demo is heavily based on it. this is also a good article about a number of Tekton features used in this demo (such as workspaces) including some others that aren’t yet being used (e.g. conditions)
Run the following script to setup the entire demo which includes a Tekton as well as a Jenkins pipeline for building a petclinic tomcat app. Pipelines will be created a project called $PROJECT_PREFIX-cicd (where $PROJECT_PREFIX is as defined below). The project will be created if it doesn’t exist already.
Finally, to minimize compatibility issues when installing from different OSes, the installation process should be run from inside a container.
-
If you are not using VSCode Remove Containers then follow these steps to get a container running in which you’ll issue all the following commands
-
Change to the root of this repository
-
Run the following command
docker run -it -v ~/.kube:/home/jboss/.kube -v ~/.oh-my-zsh:/home/jboss/.oh-my-zsh -v $(pwd):/workspaces/tekton-comparison-demo -w /workspaces/tekton-comparison-demo quay.io/mhildenb/tekton-demo-shell:latest /bin/zsh
-
-
Setup the following variables in your environment (or in the currently running shell):
-
PROJECT_PREFIXthis is set topetclinicby default, but you can override this to change the names of projects created -
SLACK_WEBHOOKthis is the webhook url to be used for triggering a "prompt to staging?" message to slack -
USERthis should be set to your RedHat access user name (used to create a pull secret for registry.redhat.io) -
PASSWORDthis should be set to your RedHat access password (used to create a pull secret for registry.redhat.io)
-
-
Run the following command to install the whole demo
-
NOTE: The
-iflag indicates that all pre-requisites should be installed (such as OpenShift Pipelines)
-
$DEMO_HOME/scripts/create-tekton-cicd.sh install --project-prefix $PROJECT_PREFIX -i --user $USER --password "$PASSWORD" --slack-webhook-url "$SLACK_WEBHOOK"
You can test access to a MySQL database in an OpenShift cluster using the Adminer image.
-
First, setup port forwarding to the service in question (assuming a petclinic based service as shown in the walkthrough)
oc port-forward svc/petclinic-mysql 3306:3306
-
Then, in another shell, run the
Adminerimage and have it port forward to 8080. NOTE: Assumes you are running on a Mac using Docker for Mac, this is where thedocker.for.mac.localhoststuff comes fromdocker run -p 8080:8080 -e ADMINER_DEFAULT_SERVER=docker.for.mac.localhost adminer:latest
-
From the
Adminerweb page, login as root (using whatever secret was used in the setup of the cluster). You can then run arbitrary commands. Here are the commands you can run to grant access to a user pc to a newly created petclinic database (from here)CREATE USER 'pc'@'%' IDENTIFIED BY 'petclinic'; CREATE DATABASE petclinic; GRANT ALL PRIVILEGES ON petclinic.* TO 'pc'@'%';
-
Or instead, you run SQL commands from the local command line
oc run mysql-client --image=mysql:5.7 --restart=Never --rm=true --attach=true --wait=true \ -- mysql -h petclinic-mysql -uroot -ppetclinic -e "CREATE USER 'pc'@'%' IDENTIFIED BY 'petclinic'; \ CREATE DATABASE petclinic; \ GRANT ALL PRIVILEGES ON petclinic.* TO 'pc'@'%';"
-
If a pipeline fails and the logs are not enough to determine the problem, you can use the fact that every task maps to a pod to your advantage.
Let’s say that the task "unit-test" failed in a recent run.
-
First look for the pod that represents that run
$ oc get pods NAME READY STATUS RESTARTS AGE petclinic-dev-pipeline-tomcat-dwjk4-checkout-vnp7v-pod-f8b5j 0/1 Completed 0 3m18s petclinic-dev-pipeline-tomcat-dwjk4-unit-tests-5pct2-pod-4gk46 0/1 Error 0 3m petclinic-dev-pipeline-tomcat-kpbx9-checkout-t78sr-pod-qnfrh 0/1 Error 0
-
Then use the
oc debugcommand to restart that pod to look around:$ oc debug po/petclinic-dev-pipeline-tomcat-dwjk4-unit-tests-5pct2-pod-4gk46 Starting pod/petclinic-dev-pipeline-tomcat-dwjk4-unit-tests-5pct2-pod-4gk46-debug, command was: /tekton/tools/entrypoint -wait_file /tekton/downward/ready -wait_file_content -post_file /tekton/tools/0 -termination_path /tekton/termination -entrypoint ./mvnw -- -Dmaven.repo.local=/workspace/source/artefacts -s /var/config/settings.xml package If you don't see a command prompt, try pressing enter. sh-4.2$
Sometimes pipelines fail to run because the workspace volume cannot be mounted. Looks like to root cause has to do with the underlying infra volume being deleted out from underneath a PersistentVolume. If you have pipelines that are timing out due to pods failing to run (usually you won’t get any log stream), take a look at the events on the pod and see if you notice these kind of mounting errors:
This can usually be remedied by deleting the PVC, but finalizers keep PVCs from being deleted if a pod has a claim.
If you run into this issue, cancel the affected pipeline (otherwise the PVC won’t be able to be deleted) and either run the following command or see the additional details that follow
scripts/util-recreate-pvc.sh pipeline-source-pvc.yaml
To see all the claims on a PVC, look for the Mounted By section of the output of the following describe command (for pvc/maven-source-pvc):
oc describe pvc/maven-source-pvc
To delete all pods that have a claim on the pvc pvc/maven-source-pvc:
oc delete pods $(oc describe pvc/maven-source-pvc | grep "Mounted By" -A40 | sed "s/ //ig" | sed "s/MountedBy://ig")
You can use the oc run command to run certain containers in a given project as a service account.
For instance, this command can be used to see what kind of permissions the builder service account has to view other projects (e.g. access to remote imagestreams)
oc run test3 --image=quay.io/openshift/origin-cli:latest --serviceaccount=builder -it --rm=true
If the tomcat extension fails to run, you can attempt the following:
-
remote any old tomcat files
rm -f /opt/webserver/webse*
-
Attempt to readd tomcat to /opt/webserver per the instructions above
-
if that still doesn’t work, rebuild container.
-
If all else fails, you can run the tomcat server locally.
Tekton allows for EventListeners, TriggerTemplates, and TriggerBindings to allow a git repo to hit a webhook and trigger a build. See also here. To get basic triggers going for both gogs and github run the following:
|
Note
|
For an example of triggers working with Tekton, see files in the template directory of this repo |
|
Note
|
You may also want to consider this tekton dashboard functionality |
YAML resources for the pipeline created for this demo can be found in these locations:
-
Resources: $DEMO_HOME/kube/tekton/resources
-
Triggers: $DEMO_HOME/kube/tekton/triggers
If the trigger doesn’t appear to fire, then check the logs of the pod that is running that represents the webhook. The probably is likely in the PipelineRun template.
You can see limited logs in the Tekton UI, but if you want the full logs, you can access these from the command line using the tkn command
# Get the list of pipelineruns in the current project tkn pipelinerun list # Output the full logs of the named pipeline run (where petclinic-deploy-dev-run-j7ktj is a pipeline run name ) tkn pipelinerun logs petclinic-deploy-dev-run-j7ktj
To output the logs of a currently running pipelinerun (pr) and follow them, use:
tkn pr logs -L -f
Pipeline setup is inspired by this post here
From the instructions here
-
Install subscription (in openshift operators)
oc apply -f $DEMO_HOME/kube/tekton/tekton-subscription.yaml
-
Optionally install tekton dashboard (for visualization) as per here
oc apply -f $DEMO_HOME/kube/tekton/openshift-tekton-dashboard-release.yaml --validate=false oc wait --for=condition=Available deployment/tekton-dashboard -n openshift-pipelines
-
Then you can open the dashboard by hitting this URL. It will authenticate using OpenShift oauth
echo "https://$(oc get route tekton-dashboard -o jsonpath='{.spec.host}' -n openshift-pipelines)/" -
When the operator has finished installing, it will install a pipeline service account in all projects that have sufficient permissions to build stuff. There is also a centralized openshift-pipelines project that holds pipeline supporting pods.
NoteSee also tips and tricks from the pipelines tutorial
The Slack integration that you find in this webhook includes a webhook Interceptor that translates the form urlencoded output of the slack callback into a json document.