jenkins-pipeline-scripts

Table of contents

// spell-checker:disable

• Initialize
• Usage
• Docker
• Kubernetes
• Graph dependency
  • sonar
• Folder Structure Conventions
  • A typical top-level directory layout
• Update documentation
• mega-linter
• Check secret
• Update README.md
  • npm-groovy-lint groovy formating for Jenkinsfile

// spell-checker:enable

NOTE: This repository is being deprecated internally at Nabla and hence will receive few updates going forward.

Initialize

direnv allow
pyenv install 3.10.4
pyenv local 3.10.4
python -m pipenv install --dev --ignore-pipfile
direnv allow
pre-commit install

Usage

This repository contains helper functions and classes to be used with the Jenkins Pipeline Plugin. This repository is used on http://albandrieu.com/jenkins/ and other Jenkins instances managed by Nabla.

Below feature have been removed on purpose

• tee

To use this library from your Jenkinsfile, make sure you have installed the GitHub Organization Folder in version 1.5 or later, then start off with:

@Library('jenkins-pipeline-scripts@master') _

OR (if jenkins is managing the version, the preferred way)

@Library('jenkins-pipeline-scripts') _

In jenkins

Library

• Set Load implicitly to false
• Allow default version to be overridden to true
• Include @Library changes in job recent changes

Behaviour

Within Repository -> Discover branches Additional -> Wipe out repository & force clone

See Extending with Shared Libraries for more information on Jenkins pipeline extensions.

See also Best Practices For Pipeline Code

Run test

./mvnw -Dtest=TestSharedLibrary test

Docker

Build and Run

./scripts/docker-build.sh

or

$ docker build -t groovy-test .
#You can reproduce issue `Conflicting module versions. Module [groovy-all is loaded in version 2.4.8 and you are trying to load version 2.4.12` with
$ docker run -it groovy-test:latest

Kubernetes

cheatsheet conventions

Install microk8s

Make docker image available to microk8s

$docker save nabla/jenkins-pipeline-scripts:1.0.3 > jenkins.tar
$microk8s ctr image import jenkins.tar

$microk8s ctr images ls

Create jenkins namespace

$k apply -f jenkins-namespace.yaml

Add deployment

k config get-contexts
k config use-context microk8s

$ #k delete pods --all
#k delete -f jenkins-deployment.yaml
$k apply -f jenkins-deployment-local.yaml -n jenkins

$k get deployments jenkins-master -n jenkins --watch
$k describe pod -n jenkins | grep jenkins
#stop deployement
$k scale --replicas=0 deployment/jenkins-master -n jenkins

Copy volume data

cp -r /jenkins/* /mnt/jenkins
chown -R albandrieu:docker /mnt/jenkins

Add service

$k create -f jenkins-service.yaml -n jenkins
$k get service -n jenkins
$k logs jenkins-master-7b49df974d-kzlrg -n jenkins

Check http://127.0.0.1:32082/

Check nfs

Add PersistentVolume

$k create -f jenkins-pvc.yaml -n jenkins
$k get pvc pvc-jenkins-home -n jenkins
$k create -f jenkins-volume.yaml -n jenkins
$k get pv jenkins-pv-volume -n jenkins

#k describe pv  -n jenkins

$k exec -it jenkins-master-7b49df974d-kzlrg -n jenkins -- /bin/bash

service-account-tokens

$k create serviceaccount jenkins-account -n jenkins
$k get serviceaccounts jenkins-account -o yaml  -n jenkins
$k get secret jenkins-token-2dmg9 -o yaml  -n jenkins

set-up-jenkins-in-a-kubernetes-cluster

$k apply -f jenkins-resourcequota.yaml -n jenkins
$k apply -f jenkins-role.yaml -f jenkins-serviceaccount.yaml -f jenkins-rolebinding.yaml -n jenkins
$k apply -f jenkins-deployment.yaml -n jenkins

$k get pods -n jenkins
$k -n jenkins port-forward jenkins-master-7b49df974d-kzlrg 8080:8080

$k get svc -n jenkins

See dns-debugging-resolution

$k get pods --namespace=kube-system -l k8s-app=kube-dns

$k -n kube-system describe configmap/coredns
$#Add 10.21.200.3 10.25.200.3 before google DNS
$k -n kube-system edit configmap coredns

Add k8s jenkins-account to jenkins set-up-jenkins-in-a-kubernetes-cluster

$kubectl get secret $(kubectl get sa jenkins-account -n jenkins -o jsonpath={.secrets[0].name}) -n jenkins -o jsonpath={.data.token} | base64 --decode
$kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " "
$kubectl get secret $(kubectl get sa jenkins-account -n jenkins -o jsonpath={.secrets[0].name}) -n jenkins -o jsonpath={.data.'ca\.crt'} | base64 --decode

Add k8s proxy

k get pod -n jenkins
k port-forward -n jenkins jenkins-master-6868bb694-m4jhb 8080:8080
http://localhost:8080/

TODO : Have proper DNS service

http://jenkins-master.jenkins.svc.cluster.local mon-service.mon-namespace.svc.cluster.local

========================

See Extending environment variables with Shared Libraries

Graph dependency

• graphviz
• webgraphviz

dot -Tps draftStage.gv -o draftStage.ps
dot -Tpng draftStage.gv -o draftStage.png
dot -Tsvg draftStage.gv -o draftStage.svg

terraform graph | grep -v -e 'meta' -e 'close' -e 's3' -e 'vpc' -e 'expand' | dot -Tpng > terraform.png

maven

Maven mvn clean deploy will be started by default if pom.xml file exists and a sonar scan will be started using sonar-maven-plugin...

sonar

Sonar will be started by default on maven projects.

sonar-maven-plugin. is the preferred way to run sonar. Otherwise sonarscanner can be used.

If sonar-project.properties file exists a sonarscanner can be started... sonarscanner is the preferred way for C++/ObjectiveC and Python projects.

Folder Structure Conventions

Folder structure options and naming conventions for software projects

A typical top-level directory layout

.
├── docs                    # Documentation files (alternatively `doc`)
docker                      # Deprectated, using packs/Dockerfile instead
docker-compose              # Put docker-compose files
├── src                     # Source files (alternatively `lib` or `app`)
├── resources               # Resources for jenkins
├── vars                    # Groovy scripts for jenkins
bower.json                  # Bower not build directly, using maven instead
Dockerfile                  # Deprectated, using packs/Dockerfile instead
Jenkinsfile
Jenkinsfile-checkmarx       # Will run Checkmarx scan
Jenkinsfile-aqua            # Will run WhiteSource. Aqua as standalone scan
package.json                # Nnpm not build directly, using maven instead
pom.xml                     # Will run maven clean install
.pre-commit-config.yaml
requirements.testing.txt    # Python package used for test and build only
requirements.txt            # Python package used for production only
tox.ini
sonar-project.properties    # Will run sonar standalone scan
LICENSE
CHANGELOG.md
README.md
├── target                  # Compiled files (alternatively `dist`) for maven
└── test                    # Automated tests

.
├── ...
├── test                    # Test files
│   ├── e2e                 # End-to-end, integration tests (alternatively `e2e`)
│   karma.conf.js
│   ├── postman             # API tests for postman
│   protractor.conf.js
│   └── spec                # Karma unit tests
└── ...

docker-compose irectory is used only to test project in jenkins
.
├── ...
├── docker-compose          # Docker compose files
│   docker-compose.yml
│   docker-compose.dev.yml  # For developper (with port open)
│   docker-compose.prod.yml # For production such as jenkins
│   docker-compose.test.yml # For tests such as newman, robot
└── ...

docker irectory is used only to build project
.
├── ...
├── docker                  # Docker files used to build project
│   ├── centos7             # End-to-end, integration tests (alternatively `e2e`)
│   ├── ubuntu18            # End-to-end, integration tests (alternatively `e2e`)
│   └── ubuntu20
│       Dockerfile          # File to build
│       config.yaml         # File to run CST
└── ...

.
├── ...
├── docs                    # Documentation files
│   ├── index.rst           # Table of contents
│   ├── faq.rst             # Frequently asked questions
│   ├── misc.rst            # Miscellaneous information
│   ├── usage.rst           # Getting started guide
│   └── ...                 # etc.
└── ...

.
├── ...
├── packs                    # Files used to build docker image and chart
│   config.yaml              # File to run CST
│   Dockerfile               # File to build docker image
│   └── newman               # Name of the helm chart
│       └── charts
│           Chart.yaml
│           README.md
│           └── templates
│               deployment.yaml
│               └── tests
│                   test-connection.yaml
│           values.yaml
└── ...

Update documentation

mvn gplus:groovydoc
# or
mvn site

Maven site and groovy doc will be published with jenkins build

README.md then CHANGELOG.md are the default entry points.

mega-linter

npx mega-linter-runner

Check secret

npx @secretlint/quick-start "**/*"

Update README.md

• github-markdown-toc
• With github-markdown-toc

npm install --save markdown-toc
markdown-toc README.md
markdown-toc CHANGELOG.md  -i

pre-commit install
git add README.md
pre-commit run markdown-toc

npm-groovy-lint groovy formating for Jenkinsfile

Tested with nodejs 12 and 16 on ubuntu 20 and 21 (not working with nodejs 11 and 16)

npm install -g npm-groovy-lint@8.2.0
npm-groovy-lint --format
ls -lrta .groovylintrc.json