Jenkins

🤵🏼‍♂️ Jenkins on Kubernetes powered by Minikube
Examples here can apply beyond local cluster tools

Getting Started
Access · Configuration as Code
Findings

Background

This project serves as a good baseline to build CI/CD environments with Jenkins.

We dive into building Jenkins on Kubernetes. This includes tinkering with the Configuration as Code Jenkins plugin to configure Jenkins controller and agents.

Better yet, all of this can be done from our local machines with Minikube to provide quicker feedback loops.

Built With

Notable resources including, but not limited to:

• Minikube
• Jenkins Helm chart
• Jenkins Docker image
• Jenkins inbound agents
• Helm
• Skaffold

Getting started

Create a batteries included Jenkins environment.

• 🤵🏼‍♂️ Jenkins controller
• 🕶 Jenkins agents
  • Node
  • Gradle
• 👀 Monitoring
  • Grafana
  • Prometheus
  • Kube state metrics
  • Metrics server

# Install a required tooling
brew bundle

# Startup Kubernetes cluster - requires at least 5 GB, 2 CPUs reserved on Docker
make local-cluster

# Deploy everything and watch for changes
make dev

# Fire and forget deployment
make deploy

Access

Create a Minikube tunnel since we've exposed ports to certain services

make tunnel

Jenkins - localhost:8080

Grafana - localhost:3000 - Jenkins dashboard

Prometheus - localhost:9090

Configuration as Code

Experienced Jenkins users rely on groovy init scripts to customize Jenkins and enforce the desired state. Those scripts directly invoke Jenkins API and, as such, can do everything (at your own risk). But they also require you to know Jenkins internals and are confident in writing groovy scripts on top of Jenkins API.

An alternative to using imperative Groovy scripts to setup Jenkins. We can use declarative configuration through the the configuration as code (CaC) Jenkins plugin.

Configuration is controlled at the Helm chart values where they get templated into Kubernetes ConfigMaps. Jenkins has a sidecar that auto-reloads configuration upon each update (Helm chart release).

Jenkins configuration

Edit jenkins.config in Helm chart values.yaml files.

jenkins:
  config:
    # Configuration as code config example
    security:
      apiToken:
        creationOfLegacyTokenEnabled: false
        tokenGenerationOnCreationEnabled: false
        usageStatisticsEnabled: false
    unclassified:
      location:
        adminAddress: 

To view existing Jenkins configuration, from Jenkins home, go to Configuration as Code -> Documentation

We can also copy examples.

Agents

Jenkins agents can be added through CasC through Helm chart values files. Helm templates configuration

• Agent plugin reference
• Example agent Docker images

values.yaml

jenkins:
  agents:
    default:
      label: "default"
      containers:
      - name: "jnlp"
        image: "jenkins/inbound-agent:4.11.2-4"
    example-gradle-monogo:
      label: "gradle"
      containers:
      - name: "jnlp"
        image: "johnnyhuy/jenkins-gradle:latest"
        privileged: true
        envVars:
          - envVar:
              key: "EXAMPLE"
              value: "foobar"
      - name: "mongo"
        image: "mongo:latest"
        args: "^${computer.jnlpmac} ^${computer.name}"
      idleMinutes: 0
      instanceCap: 2147483647
      label: default
      nodeUsageMode: EXCLUSIVE
      podRetention: Never
      showRawYaml: true
      serviceAccount: default
      slaveConnectTimeoutStr: 100
      yamlMergeStrategy: override
    example-node-mysql:
      label: "node"
      containers:
      - name: "jnlp"
        image: "johnnyhuy/jenkins-node:latest"
      - name: "mysql"
        image: "mysql:latest"

Environment variables

We can plug in environment variables into CaC config. This is done with the jenkins.env Helm chart values.

values.yaml

jenkins:
  # Jenkins master environment variables
  controller:
    containerEnv:
      - name: SPICY_SECRET
        value: spice123
  
  # Configuration as code config
  config:
    credentials:
      system:
        domainCredentials:
          - credentials:
            - string:
              id: "spicy-secret"
              secret: ${SPICY_SECRET}

Secrets

Secrets can be loaded through Kubernetes secrets, by persisting CaC YAML configuration in secret. This file needs to be in the charts/jenkins/templates folder of the Jenkins helm chart.

We'll need to add the jenkins-jenkins-config: "true" label on the Kubernetes secret to allow Jenkins to auto-reload.

charts/jenkins/template/spicy-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: spicy-secret
  labels:
    # Jenkins detects and auto-reloads
    jenkins-jenkins-config: "true"
stringData:
  # Filename to be loaded into Jenkins
  spicy-casc-config.yaml:
    # Follows the same CasC format
    config:
      credentials:
        system:
          domainCredentials:
            - credentials:
              - string:
                id: "spicy-secret"
                secret: spicy123

Source

Findings

Jenkins unstable on ARM

If we're using Jenkins on ARM systems like M1 Macs, it's recommended to use ARM64 Docker images to avoid JVM crashes. Thankfully Jenkins have provided newer images with ARM support. However, older images like jenkins/jenkins:2.263.4-lts remain on AMD64.

GitHub issue

Where and what is JNLP?

Java Network Launch Protocol aka JNLP is the protocol Jenkins agents historically use communicate between master and agent instances. JNLP has been deprecated in version 9+ of Java. Nowadays it's recommended to use either TCP or WebSockets as a replacement.