feat: enhance jenkins and jenkins-k8s-pipeline plugin

[aFlyBird0]

Pre-Checklist

Note: please complete ALL items in the following checklist.

• I have read through the CONTRIBUTING.md documentation.
• My code has the necessary comments and documentation (if needed).
• I have added relevant tests

Description

1. Refactor jenkins plugin, use gojenkins pkg. See 🍀 Proposal: Use gojenkins to operate jenkins #891 for more info.
2. Change the auth of jenkins-k8s-pipeline plugin. Use username-password rather than username-token. So that we can run both plugins jenkins and jenkins-k8s-pipeline plugins at the same time. See 🍀 Proposal: Change the way of jenkins auth to combine jenkins and pipeline plugin #892 for more info.
3. Complete Read, Update, Delete interface of jenkins-k8s-pipeline plugin. See 🍀 Proposal: Enhance jenkins-k8s-pipeline plugin, complete Read, Update, Delete interface #893 for more info.

Related Issues

close #891
close #892
close #893

New Behavior (screenshots if needed)

Let's take a look at how to run the jenkins and jenkins-k8s-pipeline plugins in an up-to-date way. All you need is a k8s environment and a pre-set GITHUB_TOKEN environment variable.

❯ echo "first, you should set GITHUB_TOKEN env"
first, you should set GITHUB_TOKEN env
❯ echo "then, you should have a K8s cluster. If your K8s is 'K8s in docker', please remeber to set up the network."
then, you should have a K8s cluster. If your K8s is 'K8s in docker', please remeber to set up the network.
❯ cat ~/tmp/k8s/kind-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 32000
    hostPort: 32000
    listenAddress: "0.0.0.0"
    protocol: tcp
❯ cd ~/tmp/k8s && kind create cluster --config kind-cluster.yaml --image=kindest/node:v1.22.0 --name=dev
Creating cluster "dev" ...
 ✓ Ensuring node image (kindest/node:v1.22.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-dev"
You can now use your cluster with:

kubectl cluster-info --context kind-dev

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
❯ cd ~ && pwd
/Users/lhp
❯ docker exec -it dev-control-plane bash
root@dev-control-plane:/# mkdir -p /Users/lhp/data/jenkins-volume/
root@dev-control-plane:/# chown -R 1000:1000 /Users/lhp/data/jenkins-volume/
root@dev-control-plane:/# exit
exit
❯ cd ~/Code/Go/opensource/devstream
❯ cat jenkins-all.yaml
---
# core config
varFile: "" # If not empty, use the specified external variables config file
toolFile: "" # If not empty, use the specified external tools config file
state: # state config, backend can be local or s3
  backend: local
  options:
    stateFile: devstream-jenkins-all.state

---

tools:
  # name of the tool
  - name: jenkins
    # id of the tool instance
    instanceID: default
    # format: name.instanceID; If specified, dtm will make sure the dependency is applied first before handling this tool.
    dependsOn: [ ]
    # options for the plugin
    options:
      # if true, the plugin will use hostpath to create a pv named `jenkins-pv`
      # and you should create the volumes directory manually, see plugin doc for details.
      test_env: true
      # need to create the namespace or not, default: false
      create_namespace: true
      # Helm repo information
      repo:
        # name of the Helm repo
        name: jenkins
        # url of the Helm repo
        url: https://charts.jenkins.io
      # Helm chart information
      chart:
        # name of the chart
        chart_name: jenkins/jenkins
        # release name of the chart
        release_name: dev
        # k8s namespace where jenkins will be installed
        namespace: jenkins
        # whether to wait for the release to be deployed or not
        wait: true
        # the time to wait for any individual Kubernetes operation (like Jobs for hooks). This defaults to 5m0s
        timeout: 5m
        # whether to perform a CRD upgrade during installation
        upgradeCRDs: true
        # custom configuration. You can refer to [Jenkins values.yaml](https://github.com/jenkinsci/helm-charts/blob/main/charts/jenkins/values.yaml)
        values_yaml: |
          persistence:
            # for prod env: the existent storageClass, please change it
            # for test env: just ignore it, but don't remove it
            storageClass: jenkins-pv
          serviceAccount:
            create: false
            name: jenkins
          controller:
            serviceType: NodePort
            nodePort: 32000
            additionalPlugins:
              # install "GitHub Pull Request Builder" plugin, see https://plugins.jenkins.io/ghprb/ for more details
              - ghprb
              # install "OWASP Markup Formatter" plugin, see https://plugins.jenkins.io/antisamy-markup-formatter/ for more details
              - antisamy-markup-formatter
            # Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
            enableRawHtmlMarkupFormatter: true
            # Jenkins Configuraction as Code, refer to https://plugins.jenkins.io/configuration-as-code/ for more details
            # notice: All configuration files that are discovered MUST be supplementary. They cannot overwrite each other's configuration values. This creates a conflict and raises a ConfiguratorException.
            JCasC:
              defaultConfig: true
              # each key-value in configScripts will be added to the ${JENKINS_HOME}/casc_configs/ directory as a file.
              configScripts:
                # this will create a file named "safe_html.yaml" in the ${JENKINS_HOME}/casc_configs/ directory.
                # it is used to configure the "Safe HTML" plugin.
                # filename must meet RFC 1123, see https://tools.ietf.org/html/rfc1123 for more details
  - name: jenkins-pipeline-kubernetes
    # id of the tool instance
    instanceID: default
    # format: name.instanceID; If specified, dtm will make sure the dependency is applied first before handling this tool.
    dependsOn: [ "jenkins.default" ]
    # options for the plugin
    options:
      jenkins:
        # jenkinsUrl, format: hostname:port
        url: ${{jenkins.default.outputs.jenkinsURL}}
        # jenkins user name, default: admin
        user: admin
        # jenkins password, you have 2 options to set the password:
        # 1. use outputs of the `jenkins` plugin, see docs for more details
        # 2. set the `JENKINS_PASSWORD` environment variable
        # 3. fill in the password in this field(not recommended)
        # if all set, devstream will read the password from the config file first.
        password: ${{jenkins.default.outputs.jenkinsPasswordOfAdmin}}
        # jenkins job name, mandatory
        jobName: jenkins-plugin-test
        # path to the pipeline file, relative to the git repo root directory. default: Jenkinsfile
        pipelineScriptPath: Jenkinsfile
      # github repo url where the pipeline script is located. mandatory
      githubRepoUrl: https://github.com/aFlyBird0/jenkins-file-test.git
❯ ./dtm apply -f jenkins-all.yaml -y
2022-07-23 22:33:59 ℹ [INFO]  Apply started.
2022-07-23 22:33:59 ℹ [INFO]  Got Backend from config: local
2022-07-23 22:33:59 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-07-23 22:33:59 ℹ [INFO]  Using local backend. State file: devstream-jenkins-all.state.
2022-07-23 22:33:59 ℹ [INFO]  Tool (jenkins/default) found in config but doesn't exist in the state, will be created.
2022-07-23 22:33:59 ℹ [INFO]  Tool (jenkins-pipeline-kubernetes/default) found in config but doesn't exist in the state, will be created.
2022-07-23 22:33:59 ℹ [INFO]  Start executing the plan.
2022-07-23 22:33:59 ℹ [INFO]  Changes count: 2.
2022-07-23 22:33:59 ℹ [INFO]  -------------------- [  Processing progress: 1/2.  ] --------------------
2022-07-23 22:33:59 ℹ [INFO]  Processing: (jenkins/default) -> Create ...
2022-07-23 22:34:00 ℹ [INFO]  Test environment is enabled. Please ensure you have created the directories correctly under the guide of plugin doc.
2022-07-23 22:34:11 ℹ [INFO]  Creating or updating helm chart ...
2022/07/23 22:34:17 creating 11 resource(s)
2022/07/23 22:34:17 beginning wait for 11 resources with timeout of 5m0s
2022/07/23 22:34:17 StatefulSet is not ready: jenkins/dev-jenkins. 0 out of 1 expected pods are ready
2022/07/23 22:34:19 StatefulSet is not ready: jenkins/dev-jenkins. 0 out of 1 expected pods are ready
2022/07/23 22:34:21 StatefulSet is not ready: jenkins/dev-jenkins. 0 out of 1 expected pods are ready
2022/07/23 22:34:23 StatefulSet is not ready: jenkins/dev-jenkins. 0 out of 1 expected pods are ready
...
...
...
2022/07/23 22:36:23 StatefulSet is not ready: jenkins/dev-jenkins. 0 out of 1 expected pods are ready
2022/07/23 22:36:25 StatefulSet is not ready: jenkins/dev-jenkins. 0 out of 1 expected pods are ready
2022/07/23 22:36:27 StatefulSet is not ready: jenkins/dev-jenkins. 0 out of 1 expected pods are ready
2022/07/23 22:36:29 release installed successfully: dev/jenkins-4.1.13
2022-07-23 22:36:29 ℹ [INFO]  Here is how to get the password of the admin user:
2022-07-23 22:36:29 ℹ [INFO]  kubectl exec --namespace jenkins -it svc/dev-jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo
2022-07-23 22:36:29 ℹ [INFO]  You are in test env. Here are the Jenkins url in host machine and Jenkins url in K8s cluster.
2022-07-23 22:36:29 ℹ [INFO]  Jenkins url in host machine: http://127.0.0.1:32000/login
2022-07-23 22:36:29 ℹ [INFO]  , Jenkins url in K8s:http://172.18.0.2:32000/login
2022-07-23 22:36:29 ✔ [SUCCESS]  Tool (jenkins/default) Create done.
2022-07-23 22:36:29 ℹ [INFO]  -------------------- [  Processing progress: 2/2.  ] --------------------
2022-07-23 22:36:29 ℹ [INFO]  Processing: (jenkins-pipeline-kubernetes/default) -> Create ...
2022-07-23 22:36:31 ℹ [INFO]  credential credential-jenkins-pipeline-kubernetes-by-devstream not found, creating...
2022-07-23 22:36:31 ℹ [INFO]  job jenkins-plugin-test not found, creating...
2022-07-23 22:36:32 ✔ [SUCCESS]  Tool (jenkins-pipeline-kubernetes/default) Create done.
2022-07-23 22:36:32 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-07-23 22:36:32 ✔ [SUCCESS]  All plugins applied successfully.
2022-07-23 22:36:32 ✔ [SUCCESS]  Apply finished.

Screenshots