feat: enhance jenkins and jenkins-k8s-pipeline plugin

docs/plugins/jenkins-pipeline-kubernetes.zh.md

@@ -4,11 +4,10 @@
 
 步骤：
 
-1. 访问 Jenkins web UI，创建 token。步骤：People -> admin ->Configure -> API Token -> Add new Token。
-2. 按需修改配置项，其中 `githubRepoUrl` 为 GitHub 仓库地址，应预先建立一个 GitHub 仓库，并创建一个名为 "Jenkinsfile" 的文件放至仓库根目录。
-3. 设置环境变量
+1. 按需修改配置项，其中 `githubRepoUrl` 为 GitHub 仓库地址，应预先建立一个 GitHub 仓库，并创建一个名为 "Jenkinsfile" 的文件放至仓库根目录。
+2. 设置环境变量
     - `GITHUB_TOKEN`
-    - `JENKINS_TOKEN`
+    - `JENKINS_PASSWORD`
 
 ## 用例
 
@@ -18,4 +17,103 @@
 
 ```
 
-目前，所有选项均为必填项。
+## 和 `jenkins` 插件一起使用
+
+这个插件可以和 `jenkins` 插件一起使用，[`jenkins` 插件文档](./jenkins.zh.md)。
+
+即在安装完 `Jenkins` 后，再建立 `Jenkins` job。
+
+首先根据 `dependsOn` 设定插件依赖，再根据 `${{jenkins.default.outputs.jenkinsURL}}` 和 `${{jenkins.default.outputs.jenkinsPasswordOfAdmin}}` 设置 Jenkins 的 URL 和 admin 密码。
+
+注意：如果你的 Kubernetes 集群是 K8s in docker 模式，请自行设置网络，确保 `jenkins` 插件中 `NodePort` 设置的端口在宿主机内能访问。
+
+```yaml
+---
+tools:
+  # name of the tool
+  - name: jenkins
+    # id of the tool instance
+    instanceID: default
+    # format: name.instanceID; If specified, dtm will make sure the dependency is applied first before handling this tool.
+    dependsOn: [ ]
+    # options for the plugin
+    options:
+      # if true, the plugin will use hostpath to create a pv named `jenkins-pv`
+      # and you should create the volumes directory manually, see plugin doc for details.
+      test_env: false
+      # need to create the namespace or not, default: false
+      create_namespace: false
+      # Helm repo information
+      repo:
+        # name of the Helm repo
+        name: jenkins
+        # url of the Helm repo
+        url: https://charts.jenkins.io
+      # Helm chart information
+      chart:
+        # name of the chart
+        chart_name: jenkins/jenkins
+        # release name of the chart
+        release_name: dev
+        # k8s namespace where jenkins will be installed
+        namespace: jenkins
+        # whether to wait for the release to be deployed or not
+        wait: true
+        # the time to wait for any individual Kubernetes operation (like Jobs for hooks). This defaults to 5m0s
+        timeout: 5m
+        # whether to perform a CRD upgrade during installation
+        upgradeCRDs: true
+        # custom configuration. You can refer to [Jenkins values.yaml](https://github.com/jenkinsci/helm-charts/blob/main/charts/jenkins/values.yaml)
+        values_yaml: |
+          persistence:
+            # for prod env: the existent storageClass, please change it
+            # for test env: just ignore it, but don't remove it
+            storageClass: jenkins-pv
+          serviceAccount:
+            create: false
+            name: jenkins
+          controller:
+            serviceType: NodePort
+            nodePort: 32000
+            additionalPlugins:
+              # install "GitHub Pull Request Builder" plugin, see https://plugins.jenkins.io/ghprb/ for more details
+              - ghprb
+              # install "OWASP Markup Formatter" plugin, see https://plugins.jenkins.io/antisamy-markup-formatter/ for more details
+              - antisamy-markup-formatter
+            # Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
+            enableRawHtmlMarkupFormatter: true
+            # Jenkins Configuraction as Code, refer to https://plugins.jenkins.io/configuration-as-code/ for more details
+            # notice: All configuration files that are discovered MUST be supplementary. They cannot overwrite each other's configuration values. This creates a conflict and raises a ConfiguratorException.
+            JCasC:
+              defaultConfig: true
+              # each key-value in configScripts will be added to the ${JENKINS_HOME}/casc_configs/ directory as a file.
+              configScripts:
+                # this will create a file named "safe_html.yaml" in the ${JENKINS_HOME}/casc_configs/ directory.
+                # it is used to configure the "Safe HTML" plugin.
+                # filename must meet RFC 1123, see https://tools.ietf.org/html/rfc1123 for more details
+  - name: jenkins-pipeline-kubernetes
+    # id of the tool instance
+    instanceID: default
+    # format: name.instanceID; If specified, dtm will make sure the dependency is applied first before handling this tool.
+    dependsOn: [ "jenkins.default" ]
+    # options for the plugin
+    options:
+      jenkins:
+        # jenkinsUrl, format: hostname:port
+        url: ${{jenkins.default.outputs.jenkinsURL}}
+        # jenkins user name, default: admin
+        user: admin
+        # jenkins password, you have 3 options to set the password:
+        # 1. use outputs of the `jenkins` plugin, see docs for more details
+        # 2. set the `JENKINS_PASSWORD` environment variable
+        # 3. fill in the password in this field(not recommended)
+        # if all set, devstream will read the password from the config file or outputs from jenkins plugin first, then env var.
+        password: ${{jenkins.default.outputs.jenkinsPasswordOfAdmin}}
+        # jenkins job name, mandatory
+        jobName:
+        # path to the pipeline file, relative to the git repo root directory. default: Jenkinsfile
+        pipelineScriptPath: Jenkinsfile
+      # github repo url where the pipeline script is located. mandatory
+      githubRepoUrl: https://github.com/xxx/xxx.git
+```
+

docs/plugins/jenkins.md

@@ -47,3 +47,10 @@ chown -R 1000:1000 <your-dtm-home-dir>/data/jenkins-volume/
 ```
 
 Currently, all the parameters in the example above are mandatory.
+
+## Outputs
+
+This plugin has two outputs:
+
+- `jenkinsURL` (format: `hostname:port`, example: "localhost:8080")
+- `jenkinsPasswordOfAdmin` 

docs/plugins/jenkins.zh.md

@@ -47,3 +47,10 @@ chown -R 1000:1000 <your-dtm-home-dir>/data/jenkins-volume/
 ```
 
 当前，所有配置项均为必填。
+
+## 输出
+
+这个插件有两个输出：
+
+- `jenkinsURL` (格式: `hostname:port`, 例如: "localhost:8080")
+- `jenkinsPasswordOfAdmin` 

go.mod

@@ -10,6 +10,7 @@ require (
 	github.com/aws/aws-sdk-go-v2 v1.16.3
 	github.com/aws/aws-sdk-go-v2/config v1.15.5
 	github.com/aws/aws-sdk-go-v2/service/s3 v1.26.9
+	github.com/bndr/gojenkins v1.1.0
 	github.com/cenkalti/backoff v2.2.1+incompatible
 	github.com/cheggaaa/pb v1.0.29
 	github.com/deckarep/golang-set/v2 v2.1.0
@@ -27,7 +28,7 @@ require (
 	github.com/spf13/viper v1.8.1
 	github.com/stretchr/testify v1.7.0
 	github.com/tcnksm/go-input v0.0.0-20180404061846-548a7d7a8ee8
-	github.com/withfig/autocomplete-tools/integrations/cobra v0.0.0-20220705165518-2761d7f4b8bc
+	github.com/withfig/autocomplete-tools/integrations/cobra v0.0.0-20220721102007-67b2515c5ea4
 	github.com/xanzy/go-gitlab v0.55.1
 	golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3
 	golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8

go.sum

@@ -213,6 +213,8 @@ github.com/blang/semver v3.1.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnweb
 github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=
 github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869/go.mod h1:Ekp36dRnpXw/yCqJaO+ZrUyxD+3VXMFFr56k5XYrpB4=
+github.com/bndr/gojenkins v1.1.0 h1:TWyJI6ST1qDAfH33DQb3G4mD8KkrBfyfSUoZBHQAvPI=
+github.com/bndr/gojenkins v1.1.0/go.mod h1:QeskxN9F/Csz0XV/01IC8y37CapKKWvOHa0UHLLX1fM=
 github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
 github.com/bombsimon/logrusr v1.0.0 h1:CTCkURYAt5nhCCnKH9eLShYayj2/8Kn/4Qg3QfiU+Ro=
 github.com/bombsimon/logrusr v1.0.0/go.mod h1:Jq0nHtvxabKE5EMwAAdgTaz7dfWE8C4i11NOltxGQpc=
@@ -1268,8 +1270,8 @@ github.com/vmihailenco/tagparser/v2 v2.0.0/go.mod h1:Wri+At7QHww0WTrCBeu4J6bNtoV
 github.com/vmware/govmomi v0.20.3/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU=
 github.com/willf/bitset v1.1.11-0.20200630133818-d5bec3311243/go.mod h1:RjeCKbqT1RxIR/KWY6phxZiaY1IyutSBfGjNPySAYV4=
 github.com/willf/bitset v1.1.11/go.mod h1:83CECat5yLh5zVOf4P1ErAgKA5UDvKtgyUABdr3+MjI=
-github.com/withfig/autocomplete-tools/integrations/cobra v0.0.0-20220705165518-2761d7f4b8bc h1:2pGkMttK5jQ8+6YhdyeQIHyVa84HMdJhILozImSWX6c=
-github.com/withfig/autocomplete-tools/integrations/cobra v0.0.0-20220705165518-2761d7f4b8bc/go.mod h1:nmuySobZb4kFgFy6BptpXp/BBw+xFSyvVPP6auoJB4k=
+github.com/withfig/autocomplete-tools/integrations/cobra v0.0.0-20220721102007-67b2515c5ea4 h1:GoQuMyofxqcdyzWqVhsv5IsCL+wHoocrhUgzhtB2Nj4=
+github.com/withfig/autocomplete-tools/integrations/cobra v0.0.0-20220721102007-67b2515c5ea4/go.mod h1:nmuySobZb4kFgFy6BptpXp/BBw+xFSyvVPP6auoJB4k=
 github.com/xanzy/go-gitlab v0.55.1 h1:IgX/DS9buV0AUz8fuJPQkdl0fQGfBiAsAHxpun8sNhg=
 github.com/xanzy/go-gitlab v0.55.1/go.mod h1:F0QEXwmqiBUxCgJm8fE9S+1veX4XC9Z4cfaAbqwk4YM=
 github.com/xanzy/ssh-agent v0.2.1 h1:TCbipTQL2JiiCprBWx9frJ2eJlCYT00NmctrHxVAr70=

internal/pkg/plugin/jenkins/create.go

@@ -26,7 +26,7 @@ func Create(options map[string]interface{}) (map[string]interface{}, error) {
 		TermateOperations: []plugininstaller.BaseOperation{
 			helm.DealWithNsWhenInterruption,
 		},
-		GetStatusOperation: helm.GetPluginStaticStateByReleaseNameWrapper(defaultStatefulsetTplList),
+		GetStatusOperation: wrapperHelmResourceAndCustomResource(helm.GetPluginStaticStateByReleaseNameWrapper(defaultStatefulsetTplList)),
 	}
 
 	// 2. execute installer get status and error

internal/pkg/plugin/jenkins/jenkins.go

@@ -1,6 +1,57 @@
 package jenkins
 
+import "github.com/devstream-io/devstream/internal/pkg/plugininstaller"
+
 var defaultStatefulsetTplList = []string{
 	// ${release-name}-jenkins
 	"%s-jenkins",
 }
+
+// wrapperHelmResourceAndCustomResource wraps helm resource and custom resource,
+// this is due to the limitation of `plugininstaller`,
+// now `plugininstaller.GetStatusOperation` only support one resource get function,
+// if we want to use both existing resource get function(such as helm's methods) and custom function,
+// we have to wrap them into one function.
+func wrapperHelmResourceAndCustomResource(helmResFunc plugininstaller.StatusOperation) plugininstaller.StatusOperation {
+	return func(options plugininstaller.RawOptions) (map[string]interface{}, error) {
+		opts, err := newOptions(options)
+		if err != nil {
+			return nil, err
+		}
+
+		// 1. get helm resource
+		resource, err := helmResFunc(options)
+		if err != nil {
+			return nil, err
+		}
+
+		// 2. get custom resource, and merge with helm resource
+		outputs := map[string]interface{}{}
+		// 2.1 get jenkins url
+		// TODO(aFlyBird0): TestEnv is not strictly as same as "K8s in docker"
+		if !opts.TestEnv {
+			jenkinsURL, err := getJenkinsURL(options)
+			if err != nil {
+				return nil, err
+			}
+			outputs["jenkinsURL"] = jenkinsURL
+		} else {
+			jenkinsURLForTestEnv, err := getJenkinsURLForTestEnv(options)
+			if err != nil {
+				return nil, err
+			}
+			outputs["jenkinsURL"] = jenkinsURLForTestEnv
+		}
+
+		// 2.2 get jenkins password of admin
+		jenkinsPassword, err := getPasswdOfAdmin(options)
+		if err != nil {
+			return nil, err
+		}
+		outputs["jenkinsPasswordOfAdmin"] = jenkinsPassword
+
+		resource["outputs"] = outputs
+
+		return resource, nil
+	}
+}

internal/pkg/plugin/jenkins/prompt.go

@@ -9,38 +9,130 @@ import (
 	"github.com/devstream-io/devstream/pkg/util/log"
 )
 
+func buildPasswdOfAdminCommand(opts jenkinsOptions) string {
+	method := fmt.Sprintf("kubectl exec --namespace jenkins -it svc/%s-jenkins -c jenkins "+
+		"-- /bin/cat /run/secrets/additional/chart-admin-password && echo", opts.Chart.ReleaseName)
+
+	return method
+}
+
 func howToGetPasswdOfAdmin(options plugininstaller.RawOptions) error {
 	opts, err := newOptions(options)
 	if err != nil {
 		return err
 	}
+
 	log.Info("Here is how to get the password of the admin user:")
-	method := fmt.Sprintf("kubectl exec --namespace jenkins -it svc/%s-jenkins -c jenkins "+
-		"-- /bin/cat /run/secrets/additional/chart-admin-password && echo", opts.Chart.ReleaseName)
-	log.Info(method)
+	command := buildPasswdOfAdminCommand(opts)
+	log.Info(command)
+
 	return nil
 }
 
-func showJenkinsUrl(options plugininstaller.RawOptions) error {
+func getPasswdOfAdmin(options plugininstaller.RawOptions) (string, error) {
 	opts, err := newOptions(options)
 	if err != nil {
-		return err
+		return "", err
+	}
+
+	commandString := buildPasswdOfAdminCommand(opts)
+	command := exec.Command("sh", "-c", commandString)
+
+	password, err := command.Output()
+	if err != nil {
+		return "", fmt.Errorf("failed to get password of admin user: %v", err)
 	}
+
+	return strings.TrimSpace(string(password)), nil
+
+}
+
+// getJenkinsURL returns the jenkins url of the jenkins, format: hostname:port
+func getJenkinsURL(options plugininstaller.RawOptions) (string, error) {
+	opts, err := newOptions(options)
+	if err != nil {
+		return "", err
+	}
+
 	commands := []string{
 		`jsonpath="{.spec.ports[0].nodePort}"`,
 		fmt.Sprintf(`NODE_PORT=$(kubectl get -n jenkins -o jsonpath=$jsonpath services %s-jenkins)`, opts.Chart.ReleaseName),
 		`jsonpath="{.items[0].status.addresses[0].address}"`,
 		`NODE_IP=$(kubectl get nodes -n jenkins -o jsonpath=$jsonpath)`,
-		`echo http://$NODE_IP:$NODE_PORT/login`,
+		`echo $NODE_IP:$NODE_PORT`,
 	}
 
 	cmd := exec.Command("sh", "-c", strings.Join(commands, " && "))
 
 	output, err := cmd.Output()
 	if err != nil {
-		log.Errorf("Failed to get jenkins url: %v", err)
+		return "", fmt.Errorf("Failed to get jenkins url: %v", err)
+	}
+
+	return strings.TrimSpace(string(output)), nil
+}
+
+// getJenkinsURL behaves like getJenkinsURL, but the hostname will be always 127.0.0.1
+func getJenkinsURLForTestEnv(options plugininstaller.RawOptions) (string, error) {
+	opts, err := newOptions(options)
+	if err != nil {
+		return "", err
+	}
+
+	commands := []string{
+		`jsonpath="{.spec.ports[0].nodePort}"`,
+		fmt.Sprintf(`NODE_PORT=$(kubectl get -n jenkins -o jsonpath=$jsonpath services %s-jenkins)`, opts.Chart.ReleaseName),
+		`jsonpath="{.items[0].status.addresses[0].address}"`,
+		`NODE_IP=127.0.0.1`,
+		`echo $NODE_IP:$NODE_PORT`,
+	}
+
+	cmd := exec.Command("sh", "-c", strings.Join(commands, " && "))
+
+	output, err := cmd.Output()
+	if err != nil {
+		return "", fmt.Errorf("Failed to get jenkins url: %v", err)
+	}
+
+	return strings.TrimSpace(string(output)), nil
+}
+
+func showJenkinsUrl(options plugininstaller.RawOptions) error {
+	opts, err := newOptions(options)
+	if err != nil {
+		return err
+	}
+
+	// prod env: just print Jenkins url
+	if !opts.TestEnv {
+		url, err := getJenkinsURL(options)
+		if err != nil {
+			log.Error(err)
+			return err
+		}
+
+		log.Infof("Jenkins url: http://%s/login", url)
+	}
+
+	// test env: print Jenkins url in host machine and Jenkins url in K8s cluster
+	if opts.TestEnv {
+		log.Info("You are in test env. Here are the Jenkins url in host machine and Jenkins url in K8s cluster.")
+
+		urlForTestEnv, err := getJenkinsURLForTestEnv(options)
+		if err != nil {
+			log.Error(err)
+			return err
+		}
+		log.Infof("Jenkins url in host machine: http://%s/login", urlForTestEnv)
+
+		urlInK8s, err := getJenkinsURL(options)
+		if err != nil {
+			log.Error(err)
+			return err
+		}
+		log.Info("Jenkins url in K8s:", fmt.Sprintf("http://%s/login", urlInK8s))
+
 	}
 
-	log.Info("Jenkins url:", string(output))
 	return nil
 }

internal/pkg/plugin/jenkins/read.go

@@ -13,7 +13,7 @@ func Read(options map[string]interface{}) (map[string]interface{}, error) {
 			helm.Validate,
 			replaceStroageClass,
 		},
-		GetStatusOperation: helm.GetPluginAllState,
+		GetStatusOperation: wrapperHelmResourceAndCustomResource(helm.GetPluginAllState),
 	}
 
 	status, err := runner.Execute(plugininstaller.RawOptions(options))