Skip to content

Configure GKE Cluster(s) To Run Jobs from Jenkins

Jump to bottom
Ramkumar edited this page Dec 3, 2022 · 1 revision

Configure Jenkins master to get to access the GKE cluster (without "gcloud" installed on it)

https://ahmet.im/blog/authenticating-to-gke-without-gcloud/

  1. Set the Project ID
gcloud config set project handy-freedom-363504
  1. Create a ServiceAccount
gcloud iam service-accounts create gke-access --description="access gke" --display-name="gke-access"

Note: The given Service-account name must be unique in the project.

======

To get Projects List and Service Accounts details:

gcloud projects list
gcloud iam service-accounts list

====

  1. Grant the required K8s role (here, we grnat "container.admin" role)
gcloud projects add-iam-policy-binding handy-freedom-363504     --member="serviceAccount:gke-access@nimble-album-352419.iam.gserviceaccount.com"     --role="roles/container.admin"
  1. create a key file (private key to SA)
gcloud iam service-accounts keys create sa-private-key.json     --iam-account=790569128722-compute@developer.gserviceaccount.com
  1. Create "kubeconfig.yaml" as below:

Replace the "CLUSTER NAME" & "ZONE NAME" with the values of the desired cluster.

GET_CMD="gcloud container clusters describe <CLUSTER NAME> --zone=<ZONE NAME>"

e.g:

GET_CMD="gcloud container clusters describe cost-optimized-cluster-1 --zone=us-central1-c"

cat > kubeconfig.yaml <<EOF
apiVersion: v1
kind: Config
current-context: my-cluster
contexts: [{name: my-cluster, context: {cluster: cluster-1, user: user-1}}]
users: [{name: user-1, user: {auth-provider: {name: gcp}}}]
clusters:
- name: cluster-1
  cluster:
    server: "https://$(eval "$GET_CMD --format='value(endpoint)'")"
    certificate-authority-data: "$(eval "$GET_CMD --format='value(masterAuth.clusterCaCertificate)'")"
EOF

This kubeconfig.yaml file does not contain secrets such as your credentials. It only points kubectl to your cluster. You can actually safely check store this file in your git repository.

Note that you can actually rotate both this master IP address and CA certificate by triggering a manual rotation. If you do that, you need to re-generate this file. (This is the only downside to this approach.)

Execute the below on Jenkins master or the client where you would need to access Jenkins:

  1. Download kubeconfig.yaml & sa-private-key.json on to Jenkins Build server.

  2. Set the below shell variables & start using the cluster.

export GOOGLE_APPLICATION_CREDENTIALS=service-account-key.json
export KUBECONFIG=kubeconfig.yaml

kubectl get nodes #← You are authenticated if this works!

Add the below lines to "/etc/containerd/config.toml" to connect to insecure private Nexus docker registry.

without this, you would end up getting the below errors:

  • Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a757f0401348019dff86e0fa45723620742eef98bc01ced00eb825405027321a": failed to find plugin "loopback" in path [/opt/cni/bin]

  • Failed to pull image "10.182.0.16:8082/appointme-admin-api:32-DEV": rpc error: code = Unknown desc = failed to pull and unpack image "10.182.0.16:8082/appointme-admin-api:32-DEV": failed to resolve reference "10.182.0.16:8082/appointme-admin-api:32-DEV": failed to do request: Head "https://10.182.0.16:8082/v2/appointme-admin-api/manifests/32-DEV": http: server gave HTTP response to HTTPS client

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.182.0.16:8082"]
  endpoint = ["http://10.182.0.16:8082"]

  1. Restart the Containerd service., if you make any changes to the above config file.

  2. Create kubernetes secret object & pass it on as "imagePullSecrets" in your deployment/pod manifest files.

    kubectl create secret generic regcred     --from-file=.dockerconfigjson=/root/.docker/config.json     --type=kubernetes.io/dockerconfigjson
Clone this wiki locally