Add multi-cluster utilities & Readme

multi-cluster-utilities/README.md

@@ -0,0 +1,54 @@
+# Multicluster Configuration
+The create-kubeconfig-secret.sh is copied from [multicluster-runtime project](https://github.com/kubernetes-sigs/multicluster-runtime/tree/main), especially, from the `Kubeconfig Provider Example`.
+
+This Readme cover only what is relevant for setting up a `Management cluster` which hosts the kubernetes operator, and `Resource Clusters` which host the Netbox Operator Resources. For more information over the scripts, and how a multicluster setup could be setup with Kubeconfig provider please read [this](https://github.com/kubernetes-sigs/multicluster-runtime/blob/main/examples/kubeconfig/README.md)
+
+## 1. Create Management Cluster
+
+Follow the guide in project's root folder.
+If all the prerequisites are in place, then `make create-kind` creates the cluster which is going to be used for the all the controller's dependencies (netbox backend, databases etc).
+This cluster is also configured with the netbox-operator CRDs but the CRs are hosted and reconciled only in Resource Clusters.
+
+## 2. Create Resource Clusters
+
+Create your Resource Clusters & Provision Netbox-Operator Custom Resource Definitions. For each resource cluster you want to create, execute:
+
+- `kind create cluster --name <res-cluster-name>`
+- `make install`
+
+Sidenote: kind create command changes the default context in which the kubectl commands point to. When you are done from this step, the kubectl context will be set at the last cluster you created.
+
+## 3. Establish cross-cluster access
+
+Set kubectl context back to `management cluster`
+`kubectl config use-context kind-kind`
+
+Execute RBAC scripts for kubeconfig provider, towards each cluster you created in the previous step.
+- `./create-kubeconfig-secret.sh -c kind-<res-cluster-name>`
+
+For each cluster that gets configured as a 'Resource' cluster, a secret is populated in the 'Management' cluster.
+Make sure that the appropriate secrets are populated in the kind-kind cluster, with names `kind-<res-cluster-name>`.
+
+## 4-1. Execute manager process locally
+At this point, you should be able to successfully start netbox operator process locally after:
+- Establishing a port forward from Management cluster to your host for the Netbox service: `kubectl port-forward deploy/netbox 8080:8080 -n default`
+- Setting environment variable `export NETBOX_HOST=localhost:8080`
+
+## 4-2. Execute manager process on management cluster
+Deploying the manager in the management cluster involves additional manual steps.
+
+From the project's parent directory, execute `make deploy-kind`
+
+### Patch cluster role of netbox operator manager
+ClusterRole `manager-role` needs to allow reading, listing and watching secrets.
+    - Execute `patch-netbox-clusterrole.sh`
+
+### Update secret in controller cluster
+The secrets generated from the `create-kubeconfig-secret` needs to refer to the correct ip:port for each resource cluster. Currently it's pointing a localhost ip, which is only reachable from the host machine.
+- Execute script `create-kubeconfig-secret-cluster.sh -c kind-<res-cluster-name> --skip-create-rbac`
+    - This script updates the secret on management cluster, to use the IP of control-plane node of the resource cluster, retrieved from ``docker inspect <res-cluster-name>-control-plane | jq '.[0].NetworkSettings.Networks.kind.IPAddress'``
+    - The port of the K8s API server is assumed to be `6443`. You can check it with `docker inspect <res-cluster-name>-control-plane | jq '.[0].NetworkSettings.Ports'`
+
+## 5. Test Reconciliation
+Apply an example CR in resource cluster and check if it's getting reconciled.
+`kubectl --context kind-<res-cluster-name> apply -f config/samples/netbox_v1_ipaddress.yaml`

multi-cluster-utilities/create-kubeconfig-secret-cluster.sh

@@ -0,0 +1,324 @@
+# File copied from https://github.com/kubernetes-sigs/multicluster-runtime/blob/main/examples/kubeconfig/README.md
+// Modified by Swisscom (Schweiz) AG.
+#!/bin/bash
+
+# Script to create a kubeconfig secret for the pod lister controller
+
+set -e
+
+# Default values
+NAMESPACE="default"
+SERVICE_ACCOUNT="multicluster-kubeconfig-provider"
+KUBECONFIG_CONTEXT=""
+SECRET_NAME=""
+ROLE_TYPE="clusterrole"
+RULES_FILE=""
+CREATE_RBAC="true"
+
+# Check for yq
+if ! command -v yq &>/dev/null; then
+  echo "ERROR: 'yq' is required but not installed. Please install yq (https://mikefarah.gitbook.io/yq/) and try again."
+  exit 1
+fi
+
+# Function to display usage information
+function show_help {
+  echo "Usage: $0 [options]"
+  echo "  -c, --context CONTEXT    Kubeconfig context to use (required)"
+  echo "  --name NAME              Name for the secret (defaults to context name)"
+  echo "  -n, --namespace NS       Namespace to create the secret in (default: ${NAMESPACE})"
+  echo "  -a, --service-account SA Service account name to use (default: ${SERVICE_ACCOUNT})"
+  echo "  -t, --role-type TYPE     Create Role or ClusterRole (role|clusterrole) (default: clusterrole)"
+  echo "  -r, --rules-file FILE    Path to rules file (default: rules.yaml in script directory)"
+  echo "  --skip-create-rbac       Skip creating RBAC resources (Role/ClusterRole and bindings)"
+  echo "  -h, --help               Show this help message"
+  echo ""
+  echo "Examples:"
+  echo "  $0 -c prod-cluster"
+  echo "  $0 -c prod-cluster -t role -r ./custom-rules.yaml"
+  echo "  $0 -c prod-cluster -t clusterrole"
+  echo "  $0 -c prod-cluster --skip-create-rbac"
+}
+
+# Function to create Role or ClusterRole
+function create_rbac {
+  local role_type="$1"
+  local rules_file="$2"
+  local role_name="$3"
+  local namespace="$4"
+
+  if [ ! -f "$rules_file" ]; then
+    echo "ERROR: Rules file not found: $rules_file"
+    exit 1
+  fi
+
+  echo "Creating ${role_type} '${role_name}'..."
+
+  if [ "$role_type" = "role" ]; then
+    # Create Role
+    ROLE_YAML=$(cat <<EOF
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ${role_name}
+  namespace: ${namespace}
+rules:
+$(yq '.rules' "$rules_file")
+EOF
+)
+
+    echo "$ROLE_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
+
+    # Create RoleBinding
+    ROLEBINDING_YAML=$(cat <<EOF
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ${role_name}-binding
+  namespace: ${namespace}
+subjects:
+- kind: ServiceAccount
+  name: ${SERVICE_ACCOUNT}
+  namespace: ${namespace}
+roleRef:
+  kind: Role
+  name: ${role_name}
+  apiGroup: rbac.authorization.k8s.io
+EOF
+)
+
+    echo "$ROLEBINDING_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
+
+  else
+    # Create ClusterRole
+    CLUSTERROLE_YAML=$(cat <<EOF
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: ${role_name}
+rules:
+$(yq '.rules' "$rules_file")
+EOF
+)
+
+    echo "$CLUSTERROLE_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
+
+    # Create ClusterRoleBinding
+    CLUSTERROLEBINDING_YAML=$(cat <<EOF
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: ${role_name}-binding
+subjects:
+- kind: ServiceAccount
+  name: ${SERVICE_ACCOUNT}
+  namespace: ${namespace}
+roleRef:
+  kind: ClusterRole
+  name: ${role_name}
+  apiGroup: rbac.authorization.k8s.io
+EOF
+)
+
+    echo "$CLUSTERROLEBINDING_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
+  fi
+
+  echo "$(tr '[:lower:]' '[:upper:]' <<< ${role_type:0:1})${role_type:1} '${role_name}' created successfully!"
+}
+
+# Function to create service account if it doesn't exist
+function ensure_service_account {
+  local namespace="$1"
+  local service_account="$2"
+
+  echo "Checking if service account '${service_account}' exists in namespace '${namespace}'..."
+
+  # Check if service account exists
+  if ! kubectl --context=${KUBECONFIG_CONTEXT} get serviceaccount ${service_account} -n ${namespace} &>/dev/null; then
+    echo "Service account '${service_account}' not found in namespace '${namespace}'. Creating..."
+
+    # Create the service account
+    SERVICE_ACCOUNT_YAML=$(cat <<EOF
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: ${service_account}
+  namespace: ${namespace}
+EOF
+)
+
+    echo "$SERVICE_ACCOUNT_YAML" | kubectl --context=${KUBECONFIG_CONTEXT} apply -f -
+    echo "Service account '${service_account}' created successfully in namespace '${namespace}'"
+  else
+    echo "Service account '${service_account}' already exists in namespace '${namespace}'"
+  fi
+}
+
+# Parse command line options
+while [[ $# -gt 0 ]]; do
+  key="$1"
+  case $key in
+    --name)
+      SECRET_NAME="$2"
+      shift 2
+      ;;
+    -n|--namespace)
+      NAMESPACE="$2"
+      shift 2
+      ;;
+    -c|--context)
+      KUBECONFIG_CONTEXT="$2"
+      shift 2
+      ;;
+    -a|--service-account)
+      SERVICE_ACCOUNT="$2"
+      shift 2
+      ;;
+    -t|--role-type)
+      ROLE_TYPE="$2"
+      shift 2
+      ;;
+    -r|--rules-file)
+      RULES_FILE="$2"
+      shift 2
+      ;;
+    --skip-create-rbac)
+      CREATE_RBAC="false"
+      shift
+      ;;
+    -h|--help)
+      show_help
+      exit 0
+      ;;
+    *)
+      echo "Unknown option: $1"
+      show_help
+      exit 1
+      ;;
+  esac
+done
+
+# Validate required arguments
+if [ -z "$KUBECONFIG_CONTEXT" ]; then
+  echo "ERROR: Kubeconfig context is required (-c, --context)"
+  show_help
+  exit 1
+fi
+
+# Validate role type if specified
+if [ -n "$ROLE_TYPE" ] && [ "$ROLE_TYPE" != "role" ] && [ "$ROLE_TYPE" != "clusterrole" ]; then
+  echo "ERROR: Invalid role type '$ROLE_TYPE'. Must be 'role' or 'clusterrole'"
+  show_help
+  exit 1
+fi
+
+# Set default rules file if not specified
+if [ -z "$RULES_FILE" ]; then
+  RULES_FILE="$(dirname "$0")/rules.yaml"
+fi
+
+# Set secret name to context if not specified
+if [ -z "$SECRET_NAME" ]; then
+  SECRET_NAME="$KUBECONFIG_CONTEXT"
+fi
+
+# Create RBAC resources by default (unless --no-create-role is specified)
+if [ "$CREATE_RBAC" = "true" ]; then
+  create_rbac "$ROLE_TYPE" "$RULES_FILE" "$SECRET_NAME" "$NAMESPACE"
+fi
+
+# Get the cluster CA certificate from the remote cluster
+CLUSTER_CA=$(kubectl --context=${KUBECONFIG_CONTEXT} config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}')
+if [ -z "$CLUSTER_CA" ]; then
+  echo "ERROR: Could not get cluster CA certificate"
+  exit 1
+fi
+
+# Get the cluster server URL from the remote cluster
+CLUSTER_SERVER=$(kubectl --context=${KUBECONFIG_CONTEXT} config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')
+if [ -z "$CLUSTER_SERVER" ]; then
+  echo "ERROR: Could not get cluster server URL"
+  exit 1
+fi
+
+# Ensure service account exists
+ensure_service_account "$NAMESPACE" "$SERVICE_ACCOUNT"
+
+# Get the service account token from the remote cluster
+SA_TOKEN=$(kubectl --context=${KUBECONFIG_CONTEXT} -n ${NAMESPACE} create token ${SERVICE_ACCOUNT} --duration=8760h)
+if [ -z "$SA_TOKEN" ]; then
+  echo "ERROR: Could not create service account token"
+  exit 1
+fi
+
+# Get the cluster's external IP
+CONTROL_PLANE_NAME="${SECRET_NAME#kind-}"-control-plane
+CLUSTER_EXTERNAL_IP=$(docker inspect ${CONTROL_PLANE_NAME} | jq -r '.[0].NetworkSettings.Networks.kind.IPAddress')
+if [ -z "$CLUSTER_EXTERNAL_IP" ]; then
+  echo "ERROR: Could not get cluster external IP"
+  exit 1
+fi
+
+# Create a new kubeconfig using the service account token
+NEW_KUBECONFIG=$(cat <<EOF
+apiVersion: v1
+kind: Config
+clusters:
+- name: ${SECRET_NAME}
+  cluster:
+    server: https://${CLUSTER_EXTERNAL_IP}:6443
+    certificate-authority-data: ${CLUSTER_CA}
+contexts:
+- name: ${SECRET_NAME}
+  context:
+    cluster: ${SECRET_NAME}
+    user: ${SERVICE_ACCOUNT}
+current-context: ${SECRET_NAME}
+users:
+- name: ${SERVICE_ACCOUNT}
+  user:
+    token: ${SA_TOKEN}
+EOF
+)
+
+# Save kubeconfig temporarily for testing
+TEMP_KUBECONFIG=$(mktemp)
+echo "$NEW_KUBECONFIG" > "$TEMP_KUBECONFIG"
+
+# echo "$NEW_KUBECONFIG"
+# # Verify the kubeconfig works
+# echo "Verifying kubeconfig..."
+# if  kubectl --kubeconfig="$TEMP_KUBECONFIG" version &>/dev/null; then
+#   rm "$TEMP_KUBECONFIG"
+#   echo "ERROR: Failed to verify kubeconfig - unable to connect to cluster."
+#   echo "- Ensure that the service account '${NAMESPACE}/${SERVICE_ACCOUNT}' on cluster '${KUBECONFIG_CONTEXT}' exists and is properly configured."
+#   echo "- You may specify a namespace using the -n flag."
+#   echo "- You may specify a service account using the -a flag."
+#   exit 1
+# fi
+# echo "Kubeconfig verified successfully!"
+
+# Encode the verified kubeconfig
+KUBECONFIG_B64=$(cat "$TEMP_KUBECONFIG" | base64 -w0)
+rm "$TEMP_KUBECONFIG"
+
+# Generate and apply the secret
+SECRET_YAML=$(cat <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ${SECRET_NAME}
+  namespace: ${NAMESPACE}
+  labels:
+    sigs.k8s.io/multicluster-runtime-kubeconfig: "true"
+type: Opaque
+data:
+  kubeconfig: ${KUBECONFIG_B64}
+EOF
+)
+
+echo "Creating kubeconfig secret..."
+echo "$SECRET_YAML" | kubectl apply -f -
+
+echo "Secret '${SECRET_NAME}' created in namespace '${NAMESPACE}'"
+echo "The operator should now be able to discover and connect to this cluster"