K8s native yaml, kubernetes changes, backup restore cases, e2e tests improve - #1032
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+64
to
+81
| runs-on: ${{ inputs.runs_on }} | ||
| steps: | ||
| - name: Checkout deployment tooling | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: simplyblock-io/simplyBlockDeploy | ||
| path: deploy | ||
| - name: Setup k8s | ||
| id: k8sdeployment | ||
| if: ${{ inputs.k8s_snode == true }} | ||
| run: | | ||
| set -e | ||
| cd deploy/bare-metal | ||
| eval $(python3 inventory.py inventory/${{ inputs.cluster }}.yml) | ||
| ./bootstrap-k3s.sh --k8s-snode | ||
| deploy: |
Comment on lines
+82
to
+216
| needs: [k8s] | ||
| runs-on: ${{ inputs.runs_on }} | ||
| outputs: | ||
| cluster_id: ${{ steps.deployment.outputs.cluster_id }} | ||
| cluster_ip: ${{ steps.deployment.outputs.cluster_ip }} | ||
| cluster_secret: ${{ steps.deployment.outputs.cluster_secret }} | ||
| mnodes: ${{ steps.deployment.outputs.mnodes }} | ||
| storage_private_ips: ${{ steps.deployment.outputs.storage_private_ips }} | ||
| steps: | ||
| - name: Checkout sbcli | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| path: sbcli | ||
| - name: Checkout deployment tooling | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: simplyblock-io/simplyBlockDeploy | ||
| path: deploy | ||
| - name: Checkout spdk-csi | ||
| if: ${{ inputs.k8s_snode == true }} | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: simplyblock-io/simplyblock-csi | ||
| path: simplyblock-csi | ||
| - name: Install Helm | ||
| if: ${{ inputs.k8s_snode == true }} | ||
| run: | | ||
| sudo yum install -y tar | ||
| curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | ||
| chmod 700 get_helm.sh | ||
| ./get_helm.sh | ||
| - name: Install kubectl | ||
| if: ${{ inputs.k8s_snode == true }} | ||
| run: | | ||
| curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" | ||
| sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl | ||
| - name: setup Credentials | ||
| if: ${{ inputs.k8s_snode == true }} | ||
| run: | | ||
| cd deploy/bare-metal | ||
| eval $(python3 inventory.py inventory/${{ inputs.cluster }}.yml) | ||
| mkdir -p $HOME/.kube | ||
| KEY="$HOME/.ssh/simplyblock-us-east-2.pem" | ||
| scp -i "$KEY" -o StrictHostKeyChecking=no root@${MNODES}:/etc/rancher/k3s/k3s.yaml $HOME/.kube/config | ||
| sed -i "s/127.0.0.1/${MNODES}/g" $HOME/.kube/config | ||
| kubectl create secret docker-registry regcred \ | ||
| --docker-server=https://index.docker.io/v1/ \ | ||
| --docker-username=hamdysimplyblock \ | ||
| --docker-password=${SECRET} \ | ||
| --docker-email=a@b.com | ||
| kubectl patch serviceaccount default --patch '{"imagePullSecrets": [{"name": "regcred"}]}' | ||
| # Create namespace simplyblock | ||
| kubectl create namespace simplyblock | ||
| # Create the secret in namespace simplyblock | ||
| kubectl create secret docker-registry regcred \ | ||
| --docker-server=https://index.docker.io/v1/ \ | ||
| --docker-username=hamdysimplyblock \ | ||
| --docker-password=${SECRET} \ | ||
| --docker-email=a@b.com \ | ||
| -n simplyblock | ||
| # Patch the default service account in namespace simplyblock | ||
| kubectl patch serviceaccount default \ | ||
| -n simplyblock \ | ||
| --patch '{"imagePullSecrets": [{"name": "regcred"}]}' | ||
| env: | ||
| SECRET: ${{ secrets.docker_pass }} | ||
| - name: Deploy cluster | ||
| id: deployment | ||
| run: | | ||
| cd deploy/bare-metal | ||
| eval $(python3 inventory.py inventory/${{ inputs.cluster }}.yml) | ||
| KEY="$HOME/.ssh/simplyblock-us-east-2.pem" | ||
| # install helm on mgmt node | ||
| ssh -i $KEY -o StrictHostKeyChecking=no root@$MNODES " | ||
| sudo yum install -y tar | ||
| curl -fsSL -o /root/get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | ||
| chmod 700 /root/get_helm.sh | ||
| /root/get_helm.sh | ||
| " | ||
| # Reset device partitions | ||
| for node in $STORAGE_PRIVATE_IPS; do | ||
| echo "restart node $node" | ||
| ssh -i $KEY -o StrictHostKeyChecking=no root@$node " | ||
| dd if=/dev/zero of=/dev/nvme0n1 bs=4096 count=1 | ||
| dd if=/dev/zero of=/dev/nvme1n1 bs=4096 count=1 | ||
| dd if=/dev/zero of=/dev/nvme2n1 bs=4096 count=1 | ||
| dd if=/dev/zero of=/dev/nvme3n1 bs=4096 count=1 | ||
| " | ||
| done | ||
| # If the cluster is running on k8s, we need to set the environment variable | ||
| DEPLOY_CMD="./bootstrap-cluster.sh \ | ||
| --max-lvol 10 --max-snap 10 | ||
| --max-size 400G --number-of-devices 1 | ||
| --sbcli-cmd sbcli-dev --spdk-debug" | ||
| if [ "${{ inputs.k8s_snode }}" == "true" ]; then | ||
| DEPLOY_CMD="$DEPLOY_CMD --mode kubernetes --k8s-snode" | ||
| echo "Setup k8s snodes" | ||
| eval $DEPLOY_CMD | ||
| else | ||
| echo "Setup docker snodes" | ||
| echo "making sure the cluster is clean" | ||
| helm uninstall sb-controller || true | ||
| helm uninstall simplyblock-csi || true | ||
| kubectl get pods | grep snode-spdk-pod | awk '{print $1}' | xargs kubectl delete pod || true | ||
| eval $DEPLOY_CMD | ||
| fi | ||
| echo "mnodes=$MNODES" >> ${GITHUB_OUTPUT:-/dev/stdout} | ||
| echo "storage_private_ips=${STORAGE_PRIVATE_IPS}" >> ${GITHUB_OUTPUT:-/dev/stdout} | ||
| env: | ||
| NDCS: ${{ inputs.ndcs }} | ||
| NPCS: ${{ inputs.npcs }} | ||
| BS: ${{ inputs.bs }} | ||
| CHUNK_BS: ${{ inputs.chunk_bs }} | ||
| NR_HUGEPAGES: 2048 | ||
| SBCLI_BRANCH: ${{ inputs.sbcli_source }} | ||
| SIMPLY_BLOCK_DOCKER_IMAGE: ${{ inputs.docker_image }} | ||
| k8s_snodes: |
Comment on lines
+217
to
+279
| needs: [k8s, deploy] | ||
| runs-on: ${{ inputs.runs_on }} | ||
| if: ${{ inputs.k8s_snode == true }} | ||
| steps: | ||
| - name: Checkout spdk-csi | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: simplyblock-io/simplyblock-csi | ||
| path: simplyblock-csi | ||
| - name: Deploy Snode on K8s | ||
| run: | | ||
| set -e | ||
| helm install simplyblock-csi ./simplyblock-csi/charts/spdk-csi/latest/spdk-csi \ | ||
| --set csiConfig.simplybk.uuid=${CLUSTER_ID} \ | ||
| --set csiConfig.simplybk.ip=${CLUSTER_IP} \ | ||
| --set csiSecret.simplybk.secret=${CLUSTER_SECRET} \ | ||
| --set logicalVolume.pool_name=testing1 \ | ||
| --set image.simplyblock.tag=${SBCLI_BRANCH} \ | ||
| --set image.csi.tag=latest \ | ||
| --set logicalVolume.numDataChunks=${NDCS} \ | ||
| --set logicalVolume.numParityChunks=${NPCS} \ | ||
| --set storagenode.create=true \ | ||
| --set storagenode.numPartitions=1 \ | ||
| --set image.storageNode.tag=latest \ | ||
| --set autoClusterActivate=true | ||
| env: | ||
| NDCS: ${{ inputs.ndcs }} | ||
| NPCS: ${{ inputs.npcs }} | ||
| CLUSTER_ID: ${{ needs.deploy.outputs.cluster_id }} | ||
| CLUSTER_IP: ${{ needs.deploy.outputs.cluster_ip }} | ||
| CLUSTER_SECRET: ${{ needs.deploy.outputs.cluster_secret }} | ||
| SBCLI_BRANCH: ${{ inputs.sbcli_source }} | ||
| MNODE: ${{ needs.deploy.outputs.mnodes }} | ||
| - name: Check Cluster Status | ||
| run: | | ||
| n=0 | ||
| until [ "$n" -ge 100 ] | ||
| do | ||
| response=$(curl -s "${CLUSTER_API_GATEWAY_ENDPOINT}/clusters/${CLUSTER_UUID}/" \ | ||
| -H "Content-Type: application/json" \ | ||
| -H "Authorization: Bearer ${CLUSTER_SECRET}") | ||
| status=$(echo $response | jq -r '.status') | ||
| if [ "$status" != "active" ]; then | ||
| echo "Cluster status is not active, current status: $status, retrying" | ||
| n=$((n+1)) | ||
| sleep 10 | ||
| else | ||
| echo "Cluster status is active" | ||
| exit 0 | ||
| fi | ||
| done | ||
| echo "Cluster status is not active" | ||
| exit 1 | ||
| env: | ||
| CLUSTER_API_GATEWAY_ENDPOINT: ${{ needs.deploy.outputs.cluster_ip }}/api/v2 | ||
| CLUSTER_UUID: ${{ needs.deploy.outputs.cluster_id }} | ||
| CLUSTER_SECRET: ${{ needs.deploy.outputs.cluster_secret }} |
Comment on lines
+111
to
+1186
| runs-on: ${{ github.event.inputs.cluster_environment == 'aws-openshift' && 'vm-runner-43' || 'self-hosted' }} | ||
| timeout-minutes: 4320 | ||
| concurrency: | ||
| group: ${{ github.workflow }} | ||
| cancel-in-progress: false | ||
| env: | ||
| TESTNAME: K8sNativeAddNodeTest | ||
| KEY_PATH: ${{ github.event.inputs.key_path || '/home/ec2-user/.ssh/simplyblock-us-east-2.pem' }} | ||
| SSH_USER: ${{ github.event.inputs.ssh_user || 'root' }} | ||
| SSH_PASSWORD: ${{ secrets.SSH_PASSWORD }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - uses: actions/checkout@master | ||
| name: Checkout helm-charts | ||
| with: | ||
| repository: simplyblock/helm-charts | ||
| ref: ${{ github.event.inputs.helm_charts_branch || 'main' }} | ||
| path: 'helm-charts' | ||
|
|
||
| - name: Resolve KEY_PATH and validate key exists | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
|
|
||
| kp="${KEY_PATH}" | ||
|
|
||
| # Strip wrapping quotes | ||
| kp="${kp%\"}"; kp="${kp#\"}" | ||
| kp="${kp%\'}"; kp="${kp#\'}" | ||
|
|
||
| # Normalize ".ssh/..." -> "$HOME/.ssh/..." | ||
| if [[ "$kp" == .ssh/* ]]; then | ||
| kp="${HOME}/${kp}" | ||
| fi | ||
|
|
||
| # Normalize "~/" -> "$HOME/" | ||
| if [[ "$kp" == ~/* ]]; then | ||
| kp="${HOME}/${kp#~/}" | ||
| fi | ||
|
|
||
| echo "Resolved KEY_PATH=$kp" | ||
| echo "KEY_PATH=$kp" >> "$GITHUB_ENV" | ||
|
|
||
| test -f "$kp" || (echo "ERROR: SSH key not found at $kp" && exit 1) | ||
| chmod 600 "$kp" || true | ||
|
|
||
| - name: Export KEY_NAME from KEY_PATH | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
| key_name="$(basename "${KEY_PATH}")" | ||
| echo "KEY_NAME=${key_name}" >> "$GITHUB_ENV" | ||
| echo "Exported KEY_NAME=${key_name}" | ||
|
|
||
| - name: Setup kubeconfig | ||
| run: | | ||
| mkdir -p $HOME/.kube | ||
| echo "${KUBECONFIG_DATA}" > $HOME/.kube/config | ||
| chmod 600 $HOME/.kube/config | ||
| env: | ||
| KUBECONFIG_DATA: ${{ | ||
| github.event.inputs.cluster_environment == 'aws-openshift' && secrets.KUBECONFIG_AWS_OPENSHIFT || | ||
| github.event.inputs.cluster_environment == 'openshift-local' && secrets.KUBECONFIG_OPENSHIFT_LOCAL || | ||
| github.event.inputs.cluster_environment == 'gcp' && secrets.KUBECONFIG_GCP || | ||
| secrets.KUBECONFIG_LOCAL || secrets.KUBECONFIG_CONTENT | ||
| }} | ||
|
|
||
| - name: Install Helm v3.15.4 | ||
| run: | | ||
| if ! helm version 2>/dev/null | grep -q 'v3.15'; then | ||
| curl -L https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz -o helm-v3.15.4-linux-amd64.tar.gz | ||
| tar -zxvf helm-v3.15.4-linux-amd64.tar.gz | ||
| sudo mv linux-amd64/helm /usr/local/bin/helm | ||
| rm -rf linux-amd64 helm-v3.15.4-linux-amd64.tar.gz | ||
| fi | ||
| helm version | ||
|
|
||
| - name: Install kubectl v1.31.0 | ||
| run: | | ||
| if ! kubectl version --client 2>/dev/null | grep -q 'v1.31'; then | ||
| curl -LO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl" | ||
| chmod +x kubectl | ||
| sudo mv kubectl /usr/local/bin/ | ||
| fi | ||
| kubectl version --client | ||
|
|
||
| - name: Set parameters | ||
| run: | | ||
| echo "NDCS=${{ github.event.inputs.ndcs || 1 }}" >> $GITHUB_ENV | ||
| echo "NPCS=${{ github.event.inputs.npcs || 1 }}" >> $GITHUB_ENV | ||
| echo "BS=${{ github.event.inputs.bs || 4096 }}" >> $GITHUB_ENV | ||
| echo "CHUNK_BS=${{ github.event.inputs.chunk_bs || 4096 }}" >> $GITHUB_ENV | ||
|
|
||
| - name: Verify kubectl connectivity | ||
| run: kubectl get nodes | ||
|
|
||
| # ── Cleanup old deployment ────────────────────────────────────────────── | ||
|
|
||
| - name: Cleanup old CSI deployment | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| set +e | ||
| NAMESPACE=simplyblock | ||
|
|
||
| echo "=== Phase 1: Helm uninstall ===" | ||
| helm uninstall spdk-csi -n $NAMESPACE 2>/dev/null || true | ||
|
|
||
| echo "=== Phase 2: Patch finalizers and delete CRs ===" | ||
| RESOURCES=( | ||
| "simplyblockpool.simplyblock.simplyblock.io simplyblock-pool" | ||
| "simplyblockpool.simplyblock.simplyblock.io simplyblock-pool2" | ||
| "simplyblocklvol.simplyblock.simplyblock.io simplyblock-lvol" | ||
| "simplyblocktask.simplyblock.simplyblock.io simplyblock-task" | ||
| "simplyblockdevices.simplyblock.simplyblock.io simplyblock-devices" | ||
| "simplyblockdevices.simplyblock.simplyblock.io simplyblock-device-action" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node2" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node-action" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster2" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster-activate" | ||
| "simplyblocksnapshotreplications.simplyblock.simplyblock.io simplyblock-snap-replication" | ||
| "simplyblocksnapshotreplications.simplyblock.simplyblock.io simplyblock-snap-replication-failback" | ||
| "pool.storage.simplyblock.io simplyblock-pool" | ||
| "pool.storage.simplyblock.io simplyblock-pool2" | ||
| "lvol.storage.simplyblock.io simplyblock-lvol" | ||
| "task.storage.simplyblock.io simplyblock-task" | ||
| "devices.storage.simplyblock.io simplyblock-devices" | ||
| "devices.storage.simplyblock.io simplyblock-device-action" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node2" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node-action" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster2" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster-activate" | ||
| "snapshotreplications.storage.simplyblock.io simplyblock-snap-replication" | ||
| "snapshotreplications.storage.simplyblock.io simplyblock-snap-replication-failback" | ||
| ) | ||
|
|
||
| patch_and_delete_crs() { | ||
| echo "Removing finalizers..." | ||
| for item in "${RESOURCES[@]}"; do | ||
| KIND=$(echo "$item" | awk '{print $1}') | ||
| NAME=$(echo "$item" | awk '{print $2}') | ||
| kubectl -n $NAMESPACE patch "$KIND" "$NAME" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "Deleting resources..." | ||
| for item in "${RESOURCES[@]}"; do | ||
| KIND=$(echo "$item" | awk '{print $1}') | ||
| NAME=$(echo "$item" | awk '{print $2}') | ||
| kubectl -n $NAMESPACE delete "$KIND" "$NAME" \ | ||
| --ignore-not-found --wait=false 2>/dev/null || true | ||
| done | ||
| } | ||
| patch_and_delete_crs | ||
|
|
||
| echo "=== Phase 3a: Delete VolumeSnapshots & VolumeSnapshotContents ===" | ||
| for VS in $(kubectl get volumesnapshot -n $NAMESPACE --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl -n $NAMESPACE patch volumesnapshot "$VS" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl -n $NAMESPACE delete volumesnapshot "$VS" --wait=false 2>/dev/null || true | ||
| done | ||
|
|
||
| for VSC in $(kubectl get volumesnapshotcontent --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl patch volumesnapshotcontent "$VSC" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl delete volumesnapshotcontent "$VSC" --wait=false 2>/dev/null || true | ||
| done | ||
|
|
||
| for VSCLASS in $(kubectl get volumesnapshotclass --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl delete volumesnapshotclass "$VSCLASS" --ignore-not-found 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3b: Delete PVCs (with timeout + retry) ===" | ||
| kubectl -n $NAMESPACE delete pvc --all --wait=false 2>/dev/null || true | ||
|
|
||
| PVC_TIMEOUT=60 | ||
| while [ $PVC_TIMEOUT -gt 0 ]; do | ||
| REMAINING=$(kubectl -n $NAMESPACE get pvc --no-headers 2>/dev/null | wc -l) | ||
| if [ "$REMAINING" -eq 0 ]; then | ||
| echo "All PVCs deleted" | ||
| break | ||
| fi | ||
| echo "Waiting for $REMAINING PVCs to delete ($PVC_TIMEOUT s remaining)..." | ||
| sleep 5 | ||
| PVC_TIMEOUT=$((PVC_TIMEOUT - 5)) | ||
| done | ||
|
|
||
| for PVC in $(kubectl -n $NAMESPACE get pvc --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| echo "Force-deleting stuck PVC: $PVC" | ||
| kubectl -n $NAMESPACE patch pvc "$PVC" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl -n $NAMESPACE delete pvc "$PVC" --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3c: Delete PVs ===" | ||
| for PV in $(kubectl get pv --no-headers -o custom-columns=:metadata.name 2>/dev/null | grep -i simplyblock 2>/dev/null); do | ||
| kubectl patch pv "$PV" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl delete pv "$PV" --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3d: Force delete remaining namespaced resources ===" | ||
| for RTYPE in pod jobs service ds statefulset deployment replicaset secret sa configmap; do | ||
| kubectl -n $NAMESPACE delete $RTYPE --all --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 5: Verify nothing remains ===" | ||
| echo "Namespaced resources:" | ||
| kubectl -n $NAMESPACE get all 2>/dev/null || echo "No resources found" | ||
| echo "" | ||
| echo "CRDs:" | ||
| for CR_TYPE in \ | ||
| "simplyblockpool.simplyblock.simplyblock.io" \ | ||
| "simplyblocklvol.simplyblock.simplyblock.io" \ | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io" \ | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io" \ | ||
| "pool.storage.simplyblock.io" \ | ||
| "lvol.storage.simplyblock.io" \ | ||
| "storagenodes.storage.simplyblock.io" \ | ||
| "storageclusters.storage.simplyblock.io"; do | ||
| kubectl -n $NAMESPACE get "$CR_TYPE" 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 6: Delete namespace ===" | ||
| kubectl delete namespace $NAMESPACE --wait=false 2>/dev/null || true | ||
|
|
||
| for i in $(seq 1 36); do | ||
| if ! kubectl get namespace $NAMESPACE 2>/dev/null; then | ||
| echo "Namespace $NAMESPACE deleted" | ||
| break | ||
| fi | ||
|
|
||
| echo "Namespace still terminating, re-patching finalizers ($i/36)..." | ||
| patch_and_delete_crs | ||
|
|
||
| if [ "$i" -ge 6 ]; then | ||
| echo "Force-removing namespace finalizers..." | ||
| kubectl get namespace $NAMESPACE -o json 2>/dev/null | \ | ||
| jq '.spec.finalizers = []' | \ | ||
| kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f - 2>/dev/null || true | ||
| fi | ||
|
|
||
| sleep 5 | ||
| done | ||
|
|
||
| echo "=== Phase 7: Delete Released PVs from simplyblock ===" | ||
| for pv in $(kubectl get pv --no-headers 2>/dev/null | grep 'simplyblock/' | awk '{print $1}'); do | ||
| echo "Deleting PV $pv" | ||
| kubectl delete pv "$pv" --ignore-not-found 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Cleanup complete ===" | ||
|
|
||
| # ── Label nodes ───────────────────────────────────────────────────────── | ||
|
|
||
| - name: Label initial worker nodes | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| for NODE in "${NODES[@]}"; do | ||
| if [ "$CLUSTER_ENV" = "local" ] || [ "$CLUSTER_ENV" = "openshift-local" ]; then | ||
| echo "Labeling node $NODE with zone=default (local cluster)" | ||
| kubectl label node "$NODE" topology.kubernetes.io/zone=default --overwrite | ||
| else | ||
| echo "Skipping zone label for $NODE (cloud cluster: $CLUSTER_ENV)" | ||
| fi | ||
| kubectl label node "$NODE" simplyblock.io/role=mgmt-plane --overwrite | ||
| done | ||
|
|
||
| - name: Label new worker nodes | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.new_worker_nodes }}" | ||
| for NODE in "${NODES[@]}"; do | ||
| if [ "$CLUSTER_ENV" = "local" ] || [ "$CLUSTER_ENV" = "openshift-local" ]; then | ||
| echo "Labeling new node $NODE with zone=default (local cluster)" | ||
| kubectl label node "$NODE" topology.kubernetes.io/zone=default --overwrite | ||
| else | ||
| echo "Skipping zone label for new node $NODE (cloud cluster: $CLUSTER_ENV)" | ||
| fi | ||
| kubectl label node "$NODE" simplyblock.io/role=mgmt-plane --overwrite | ||
| done | ||
|
|
||
| # ── Prepare namespace ─────────────────────────────────────────────────── | ||
|
|
||
| - name: Create namespace and set pod-security labels | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| kubectl create namespace simplyblock --dry-run=client -o yaml | kubectl apply -f - | ||
| kubectl label namespace simplyblock \ | ||
| pod-security.kubernetes.io/enforce=privileged \ | ||
| pod-security.kubernetes.io/warn=privileged \ | ||
| pod-security.kubernetes.io/audit=privileged \ | ||
| --overwrite | ||
|
|
||
| - name: Configure OpenShift SCC policies | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| oc adm policy add-scc-to-user privileged -z default -n simplyblock | ||
| oc adm policy add-scc-to-user anyuid -z default -n simplyblock | ||
| oc label namespace simplyblock \ | ||
| pod-security.kubernetes.io/enforce=privileged \ | ||
| pod-security.kubernetes.io/audit=privileged \ | ||
| pod-security.kubernetes.io/warn=privileged \ | ||
| --overwrite | ||
|
|
||
| - name: Wait before helm install | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: sleep 30 | ||
|
|
||
| # ── Deploy CSI stack ──────────────────────────────────────────────────── | ||
|
|
||
| - name: Install Helm Chart for simplyblock-operator | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| cd $GITHUB_WORKSPACE/helm-charts/charts/simplyblock-operator/ | ||
|
|
||
| helm upgrade --install spdk-csi ./ \ | ||
| --namespace simplyblock \ | ||
| --create-namespace \ | ||
| --debug \ | ||
| --timeout 10m \ | ||
| --set controlplane.enabled=true \ | ||
| --set operator.enabled=true \ | ||
| --set controlplane.csiHostpathDriver.enabled=true \ | ||
| --set controlplane.storageclass.name=local-hostpath \ | ||
| --set image.simplyblock.repository="${{ github.event.inputs.simplyblock_repository || 'public.ecr.aws/simply-block/simplyblock' }}" \ | ||
| --set image.simplyblock.tag="${{ github.event.inputs.simplyblock_image }}" \ | ||
| --set image.operator.repository="${{ github.event.inputs.operator_repository || 'simplyblock/simplyblock-operator' }}" \ | ||
| --set image.operator.tag="${{ github.event.inputs.operator_tag || 'main' }}" \ | ||
| --set csiConfig.simplybk.ip="http://simplyblock-webappapi.simplyblock:5000" \ | ||
| --set prometheus.server.persistentVolume.storageClass=local-hostpath \ | ||
| --set controlplane.observability.enabled=true \ | ||
| --set opensearch.persistence.storageClass=local-hostpath \ | ||
| --set image.simplyblock.pullPolicy=IfNotPresent \ | ||
| --set image.operator.pullPolicy=IfNotPresent | ||
|
|
||
| - name: Grant OpenShift SCC to all service accounts (post-helm) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| for sa in $(oc get sa -n simplyblock -o name | cut -d/ -f2); do | ||
| oc adm policy add-scc-to-user privileged -z $sa -n simplyblock | ||
| echo "Granted privileged SCC to $sa" | ||
| done | ||
|
|
||
| - name: Patch fluent-bit daemonset (post-helm) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== Waiting for simplyblock-fluent-bit daemonset (up to 5 min) ===" | ||
| PATCHED=false | ||
| for i in $(seq 1 30); do | ||
| if kubectl get daemonset simplyblock-fluent-bit -n $NAMESPACE &>/dev/null; then | ||
| echo "fluent-bit daemonset found, patching affinity..." | ||
| kubectl patch daemonset simplyblock-fluent-bit -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"affinity":null}}}}' | ||
| echo "fluent-bit daemonset patched successfully" | ||
| # Also patch CSI node daemonset with client toleration | ||
| if kubectl get daemonset simplyblock-csi-node -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-csi-node -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role","operator":"Equal","value":"client","effect":"NoSchedule"}]}}}}' | ||
| echo "csi-node daemonset patched with client toleration" | ||
| fi | ||
| PATCHED=true | ||
| break | ||
| fi | ||
| echo "fluent-bit daemonset not found yet ($i/30), waiting 10s..." | ||
| sleep 10 | ||
| done | ||
| echo "FLUENTBIT_PATCHED=$PATCHED" >> $GITHUB_ENV | ||
|
|
||
| # ── Apply custom resources (initial nodes only) ──────────────────────── | ||
|
|
||
| - name: Wait for operator pod to be ready | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== Waiting for operator pod ===" | ||
| for i in $(seq 1 60); do | ||
| POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "$POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Operator pod $POD is Running" | ||
| break | ||
| fi | ||
| fi | ||
| echo "Waiting for operator pod ($i/60)..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| - name: Apply operator custom resources | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| MGMT_IFC="${{ github.event.inputs.mgmt_ifc || 'ens18' }}" | ||
| SB_REPO="${{ github.event.inputs.simplyblock_repository || 'public.ecr.aws/simply-block/simplyblock' }}" | ||
| SB_TAG="${{ github.event.inputs.simplyblock_image }}" | ||
| SPDK_IMAGE="${{ github.event.inputs.spdk_image }}" | ||
| DATA_NICS="${{ github.event.inputs.data_nics || 'enp1s0' }}" | ||
| PARTITIONS="${{ github.event.inputs.partitions || '0' }}" | ||
| JM_COUNT="${{ github.event.inputs.journal_manager_count || '3' }}" | ||
| OPENSHIFT_CLUSTER="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'true' || 'false' }}" | ||
| CORE_PERCENTAGE="${{ github.event.inputs.cluster_environment == 'aws-openshift' && '50' || github.event.inputs.cluster_environment == 'openshift-local' && '50' || github.event.inputs.cluster_environment == 'local' && '35' || '65' }}" | ||
| SKIP_KUBELET_CONFIG="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'false' || 'true' }}" | ||
| FORCE_FORMAT_4K="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'true' || 'false' }}" | ||
|
|
||
| # Build workerNodes YAML list from initial worker nodes only | ||
| WORKER_YAML="" | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| for NODE in "${NODES[@]}"; do | ||
| WORKER_YAML="${WORKER_YAML} - ${NODE}"$'\n' | ||
| done | ||
|
|
||
| echo "=== Applying custom resources (initial ${#NODES[@]} workers) ===" | ||
| cat <<EOF | kubectl apply -f - | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageCluster | ||
| metadata: | ||
| name: simplyblock-cluster | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| mgmtIfname: ${MGMT_IFC} | ||
| fabricType: tcp | ||
| isSingleNode: false | ||
| enableNodeAffinity: true | ||
| strictNodeAntiAffinity: false | ||
| stripe: | ||
| dataChunks: ${NDCS} | ||
| parityChunks: ${NPCS} | ||
| warningThreshold: | ||
| capacity: 95 | ||
| provisionedCapacity: 97 | ||
| criticalThreshold: | ||
| capacity: 96 | ||
| provisionedCapacity: 98 | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: Pool | ||
| metadata: | ||
| name: simplyblock-pool | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| name: simplyblock-pool | ||
| clusterName: simplyblock-cluster | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageNode | ||
| metadata: | ||
| name: simplyblock-node | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| clusterImage: "${SB_REPO}:${SB_TAG}" | ||
| spdkImage: "${SPDK_IMAGE}" | ||
| mgmtIfname: ${MGMT_IFC} | ||
| dataIfname: | ||
| - ${DATA_NICS} | ||
| maxLogicalVolumeCount: 30 | ||
| partitions: ${PARTITIONS} | ||
| corePercentage: ${CORE_PERCENTAGE} | ||
| coreIsolation: false | ||
| journalManager: | ||
| count: ${JM_COUNT} | ||
| percentPerDevice: 3 | ||
| skipKubeletConfiguration: ${SKIP_KUBELET_CONFIG} | ||
| enableCpuTopology: true | ||
| openShiftCluster: ${OPENSHIFT_CLUSTER} | ||
| ubuntuHost: false | ||
| forceFormat4K: ${FORCE_FORMAT_4K} | ||
| workerNodes: | ||
| ${WORKER_YAML} | ||
| EOF | ||
|
|
||
| echo "Custom resources applied with initial workers" | ||
| sleep 10 | ||
| env: | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
|
|
||
| - name: Grant OpenShift SCC and restart storage daemonset (post-CR) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| echo "Waiting for simplyblock-storage-node-sa service account..." | ||
| for i in $(seq 1 60); do | ||
| if kubectl get sa simplyblock-storage-node-sa -n simplyblock &>/dev/null; then | ||
| echo "Service account simplyblock-storage-node-sa found" | ||
| break | ||
| fi | ||
| echo "Attempt $i/60: SA not found yet, waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "Granting privileged SCC to all service accounts..." | ||
| for sa in $(oc get sa -n simplyblock -o name | cut -d/ -f2); do | ||
| oc adm policy add-scc-to-user privileged -z $sa -n simplyblock | ||
| echo "Granted privileged SCC to $sa" | ||
| done | ||
|
|
||
| echo "Waiting for storage node daemonset to be created..." | ||
| for i in $(seq 1 60); do | ||
| if kubectl get daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock &>/dev/null; then | ||
| echo "Daemonset found, waiting for rollout to complete..." | ||
| kubectl rollout status daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock --timeout=300s | ||
| echo "Restarting daemonset..." | ||
| kubectl rollout restart daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock | ||
| break | ||
| fi | ||
| echo "Attempt $i/60: daemonset not found yet, waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| # ── Wait for cluster readiness ────────────────────────────────────────── | ||
|
|
||
| - name: Wait for cluster to become active | ||
| timeout-minutes: ${{ github.event.inputs.cluster_environment == 'local' && 25 || 80 }} | ||
| id: cluster_status | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| ADMIN_POD="" | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| if [ "$CLUSTER_ENV" = "local" ]; then | ||
| MAX_POLL=180 | ||
| else | ||
| MAX_POLL=720 | ||
| fi | ||
|
|
||
| echo "=== Waiting for admin pod to be ready ===" | ||
| for i in $(seq 1 60); do | ||
| ADMIN_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
|
|
||
| if [ -n "$ADMIN_POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$ADMIN_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Admin pod $ADMIN_POD is Running" | ||
| break | ||
| fi | ||
| echo "Admin pod $ADMIN_POD phase=$PHASE ($i/60)" | ||
| else | ||
| echo "Admin pod not found yet ($i/60)" | ||
| fi | ||
| sleep 10 | ||
| done | ||
|
|
||
| if [ -z "$ADMIN_POD" ]; then | ||
| echo "ERROR: Admin pod never appeared" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Expected number of snode-spdk pods = number of initial worker nodes | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| EXPECTED_SNODES=${#NODES[@]} | ||
| echo "=== Waiting for $EXPECTED_SNODES snode-spdk pods to be ready (MAX_POLL=$MAX_POLL) ===" | ||
|
|
||
| for i in $(seq 1 $MAX_POLL); do | ||
| TOTAL=$(kubectl -n $NAMESPACE get pods -l role=simplyblock-storage-node --no-headers 2>/dev/null | wc -l) | ||
| READY=$(kubectl -n $NAMESPACE get pods -l role=simplyblock-storage-node --no-headers 2>/dev/null | grep -c "Running" || true) | ||
|
|
||
| if [ "$READY" -ge "$EXPECTED_SNODES" ]; then | ||
| echo "All $READY/$EXPECTED_SNODES snode-spdk pods are Running" | ||
| break | ||
| fi | ||
|
|
||
| echo "snode-spdk pods: $READY/$EXPECTED_SNODES Running (total=$TOTAL) ($i/$MAX_POLL)" | ||
| if [ "$i" -eq "$MAX_POLL" ]; then | ||
| echo "ERROR: snode-spdk pods did not become ready within timeout" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "=== Re-resolving admin pod before cluster status polling ===" | ||
| ADMIN_POD="" | ||
| for i in $(seq 1 30); do | ||
| ADMIN_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "$ADMIN_POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$ADMIN_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Admin pod resolved: $ADMIN_POD (Running)" | ||
| break | ||
| fi | ||
| echo "Admin pod $ADMIN_POD phase=$PHASE, retrying ($i/30)..." | ||
| ADMIN_POD="" | ||
| else | ||
| echo "Admin pod not found, retrying ($i/30)..." | ||
| fi | ||
| sleep 10 | ||
| done | ||
| if [ -z "$ADMIN_POD" ]; then | ||
| echo "ERROR: Could not find running admin pod before polling" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "=== Polling cluster status (MAX_POLL=$MAX_POLL) ===" | ||
| for i in $(seq 1 $MAX_POLL); do | ||
| # Re-resolve admin pod each iteration in case it gets recreated | ||
| NEW_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| NEW_PHASE=$(kubectl -n $NAMESPACE get pod "$NEW_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ -n "$NEW_POD" ] && [ "$NEW_PHASE" = "Running" ] && [ "$NEW_POD" != "$ADMIN_POD" ]; then | ||
| echo "Admin pod changed: $ADMIN_POD -> $NEW_POD" | ||
| ADMIN_POD="$NEW_POD" | ||
| fi | ||
|
|
||
| echo "[poll $i/$MAX_POLL] Using admin pod: $ADMIN_POD" | ||
| OUTPUT=$(kubectl -n $NAMESPACE exec "$ADMIN_POD" -- \ | ||
| sbctl cluster list 2>&1) || true | ||
| echo "[poll $i/$MAX_POLL] sbctl cluster list output:" | ||
| echo "$OUTPUT" | ||
|
|
||
| if echo "$OUTPUT" | grep -qi "active"; then | ||
| echo "Cluster is active!" | ||
|
|
||
| # Extract cluster ID and secret | ||
| CLUSTER_ID=$(echo "$OUTPUT" | awk 'NR==4{print $2}') | ||
| CLUSTER_SECRET=$(echo "$OUTPUT" | awk 'NR==4{print $NF}') | ||
|
|
||
| # Fallback: try JSON format if table parsing fails | ||
| if [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" = "+" ]; then | ||
| echo "Table parsing failed (CLUSTER_ID=$CLUSTER_ID), trying JSON..." | ||
| JSON_OUT=$(kubectl -n $NAMESPACE exec "$ADMIN_POD" -- \ | ||
| sbctl cluster list --json 2>&1) || true | ||
| echo "JSON output: $JSON_OUT" | ||
| CLUSTER_ID=$(echo "$JSON_OUT" | jq -r '.[0].id // .[0].uuid // empty') | ||
| CLUSTER_SECRET=$(echo "$JSON_OUT" | jq -r '.[0].secret // empty') | ||
| fi | ||
|
|
||
| echo "Parsed CLUSTER_ID=${CLUSTER_ID}" | ||
| echo "CLUSTER_ID=${CLUSTER_ID}" >> $GITHUB_ENV | ||
| echo "CLUSTER_SECRET=${CLUSTER_SECRET}" >> $GITHUB_ENV | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo "Cluster not active yet ($i/$MAX_POLL)..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "ERROR: Cluster did not become active within timeout" | ||
| kubectl -n $NAMESPACE get pods | ||
| kubectl -n $NAMESPACE exec "$ADMIN_POD" -- sbctl cluster list 2>&1 || true | ||
| exit 1 | ||
|
|
||
| - name: Patch fluent-bit daemonset (post-cluster-active) | ||
| if: ${{ env.FLUENTBIT_PATCHED != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== fluent-bit was not patched after helm install, retrying now ===" | ||
| for i in $(seq 1 30); do | ||
| if kubectl get daemonset simplyblock-fluent-bit -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-fluent-bit -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"affinity":null}}}}' | ||
| echo "fluent-bit daemonset patched successfully" | ||
| # Also patch CSI node daemonset with client toleration | ||
| if kubectl get daemonset simplyblock-csi-node -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-csi-node -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role","operator":"Equal","value":"client","effect":"NoSchedule"}]}}}}' | ||
| echo "csi-node daemonset patched with client toleration" | ||
| fi | ||
| break | ||
| fi | ||
| echo "fluent-bit daemonset not found yet ($i/30), waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| # ── Run tests ─────────────────────────────────────────────────────────── | ||
|
|
||
| - name: Set RUN_DIR_FILE | ||
| run: echo "RUN_DIR_FILE=/tmp/sb_k8s_run_dir_${GITHUB_RUN_ID}_${GITHUB_RUN_ATTEMPT}.txt" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Record Test Start Time | ||
| run: echo "TEST_START_TIME=$(date +%s)" >> $GITHUB_ENV | ||
|
|
||
| - name: Setup and Run Tests | ||
| run: | | ||
| set -o pipefail | ||
| cd $GITHUB_WORKSPACE/e2e | ||
| python3 -m venv myenv | ||
| source myenv/bin/activate | ||
| python3 -m pip install -r requirements.txt | ||
|
|
||
| export CLUSTER_ID="${{ env.CLUSTER_ID }}" | ||
| export CLUSTER_SECRET="${{ env.CLUSTER_SECRET }}" | ||
| export K8S_LOCAL_KUBECTL=1 | ||
| export SBCLI_CMD=sbctl | ||
|
|
||
| python3 -u e2e.py \ | ||
| --testname K8sNativeAddNodeTest \ | ||
| --new_worker_nodes "${{ github.event.inputs.new_worker_nodes }}" \ | ||
| --ndcs $NDCS --npcs $NPCS --bs $BS --chunk_bs $CHUNK_BS \ | ||
| --run_k8s True 2>&1 | tee output.log | ||
| env: | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| BS: ${{ env.BS }} | ||
| CHUNK_BS: ${{ env.CHUNK_BS }} | ||
| SUPABASE_ANON_KEY: ${{ secrets.SUPABASE_ANON_KEY }} | ||
| SKIP_NFS: ${{ github.event.inputs.skip_nfs || 'false' }} | ||
| RUN_DIR_FILE: ${{ env.RUN_DIR_FILE }} | ||
|
|
||
| # ── Post-test: timing, logs, notifications ────────────────────────────── | ||
|
|
||
| - name: Export RUN_BASE_DIR from RUN_DIR_FILE | ||
| if: always() | ||
| run: | | ||
| test -f "${RUN_DIR_FILE}" || (echo "RUN_DIR_FILE not found — skipping"; exit 0) | ||
| RUN_BASE_DIR="$(cat "${RUN_DIR_FILE}" | tr -d '\r\n')" | ||
| [[ -n "${RUN_BASE_DIR}" ]] && echo "RUN_BASE_DIR=${RUN_BASE_DIR}" >> "$GITHUB_ENV" || true | ||
|
|
||
| - name: Record Test End Time | ||
| if: always() | ||
| run: echo "TEST_END_TIME=$(date +%s)" >> $GITHUB_ENV | ||
|
|
||
| - name: Calculate Total Time Taken | ||
| if: always() | ||
| run: | | ||
| TEST_TIME=$(($TEST_END_TIME - $TEST_START_TIME)) | ||
| TEST_TIME_HOURS=$(($TEST_TIME / 3600)) | ||
| TEST_TIME_MINS=$((($TEST_TIME % 3600) / 60)) | ||
| TEST_TIME_SECS=$(($TEST_TIME % 60)) | ||
| echo "Test runtime: ${TEST_TIME_HOURS}h ${TEST_TIME_MINS}m ${TEST_TIME_SECS}s" | ||
| echo "TEST_TIME_HOURS=$TEST_TIME_HOURS" >> $GITHUB_ENV | ||
| echo "TEST_TIME_MINS=$TEST_TIME_MINS" >> $GITHUB_ENV | ||
| echo "TEST_TIME_SECS=$TEST_TIME_SECS" >> $GITHUB_ENV | ||
|
|
||
| - name: Collect Graylog/OpenSearch logs | ||
| if: always() | ||
| timeout-minutes: 240 | ||
| run: | | ||
| set +e | ||
| echo "=== Collecting Graylog/OpenSearch logs (per-hour chunks) ===" | ||
| NAMESPACE=simplyblock | ||
|
|
||
| if [ -z "${TEST_START_TIME}" ] || [ -z "${TEST_END_TIME}" ]; then | ||
| echo "WARN: TEST_START_TIME or TEST_END_TIME not set, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| ELAPSED=$((TEST_END_TIME - TEST_START_TIME)) | ||
| if [ "${ELAPSED}" -le 0 ]; then | ||
| echo "WARN: Elapsed time is zero or negative, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Total window: 1h before test start → 1h after test end | ||
| WINDOW_START=$((TEST_START_TIME - 3600)) | ||
| WINDOW_END=$((TEST_END_TIME + 3600)) | ||
| TOTAL_SECONDS=$((WINDOW_END - WINDOW_START)) | ||
| TOTAL_HOURS=$(( (TOTAL_SECONDS + 3599) / 3600 )) | ||
| echo " Total window: ${TOTAL_HOURS} hour(s) to collect" | ||
|
|
||
| # Resolve admin pod | ||
| ADMIN_POD="" | ||
| for i in $(seq 1 12); do | ||
| ADMIN_POD=$(kubectl -n ${NAMESPACE} get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "${ADMIN_POD}" ]; then | ||
| PHASE=$(kubectl -n ${NAMESPACE} get pod "${ADMIN_POD}" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| [ "${PHASE}" = "Running" ] && break | ||
| ADMIN_POD="" | ||
| fi | ||
| sleep 10 | ||
| done | ||
| if [ -z "${ADMIN_POD}" ]; then | ||
| echo "ERROR: No running admin pod found, skipping" | ||
| exit 0 | ||
| fi | ||
| echo " Admin pod: ${ADMIN_POD}" | ||
|
|
||
| # Get Graylog service ClusterIP for mgmt-ip | ||
| MGMT_IP=$(kubectl get svc -n ${NAMESPACE} | grep graylog | awk '{print $3}') | ||
| echo " Graylog IP: ${MGMT_IP}" | ||
|
|
||
| # Determine output dir from RUN_BASE_DIR | ||
| OUTPUT_DIR="" | ||
| if [ -n "${RUN_BASE_DIR:-}" ] && [ -d "${RUN_BASE_DIR}" ]; then | ||
| OUTPUT_DIR="${RUN_BASE_DIR}/graylog_collected" | ||
| else | ||
| NFS_BASE="${NFS_MOUNTPOINT:-/mnt/nfs_share}" | ||
| TIMESTAMP=$(date -u "+%Y%m%d-%H%M%S") | ||
| OUTPUT_DIR="${NFS_BASE}/graylog_collected-${TIMESTAMP}" | ||
| fi | ||
| mkdir -p "${OUTPUT_DIR}" 2>/dev/null || true | ||
| echo " Output: ${OUTPUT_DIR}" | ||
|
|
||
| # Helper: convert epoch to ISO-8601 | ||
| epoch_to_iso() { | ||
| local ep=$1 | ||
| local iso | ||
| iso=$(date -u -d "@${ep}" "+%Y-%m-%dT%H:%M:%S" 2>/dev/null) | ||
| if [ -z "${iso}" ]; then | ||
| iso=$(python3 -c "from datetime import datetime,timezone; print(datetime.fromtimestamp(${ep},tz=timezone.utc).strftime('%Y-%m-%dT%H:%M:%S'))") | ||
| fi | ||
| echo "${iso}" | ||
| } | ||
|
|
||
| # Collect logs one hour at a time | ||
| CHUNK=0 | ||
| CURRENT=${WINDOW_START} | ||
| while [ ${CURRENT} -lt ${WINDOW_END} ]; do | ||
| CHUNK=$((CHUNK + 1)) | ||
| CHUNK_END=$((CURRENT + 3600)) | ||
| # Cap the last chunk at WINDOW_END | ||
| if [ ${CHUNK_END} -gt ${WINDOW_END} ]; then | ||
| CHUNK_END=${WINDOW_END} | ||
| fi | ||
| CHUNK_SECONDS=$((CHUNK_END - CURRENT)) | ||
| CHUNK_MINUTES=$(( (CHUNK_SECONDS + 59) / 60 )) | ||
| CHUNK_ISO=$(epoch_to_iso ${CURRENT}) | ||
|
|
||
| echo "" | ||
| echo "--- Chunk ${CHUNK}/${TOTAL_HOURS}: ${CHUNK_ISO} for ${CHUNK_MINUTES}m ---" | ||
|
|
||
| POD_OUTPUT_DIR="/tmp/graylog_collect_chunk${CHUNK}" | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- mkdir -p "${POD_OUTPUT_DIR}" 2>/dev/null || true | ||
|
|
||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| python3 /usr/local/lib/python3.12/site-packages/simplyblock_core/scripts/collect_logs.py \ | ||
| "${CHUNK_ISO}" "${CHUNK_MINUTES}" \ | ||
| --mode kubernetes \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --monitoring-secret "${MON_SECRET}" \ | ||
| --output-dir "${POD_OUTPUT_DIR}" \ | ||
| ${MGMT_IP:+--mgmt-ip "${MGMT_IP}"} \ | ||
| ${CLUSTER_ID:+--cluster-id "${CLUSTER_ID}"} \ | ||
| 2>&1 || { | ||
| echo "WARN: Graylog collect failed for chunk ${CHUNK}, retrying with --use-opensearch..." | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| python3 /usr/local/lib/python3.12/site-packages/simplyblock_core/scripts/collect_logs.py \ | ||
| "${CHUNK_ISO}" "${CHUNK_MINUTES}" \ | ||
| --mode kubernetes \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --monitoring-secret "${MON_SECRET}" \ | ||
| --output-dir "${POD_OUTPUT_DIR}" \ | ||
| --use-opensearch \ | ||
| ${MGMT_IP:+--mgmt-ip "${MGMT_IP}"} \ | ||
| ${CLUSTER_ID:+--cluster-id "${CLUSTER_ID}"} \ | ||
| 2>&1 || echo "WARN: OpenSearch fallback also failed for chunk ${CHUNK}" | ||
| } | ||
|
|
||
| # Copy tarballs from pod to NFS for this chunk | ||
| TARBALLS=$(kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| find "${POD_OUTPUT_DIR}" -name "*.tar.gz" -type f 2>/dev/null) || true | ||
|
|
||
| if [ -n "${TARBALLS}" ]; then | ||
| for TB in ${TARBALLS}; do | ||
| TB_NAME=$(basename "${TB}") | ||
| echo " Copying ${TB_NAME}..." | ||
| kubectl -n ${NAMESPACE} cp "${ADMIN_POD}:${TB}" "${OUTPUT_DIR}/${TB_NAME}" 2>&1 || true | ||
| done | ||
| for TB_FILE in "${OUTPUT_DIR}"/*.tar.gz; do | ||
| [ -f "${TB_FILE}" ] && tar -xzf "${TB_FILE}" -C "${OUTPUT_DIR}/" 2>/dev/null || true | ||
| done | ||
| else | ||
| echo "WARN: No tarballs found for chunk ${CHUNK}" | ||
| fi | ||
|
|
||
| # Cleanup this chunk in pod | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- rm -rf "${POD_OUTPUT_DIR}" 2>/dev/null || true | ||
|
|
||
| CURRENT=${CHUNK_END} | ||
| done | ||
|
|
||
| echo "" | ||
| echo "=== Graylog collection complete (${CHUNK} chunks): ${OUTPUT_DIR} ===" | ||
| ls -la "${OUTPUT_DIR}/" 2>/dev/null || true | ||
| env: | ||
| CLUSTER_ID: ${{ env.CLUSTER_ID }} | ||
| MON_SECRET: ${{ secrets.MON_SECRET }} | ||
|
|
||
| - name: Write Job Summary | ||
| if: always() | ||
| shell: bash | ||
| run: | | ||
| set +e | ||
| out_log="$GITHUB_WORKSPACE/e2e/output.log" | ||
|
|
||
| dur_fmt="${TEST_TIME_HOURS:-0}h ${TEST_TIME_MINS:-0}m ${TEST_TIME_SECS:-0}s" | ||
|
|
||
| total_cases="$(grep -E 'Number of Total Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| passed_cnt="$(grep -E 'Number of Passed Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| failed_cnt="$(grep -E 'Number of Failed Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| skipped_cnt="$(grep -E 'Number of Skipped Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '0')" | ||
| test_wise="$(grep -E '(PASSED|FAILED|SKIPPED) CASE' "${out_log}" 2>/dev/null \ | ||
| | sed 's/\x1b\[[0-9;]*m//g' | sed 's/.*INFO - //' || true)" | ||
|
|
||
| failure_summary="(unknown)" | ||
| log_tail="" | ||
| if [[ -f "${out_log}" ]]; then | ||
| failure_summary="$(grep -oE 'MultipleExceptions: .+|SkippedTestsException: .+' "${out_log}" \ | ||
| | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| if [[ -z "${failure_summary}" ]]; then | ||
| failure_summary="$(grep -E 'RuntimeError:|AssertionError:|Error:' "${out_log}" \ | ||
| | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| fi | ||
| [[ -z "${failure_summary}" ]] && failure_summary="(no exception line found)" | ||
| log_tail="$(tail -n 50 "${out_log}" | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| fi | ||
|
|
||
| conclusion="SUCCESS" | ||
| if [[ "${{ job.status }}" != "success" ]]; then | ||
| conclusion="FAILED" | ||
| fi | ||
|
|
||
| { | ||
| echo "## K8s Native Add Node Summary" | ||
| echo "" | ||
| echo "**Result:** ${conclusion}" | ||
| echo "" | ||
| echo "### Run Info" | ||
| echo "- **Test class:** \`K8sNativeAddNodeTest\`" | ||
| echo "- **Cluster ID:** \`${CLUSTER_ID}\`" | ||
| echo "- **Branch:** \`${{ github.ref_name }}\`" | ||
| echo "- **Helm Charts Branch:** \`${{ github.event.inputs.helm_charts_branch }}\`" | ||
| echo "- **NDCS/NPCS:** \`${NDCS}/${NPCS}\`" | ||
| echo "- **BS/Chunk BS:** \`${BS}/${CHUNK_BS}\`" | ||
| echo "- **Initial Workers:** \`${{ github.event.inputs.worker_nodes }}\`" | ||
| echo "- **New Workers:** \`${{ github.event.inputs.new_worker_nodes }}\`" | ||
| echo "- **Duration:** ${dur_fmt}" | ||
| echo "" | ||
| echo "### Test Results" | ||
| echo "- **Total:** ${total_cases} | **Passed:** ${passed_cnt} | **Failed:** ${failed_cnt} | **Skipped:** ${skipped_cnt}" | ||
| if [[ -n "${test_wise}" ]]; then | ||
| echo '```' | ||
| printf '%s\n' "${test_wise}" | ||
| echo '```' | ||
| fi | ||
| echo "" | ||
| echo "### Failure Reason" | ||
| echo '```' | ||
| printf '%s\n' "${failure_summary}" | ||
| echo '```' | ||
| if [[ -n "${log_tail}" ]]; then | ||
| echo "" | ||
| echo "<details><summary>Last 50 lines of output.log</summary>" | ||
| echo "" | ||
| echo '```' | ||
| printf '%s\n' "${log_tail}" | ||
| echo '```' | ||
| echo "</details>" | ||
| fi | ||
| echo "" | ||
| echo "### NFS Log Locations" | ||
| echo "- **RUN_BASE_DIR:** \`${RUN_BASE_DIR:-not detected}\`" | ||
| echo "- Graylog logs: \`${RUN_BASE_DIR:-<run_dir>}/graylog_collected/\`" | ||
| if [[ -n "${RUN_BASE_DIR:-}" && -d "${RUN_BASE_DIR}" ]]; then | ||
| echo "" | ||
| echo "<details><summary>NFS directory listing</summary>" | ||
| echo "" | ||
| echo '```' | ||
| ls -la "${RUN_BASE_DIR}/" 2>/dev/null || echo "(empty)" | ||
| echo '```' | ||
| echo "</details>" | ||
| fi | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| env: | ||
| CLUSTER_ID: ${{ env.CLUSTER_ID }} | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| BS: ${{ env.BS }} | ||
| CHUNK_BS: ${{ env.CHUNK_BS }} | ||
| RUN_BASE_DIR: ${{ env.RUN_BASE_DIR }} | ||
| TEST_TIME_HOURS: ${{ env.TEST_TIME_HOURS }} | ||
| TEST_TIME_MINS: ${{ env.TEST_TIME_MINS }} | ||
| TEST_TIME_SECS: ${{ env.TEST_TIME_SECS }} | ||
|
|
||
| - name: Send Slack Notification | ||
| if: always() && (github.event.inputs.send_slack_notification || 'true') == 'true' | ||
| shell: bash | ||
| env: | ||
| SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} | ||
| JOB_STATUS: ${{ job.status }} | ||
| SLACK_RUN_URL: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | ||
| GITHUB_REF_NAME: ${{ github.ref_name }} | ||
| SLACK_WF_NAME: "K8s Native Add Node" | ||
| TESTNAME: K8sNativeAddNodeTest | ||
| RUN_BASE_DIR: ${{ env.RUN_BASE_DIR }} | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| TEST_START_TIME: ${{ env.TEST_START_TIME }} | ||
| TEST_END_TIME: ${{ env.TEST_END_TIME }} | ||
| run: | | ||
| python3 - <<'PYEOF' | ||
| import json, os, re, sys, urllib.request, urllib.error | ||
| webhook = os.environ.get("SLACK_WEBHOOK_URL", "") | ||
| if not webhook: | ||
| print("No SLACK_WEBHOOK_URL set, skipping.") | ||
| sys.exit(0) | ||
| out_log = os.path.join(os.environ.get("GITHUB_WORKSPACE", "."), "e2e", "output.log") | ||
| total = passed = failed = skipped = 0 | ||
| if os.path.isfile(out_log): | ||
| content = open(out_log).read() | ||
| def px(pat): | ||
| m = re.search(pat, content) | ||
| return int(m.group(1)) if m else 0 | ||
| total = px(r'Number of Total Cases:\s*(\d+)') | ||
| passed = px(r'Number of Passed Cases:\s*(\d+)') | ||
| failed = px(r'Number of Failed Cases:\s*(\d+)') | ||
| skipped = px(r'Number of Skipped Cases:\s*(\d+)') | ||
| pass_pct = (passed * 100 // total) if total > 0 else 0 | ||
| s = int(os.environ.get("TEST_START_TIME", "0") or "0") | ||
| e = int(os.environ.get("TEST_END_TIME", "0") or "0") | ||
| secs = max(0, e - s) if e >= s > 0 else 0 | ||
| dur = f"{secs//3600}h {(secs%3600)//60}m {secs%60}s" | ||
| run_url = os.environ.get("SLACK_RUN_URL", "") | ||
| log_dir = os.environ.get("RUN_BASE_DIR", "N/A") | ||
| ndcs = os.environ.get("NDCS", "?") | ||
| npcs = os.environ.get("NPCS", "?") | ||
| test_cls = os.environ.get("TESTNAME", "") or "all" | ||
| branch = os.environ.get("GITHUB_REF_NAME", "?") | ||
| wf_name = os.environ.get("SLACK_WF_NAME", "Run") | ||
| ok = os.environ.get("JOB_STATUS", "") == "success" | ||
| icon = ":white_check_mark:" if ok else ":x:" | ||
| status = "SUCCESS" if ok else "FAILURE" | ||
| mention = "" if ok else " <!channel>" | ||
| lines = [ | ||
| f"{icon} *SimplyBlock {wf_name}*{mention}", | ||
| f"*Status:* {status} | *Duration:* {dur}", | ||
| f"*Branch:* `{branch}` | *NDCS/NPCS:* `{ndcs}/{npcs}` | *Test class:* `{test_cls}`", | ||
| "", | ||
| ] | ||
| if total > 0: | ||
| lines += [ | ||
| f":white_check_mark: *Passed:* {passed}/{total} ({pass_pct}%)", | ||
| f":x: *Failed:* {failed}", | ||
| f":fast_forward: *Skipped:* {skipped}", | ||
| ] | ||
| else: | ||
| lines.append("_(test counts not found in log)_") | ||
| lines += [ | ||
| "", | ||
| f":link: *Run:* <{run_url}|View on GitHub>", | ||
| f":file_folder: *Final Logs:* `{log_dir}`", | ||
| ] | ||
| payload = {"text": "\n".join(lines)} | ||
| req = urllib.request.Request( | ||
| webhook, | ||
| data=json.dumps(payload).encode(), | ||
| headers={"Content-Type": "application/json"}, | ||
| ) | ||
| try: | ||
| urllib.request.urlopen(req, timeout=15) | ||
| print("Slack notification sent.") | ||
| except Exception as exc: | ||
| print(f"WARN: Slack notification failed: {exc}", file=sys.stderr) | ||
| PYEOF | ||
|
|
||
| - name: Cleanup build folder | ||
| if: always() | ||
| run: | | ||
| rm -rf ./* || true | ||
| rm -rf ./.??* || true | ||
| rm -rf $HOME/.kube |
Comment on lines
+111
to
+1184
| runs-on: ${{ github.event.inputs.cluster_environment == 'aws-openshift' && 'vm-runner-43' || 'self-hosted' }} | ||
| timeout-minutes: 4320 | ||
| concurrency: | ||
| group: ${{ github.workflow }} | ||
| cancel-in-progress: false | ||
| env: | ||
| TESTNAME: K8sNativeNodeMigrationTest | ||
| KEY_PATH: ${{ github.event.inputs.key_path || '/home/ec2-user/.ssh/simplyblock-us-east-2.pem' }} | ||
| SSH_USER: ${{ github.event.inputs.ssh_user || 'root' }} | ||
| SSH_PASSWORD: ${{ secrets.SSH_PASSWORD }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - uses: actions/checkout@master | ||
| name: Checkout helm-charts | ||
| with: | ||
| repository: simplyblock/helm-charts | ||
| ref: ${{ github.event.inputs.helm_charts_branch || 'main' }} | ||
| path: 'helm-charts' | ||
|
|
||
| - name: Resolve KEY_PATH and validate key exists | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
|
|
||
| kp="${KEY_PATH}" | ||
|
|
||
| # Strip wrapping quotes | ||
| kp="${kp%\"}"; kp="${kp#\"}" | ||
| kp="${kp%\'}"; kp="${kp#\'}" | ||
|
|
||
| # Normalize ".ssh/..." -> "$HOME/.ssh/..." | ||
| if [[ "$kp" == .ssh/* ]]; then | ||
| kp="${HOME}/${kp}" | ||
| fi | ||
|
|
||
| # Normalize "~/" -> "$HOME/" | ||
| if [[ "$kp" == ~/* ]]; then | ||
| kp="${HOME}/${kp#~/}" | ||
| fi | ||
|
|
||
| echo "Resolved KEY_PATH=$kp" | ||
| echo "KEY_PATH=$kp" >> "$GITHUB_ENV" | ||
|
|
||
| test -f "$kp" || (echo "ERROR: SSH key not found at $kp" && exit 1) | ||
| chmod 600 "$kp" || true | ||
|
|
||
| - name: Export KEY_NAME from KEY_PATH | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
| key_name="$(basename "${KEY_PATH}")" | ||
| echo "KEY_NAME=${key_name}" >> "$GITHUB_ENV" | ||
| echo "Exported KEY_NAME=${key_name}" | ||
|
|
||
| - name: Setup kubeconfig | ||
| run: | | ||
| mkdir -p $HOME/.kube | ||
| echo "${KUBECONFIG_DATA}" > $HOME/.kube/config | ||
| chmod 600 $HOME/.kube/config | ||
| env: | ||
| KUBECONFIG_DATA: ${{ | ||
| github.event.inputs.cluster_environment == 'aws-openshift' && secrets.KUBECONFIG_AWS_OPENSHIFT || | ||
| github.event.inputs.cluster_environment == 'openshift-local' && secrets.KUBECONFIG_OPENSHIFT_LOCAL || | ||
| github.event.inputs.cluster_environment == 'gcp' && secrets.KUBECONFIG_GCP || | ||
| secrets.KUBECONFIG_LOCAL || secrets.KUBECONFIG_CONTENT | ||
| }} | ||
|
|
||
| - name: Install Helm v3.15.4 | ||
| run: | | ||
| if ! helm version 2>/dev/null | grep -q 'v3.15'; then | ||
| curl -L https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz -o helm-v3.15.4-linux-amd64.tar.gz | ||
| tar -zxvf helm-v3.15.4-linux-amd64.tar.gz | ||
| sudo mv linux-amd64/helm /usr/local/bin/helm | ||
| rm -rf linux-amd64 helm-v3.15.4-linux-amd64.tar.gz | ||
| fi | ||
| helm version | ||
|
|
||
| - name: Install kubectl v1.31.0 | ||
| run: | | ||
| if ! kubectl version --client 2>/dev/null | grep -q 'v1.31'; then | ||
| curl -LO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl" | ||
| chmod +x kubectl | ||
| sudo mv kubectl /usr/local/bin/ | ||
| fi | ||
| kubectl version --client | ||
|
|
||
| - name: Set parameters | ||
| run: | | ||
| echo "NDCS=${{ github.event.inputs.ndcs || 1 }}" >> $GITHUB_ENV | ||
| echo "NPCS=${{ github.event.inputs.npcs || 1 }}" >> $GITHUB_ENV | ||
| echo "BS=${{ github.event.inputs.bs || 4096 }}" >> $GITHUB_ENV | ||
| echo "CHUNK_BS=${{ github.event.inputs.chunk_bs || 4096 }}" >> $GITHUB_ENV | ||
|
|
||
| - name: Verify kubectl connectivity | ||
| run: kubectl get nodes | ||
|
|
||
| # ── Cleanup old deployment ────────────────────────────────────────────── | ||
|
|
||
| - name: Cleanup old CSI deployment | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| set +e | ||
| NAMESPACE=simplyblock | ||
|
|
||
| echo "=== Phase 1: Helm uninstall ===" | ||
| helm uninstall spdk-csi -n $NAMESPACE 2>/dev/null || true | ||
|
|
||
| echo "=== Phase 2: Patch finalizers and delete CRs ===" | ||
| RESOURCES=( | ||
| "simplyblockpool.simplyblock.simplyblock.io simplyblock-pool" | ||
| "simplyblockpool.simplyblock.simplyblock.io simplyblock-pool2" | ||
| "simplyblocklvol.simplyblock.simplyblock.io simplyblock-lvol" | ||
| "simplyblocktask.simplyblock.simplyblock.io simplyblock-task" | ||
| "simplyblockdevices.simplyblock.simplyblock.io simplyblock-devices" | ||
| "simplyblockdevices.simplyblock.simplyblock.io simplyblock-device-action" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node2" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node-action" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster2" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster-activate" | ||
| "simplyblocksnapshotreplications.simplyblock.simplyblock.io simplyblock-snap-replication" | ||
| "simplyblocksnapshotreplications.simplyblock.simplyblock.io simplyblock-snap-replication-failback" | ||
| "pool.storage.simplyblock.io simplyblock-pool" | ||
| "pool.storage.simplyblock.io simplyblock-pool2" | ||
| "lvol.storage.simplyblock.io simplyblock-lvol" | ||
| "task.storage.simplyblock.io simplyblock-task" | ||
| "devices.storage.simplyblock.io simplyblock-devices" | ||
| "devices.storage.simplyblock.io simplyblock-device-action" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node2" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node-action" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster2" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster-activate" | ||
| "snapshotreplications.storage.simplyblock.io simplyblock-snap-replication" | ||
| "snapshotreplications.storage.simplyblock.io simplyblock-snap-replication-failback" | ||
| ) | ||
|
|
||
| patch_and_delete_crs() { | ||
| echo "Removing finalizers..." | ||
| for item in "${RESOURCES[@]}"; do | ||
| KIND=$(echo "$item" | awk '{print $1}') | ||
| NAME=$(echo "$item" | awk '{print $2}') | ||
| kubectl -n $NAMESPACE patch "$KIND" "$NAME" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "Deleting resources..." | ||
| for item in "${RESOURCES[@]}"; do | ||
| KIND=$(echo "$item" | awk '{print $1}') | ||
| NAME=$(echo "$item" | awk '{print $2}') | ||
| kubectl -n $NAMESPACE delete "$KIND" "$NAME" \ | ||
| --ignore-not-found --wait=false 2>/dev/null || true | ||
| done | ||
| } | ||
| patch_and_delete_crs | ||
|
|
||
| echo "=== Phase 3a: Delete VolumeSnapshots & VolumeSnapshotContents ===" | ||
| for VS in $(kubectl get volumesnapshot -n $NAMESPACE --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl -n $NAMESPACE patch volumesnapshot "$VS" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl -n $NAMESPACE delete volumesnapshot "$VS" --wait=false 2>/dev/null || true | ||
| done | ||
|
|
||
| for VSC in $(kubectl get volumesnapshotcontent --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl patch volumesnapshotcontent "$VSC" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl delete volumesnapshotcontent "$VSC" --wait=false 2>/dev/null || true | ||
| done | ||
|
|
||
| for VSCLASS in $(kubectl get volumesnapshotclass --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl delete volumesnapshotclass "$VSCLASS" --ignore-not-found 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3b: Delete PVCs (with timeout + retry) ===" | ||
| kubectl -n $NAMESPACE delete pvc --all --wait=false 2>/dev/null || true | ||
|
|
||
| PVC_TIMEOUT=60 | ||
| while [ $PVC_TIMEOUT -gt 0 ]; do | ||
| REMAINING=$(kubectl -n $NAMESPACE get pvc --no-headers 2>/dev/null | wc -l) | ||
| if [ "$REMAINING" -eq 0 ]; then | ||
| echo "All PVCs deleted" | ||
| break | ||
| fi | ||
| echo "Waiting for $REMAINING PVCs to delete ($PVC_TIMEOUT s remaining)..." | ||
| sleep 5 | ||
| PVC_TIMEOUT=$((PVC_TIMEOUT - 5)) | ||
| done | ||
|
|
||
| for PVC in $(kubectl -n $NAMESPACE get pvc --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| echo "Force-deleting stuck PVC: $PVC" | ||
| kubectl -n $NAMESPACE patch pvc "$PVC" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl -n $NAMESPACE delete pvc "$PVC" --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3c: Delete PVs ===" | ||
| for PV in $(kubectl get pv --no-headers -o custom-columns=:metadata.name 2>/dev/null | grep -i simplyblock 2>/dev/null); do | ||
| kubectl patch pv "$PV" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl delete pv "$PV" --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3d: Force delete remaining namespaced resources ===" | ||
| for RTYPE in pod jobs service ds statefulset deployment replicaset secret sa configmap; do | ||
| kubectl -n $NAMESPACE delete $RTYPE --all --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 5: Verify nothing remains ===" | ||
| echo "Namespaced resources:" | ||
| kubectl -n $NAMESPACE get all 2>/dev/null || echo "No resources found" | ||
| echo "" | ||
| echo "CRDs:" | ||
| for CR_TYPE in \ | ||
| "simplyblockpool.simplyblock.simplyblock.io" \ | ||
| "simplyblocklvol.simplyblock.simplyblock.io" \ | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io" \ | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io" \ | ||
| "pool.storage.simplyblock.io" \ | ||
| "lvol.storage.simplyblock.io" \ | ||
| "storagenodes.storage.simplyblock.io" \ | ||
| "storageclusters.storage.simplyblock.io"; do | ||
| kubectl -n $NAMESPACE get "$CR_TYPE" 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 6: Delete namespace ===" | ||
| kubectl delete namespace $NAMESPACE --wait=false 2>/dev/null || true | ||
|
|
||
| for i in $(seq 1 36); do | ||
| if ! kubectl get namespace $NAMESPACE 2>/dev/null; then | ||
| echo "Namespace $NAMESPACE deleted" | ||
| break | ||
| fi | ||
|
|
||
| echo "Namespace still terminating, re-patching finalizers ($i/36)..." | ||
| patch_and_delete_crs | ||
|
|
||
| if [ "$i" -ge 6 ]; then | ||
| echo "Force-removing namespace finalizers..." | ||
| kubectl get namespace $NAMESPACE -o json 2>/dev/null | \ | ||
| jq '.spec.finalizers = []' | \ | ||
| kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f - 2>/dev/null || true | ||
| fi | ||
|
|
||
| sleep 5 | ||
| done | ||
|
|
||
| echo "=== Phase 7: Delete Released PVs from simplyblock ===" | ||
| for pv in $(kubectl get pv --no-headers 2>/dev/null | grep 'simplyblock/' | awk '{print $1}'); do | ||
| echo "Deleting PV $pv" | ||
| kubectl delete pv "$pv" --ignore-not-found 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Cleanup complete ===" | ||
|
|
||
| # ── Label nodes ───────────────────────────────────────────────────────── | ||
|
|
||
| - name: Label worker nodes | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| for NODE in "${NODES[@]}"; do | ||
| if [ "$CLUSTER_ENV" = "local" ] || [ "$CLUSTER_ENV" = "openshift-local" ]; then | ||
| echo "Labeling node $NODE with zone=default (local cluster)" | ||
| kubectl label node "$NODE" topology.kubernetes.io/zone=default --overwrite | ||
| else | ||
| echo "Skipping zone label for $NODE (cloud cluster: $CLUSTER_ENV)" | ||
| fi | ||
| kubectl label node "$NODE" simplyblock.io/role=mgmt-plane --overwrite | ||
| done | ||
|
|
||
| - name: Label migration target worker node | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| NODE="${{ github.event.inputs.migrate_to_worker }}" | ||
| echo "Labeling migration target node $NODE" | ||
| if [ "$CLUSTER_ENV" = "local" ] || [ "$CLUSTER_ENV" = "openshift-local" ]; then | ||
| kubectl label node "$NODE" topology.kubernetes.io/zone=default --overwrite | ||
| else | ||
| echo "Skipping zone label for migration target $NODE (cloud cluster: $CLUSTER_ENV)" | ||
| fi | ||
| kubectl label node "$NODE" simplyblock.io/role=mgmt-plane --overwrite | ||
|
|
||
| # ── Prepare namespace ─────────────────────────────────────────────────── | ||
|
|
||
| - name: Create namespace and set pod-security labels | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| kubectl create namespace simplyblock --dry-run=client -o yaml | kubectl apply -f - | ||
| kubectl label namespace simplyblock \ | ||
| pod-security.kubernetes.io/enforce=privileged \ | ||
| pod-security.kubernetes.io/warn=privileged \ | ||
| pod-security.kubernetes.io/audit=privileged \ | ||
| --overwrite | ||
|
|
||
| - name: Configure OpenShift SCC policies | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| oc adm policy add-scc-to-user privileged -z default -n simplyblock | ||
| oc adm policy add-scc-to-user anyuid -z default -n simplyblock | ||
| oc label namespace simplyblock \ | ||
| pod-security.kubernetes.io/enforce=privileged \ | ||
| pod-security.kubernetes.io/audit=privileged \ | ||
| pod-security.kubernetes.io/warn=privileged \ | ||
| --overwrite | ||
|
|
||
| - name: Wait before helm install | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: sleep 30 | ||
|
|
||
| # ── Deploy CSI stack ──────────────────────────────────────────────────── | ||
|
|
||
| - name: Install Helm Chart for simplyblock-operator | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| cd $GITHUB_WORKSPACE/helm-charts/charts/simplyblock-operator/ | ||
|
|
||
| helm upgrade --install spdk-csi ./ \ | ||
| --namespace simplyblock \ | ||
| --create-namespace \ | ||
| --debug \ | ||
| --timeout 10m \ | ||
| --set controlplane.enabled=true \ | ||
| --set operator.enabled=true \ | ||
| --set controlplane.csiHostpathDriver.enabled=true \ | ||
| --set controlplane.storageclass.name=local-hostpath \ | ||
| --set image.simplyblock.repository="${{ github.event.inputs.simplyblock_repository || 'public.ecr.aws/simply-block/simplyblock' }}" \ | ||
| --set image.simplyblock.tag="${{ github.event.inputs.simplyblock_image }}" \ | ||
| --set image.operator.repository="${{ github.event.inputs.operator_repository || 'simplyblock/simplyblock-operator' }}" \ | ||
| --set image.operator.tag="${{ github.event.inputs.operator_tag || 'main' }}" \ | ||
| --set csiConfig.simplybk.ip="http://simplyblock-webappapi.simplyblock:5000" \ | ||
| --set prometheus.server.persistentVolume.storageClass=local-hostpath \ | ||
| --set controlplane.observability.enabled=true \ | ||
| --set opensearch.persistence.storageClass=local-hostpath \ | ||
| --set image.simplyblock.pullPolicy=IfNotPresent \ | ||
| --set image.operator.pullPolicy=IfNotPresent | ||
|
|
||
| - name: Grant OpenShift SCC to all service accounts (post-helm) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| for sa in $(oc get sa -n simplyblock -o name | cut -d/ -f2); do | ||
| oc adm policy add-scc-to-user privileged -z $sa -n simplyblock | ||
| echo "Granted privileged SCC to $sa" | ||
| done | ||
|
|
||
| - name: Patch fluent-bit daemonset (post-helm) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== Waiting for simplyblock-fluent-bit daemonset (up to 5 min) ===" | ||
| PATCHED=false | ||
| for i in $(seq 1 30); do | ||
| if kubectl get daemonset simplyblock-fluent-bit -n $NAMESPACE &>/dev/null; then | ||
| echo "fluent-bit daemonset found, patching affinity..." | ||
| kubectl patch daemonset simplyblock-fluent-bit -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"affinity":null}}}}' | ||
| echo "fluent-bit daemonset patched successfully" | ||
| # Also patch CSI node daemonset with client toleration | ||
| if kubectl get daemonset simplyblock-csi-node -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-csi-node -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role","operator":"Equal","value":"client","effect":"NoSchedule"}]}}}}' | ||
| echo "csi-node daemonset patched with client toleration" | ||
| fi | ||
| PATCHED=true | ||
| break | ||
| fi | ||
| echo "fluent-bit daemonset not found yet ($i/30), waiting 10s..." | ||
| sleep 10 | ||
| done | ||
| echo "FLUENTBIT_PATCHED=$PATCHED" >> $GITHUB_ENV | ||
|
|
||
| # ── Apply custom resources ───────────────────────────────────────────── | ||
|
|
||
| - name: Wait for operator pod to be ready | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== Waiting for operator pod ===" | ||
| for i in $(seq 1 60); do | ||
| POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "$POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Operator pod $POD is Running" | ||
| break | ||
| fi | ||
| fi | ||
| echo "Waiting for operator pod ($i/60)..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| - name: Apply operator custom resources | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| MGMT_IFC="${{ github.event.inputs.mgmt_ifc || 'ens18' }}" | ||
| SB_REPO="${{ github.event.inputs.simplyblock_repository || 'public.ecr.aws/simply-block/simplyblock' }}" | ||
| SB_TAG="${{ github.event.inputs.simplyblock_image }}" | ||
| SPDK_IMAGE="${{ github.event.inputs.spdk_image }}" | ||
| DATA_NICS="${{ github.event.inputs.data_nics || 'enp1s0' }}" | ||
| PARTITIONS="${{ github.event.inputs.partitions || '0' }}" | ||
| JM_COUNT="${{ github.event.inputs.journal_manager_count || '3' }}" | ||
| OPENSHIFT_CLUSTER="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'true' || 'false' }}" | ||
| CORE_PERCENTAGE="${{ github.event.inputs.cluster_environment == 'aws-openshift' && '50' || github.event.inputs.cluster_environment == 'openshift-local' && '50' || github.event.inputs.cluster_environment == 'local' && '35' || '65' }}" | ||
| SKIP_KUBELET_CONFIG="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'false' || 'true' }}" | ||
| FORCE_FORMAT_4K="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'true' || 'false' }}" | ||
|
|
||
| # Build workerNodes YAML list from all worker nodes | ||
| WORKER_YAML="" | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| for NODE in "${NODES[@]}"; do | ||
| WORKER_YAML="${WORKER_YAML} - ${NODE}"$'\n' | ||
| done | ||
|
|
||
| echo "=== Applying custom resources (${#NODES[@]} workers) ===" | ||
| cat <<EOF | kubectl apply -f - | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageCluster | ||
| metadata: | ||
| name: simplyblock-cluster | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| mgmtIfname: ${MGMT_IFC} | ||
| fabricType: tcp | ||
| isSingleNode: false | ||
| enableNodeAffinity: true | ||
| strictNodeAntiAffinity: false | ||
| stripe: | ||
| dataChunks: ${NDCS} | ||
| parityChunks: ${NPCS} | ||
| warningThreshold: | ||
| capacity: 95 | ||
| provisionedCapacity: 97 | ||
| criticalThreshold: | ||
| capacity: 96 | ||
| provisionedCapacity: 98 | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: Pool | ||
| metadata: | ||
| name: simplyblock-pool | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| name: simplyblock-pool | ||
| clusterName: simplyblock-cluster | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageNode | ||
| metadata: | ||
| name: simplyblock-node | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| clusterImage: "${SB_REPO}:${SB_TAG}" | ||
| spdkImage: "${SPDK_IMAGE}" | ||
| mgmtIfname: ${MGMT_IFC} | ||
| dataIfname: | ||
| - ${DATA_NICS} | ||
| maxLogicalVolumeCount: 30 | ||
| partitions: ${PARTITIONS} | ||
| corePercentage: ${CORE_PERCENTAGE} | ||
| coreIsolation: false | ||
| journalManager: | ||
| count: ${JM_COUNT} | ||
| percentPerDevice: 3 | ||
| skipKubeletConfiguration: ${SKIP_KUBELET_CONFIG} | ||
| enableCpuTopology: true | ||
| openShiftCluster: ${OPENSHIFT_CLUSTER} | ||
| ubuntuHost: false | ||
| forceFormat4K: ${FORCE_FORMAT_4K} | ||
| workerNodes: | ||
| ${WORKER_YAML} | ||
| EOF | ||
|
|
||
| echo "Custom resources applied" | ||
| sleep 10 | ||
| env: | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
|
|
||
| - name: Grant OpenShift SCC and restart storage daemonset (post-CR) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && github.event.inputs.cluster_environment == 'aws-openshift' }} | ||
| run: | | ||
| echo "Waiting for simplyblock-storage-node-sa service account..." | ||
| for i in $(seq 1 60); do | ||
| if kubectl get sa simplyblock-storage-node-sa -n simplyblock &>/dev/null; then | ||
| echo "Service account simplyblock-storage-node-sa found" | ||
| break | ||
| fi | ||
| echo "Attempt $i/60: SA not found yet, waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "Granting privileged SCC to all service accounts..." | ||
| for sa in $(oc get sa -n simplyblock -o name | cut -d/ -f2); do | ||
| oc adm policy add-scc-to-user privileged -z $sa -n simplyblock | ||
| echo "Granted privileged SCC to $sa" | ||
| done | ||
|
|
||
| echo "Waiting for storage node daemonset to be created..." | ||
| for i in $(seq 1 60); do | ||
| if kubectl get daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock &>/dev/null; then | ||
| echo "Daemonset found, waiting for rollout to complete..." | ||
| kubectl rollout status daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock --timeout=300s | ||
| echo "Restarting daemonset..." | ||
| kubectl rollout restart daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock | ||
| break | ||
| fi | ||
| echo "Attempt $i/60: daemonset not found yet, waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| # ── Wait for cluster readiness ────────────────────────────────────────── | ||
|
|
||
| - name: Wait for cluster to become active | ||
| timeout-minutes: ${{ github.event.inputs.cluster_environment == 'local' && 25 || 80 }} | ||
| id: cluster_status | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| ADMIN_POD="" | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| if [ "$CLUSTER_ENV" = "local" ]; then | ||
| MAX_POLL=180 | ||
| else | ||
| MAX_POLL=720 | ||
| fi | ||
|
|
||
| echo "=== Waiting for admin pod to be ready ===" | ||
| for i in $(seq 1 60); do | ||
| ADMIN_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
|
|
||
| if [ -n "$ADMIN_POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$ADMIN_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Admin pod $ADMIN_POD is Running" | ||
| break | ||
| fi | ||
| echo "Admin pod $ADMIN_POD phase=$PHASE ($i/60)" | ||
| else | ||
| echo "Admin pod not found yet ($i/60)" | ||
| fi | ||
| sleep 10 | ||
| done | ||
|
|
||
| if [ -z "$ADMIN_POD" ]; then | ||
| echo "ERROR: Admin pod never appeared" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Expected number of snode-spdk pods = number of worker nodes | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| EXPECTED_SNODES=${#NODES[@]} | ||
| echo "=== Waiting for $EXPECTED_SNODES snode-spdk pods to be ready (MAX_POLL=$MAX_POLL) ===" | ||
|
|
||
| for i in $(seq 1 $MAX_POLL); do | ||
| TOTAL=$(kubectl -n $NAMESPACE get pods -l role=simplyblock-storage-node --no-headers 2>/dev/null | wc -l) | ||
| READY=$(kubectl -n $NAMESPACE get pods -l role=simplyblock-storage-node --no-headers 2>/dev/null | grep -c "Running" || true) | ||
|
|
||
| if [ "$READY" -ge "$EXPECTED_SNODES" ]; then | ||
| echo "All $READY/$EXPECTED_SNODES snode-spdk pods are Running" | ||
| break | ||
| fi | ||
|
|
||
| echo "snode-spdk pods: $READY/$EXPECTED_SNODES Running (total=$TOTAL) ($i/$MAX_POLL)" | ||
| if [ "$i" -eq "$MAX_POLL" ]; then | ||
| echo "ERROR: snode-spdk pods did not become ready within timeout" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "=== Re-resolving admin pod before cluster status polling ===" | ||
| ADMIN_POD="" | ||
| for i in $(seq 1 30); do | ||
| ADMIN_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "$ADMIN_POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$ADMIN_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Admin pod resolved: $ADMIN_POD (Running)" | ||
| break | ||
| fi | ||
| echo "Admin pod $ADMIN_POD phase=$PHASE, retrying ($i/30)..." | ||
| ADMIN_POD="" | ||
| else | ||
| echo "Admin pod not found, retrying ($i/30)..." | ||
| fi | ||
| sleep 10 | ||
| done | ||
| if [ -z "$ADMIN_POD" ]; then | ||
| echo "ERROR: Could not find running admin pod before polling" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "=== Polling cluster status (MAX_POLL=$MAX_POLL) ===" | ||
| for i in $(seq 1 $MAX_POLL); do | ||
| # Re-resolve admin pod each iteration in case it gets recreated | ||
| NEW_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| NEW_PHASE=$(kubectl -n $NAMESPACE get pod "$NEW_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ -n "$NEW_POD" ] && [ "$NEW_PHASE" = "Running" ] && [ "$NEW_POD" != "$ADMIN_POD" ]; then | ||
| echo "Admin pod changed: $ADMIN_POD -> $NEW_POD" | ||
| ADMIN_POD="$NEW_POD" | ||
| fi | ||
|
|
||
| echo "[poll $i/$MAX_POLL] Using admin pod: $ADMIN_POD" | ||
| OUTPUT=$(kubectl -n $NAMESPACE exec "$ADMIN_POD" -- \ | ||
| sbctl cluster list 2>&1) || true | ||
| echo "[poll $i/$MAX_POLL] sbctl cluster list output:" | ||
| echo "$OUTPUT" | ||
|
|
||
| if echo "$OUTPUT" | grep -qi "active"; then | ||
| echo "Cluster is active!" | ||
|
|
||
| # Extract cluster ID and secret | ||
| CLUSTER_ID=$(echo "$OUTPUT" | awk 'NR==4{print $2}') | ||
| CLUSTER_SECRET=$(echo "$OUTPUT" | awk 'NR==4{print $NF}') | ||
|
|
||
| # Fallback: try JSON format if table parsing fails | ||
| if [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" = "+" ]; then | ||
| echo "Table parsing failed (CLUSTER_ID=$CLUSTER_ID), trying JSON..." | ||
| JSON_OUT=$(kubectl -n $NAMESPACE exec "$ADMIN_POD" -- \ | ||
| sbctl cluster list --json 2>&1) || true | ||
| echo "JSON output: $JSON_OUT" | ||
| CLUSTER_ID=$(echo "$JSON_OUT" | jq -r '.[0].id // .[0].uuid // empty') | ||
| CLUSTER_SECRET=$(echo "$JSON_OUT" | jq -r '.[0].secret // empty') | ||
| fi | ||
|
|
||
| echo "Parsed CLUSTER_ID=${CLUSTER_ID}" | ||
| echo "CLUSTER_ID=${CLUSTER_ID}" >> $GITHUB_ENV | ||
| echo "CLUSTER_SECRET=${CLUSTER_SECRET}" >> $GITHUB_ENV | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo "Cluster not active yet ($i/$MAX_POLL)..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "ERROR: Cluster did not become active within timeout" | ||
| kubectl -n $NAMESPACE get pods | ||
| kubectl -n $NAMESPACE exec "$ADMIN_POD" -- sbctl cluster list 2>&1 || true | ||
| exit 1 | ||
|
|
||
| - name: Patch fluent-bit daemonset (post-cluster-active) | ||
| if: ${{ env.FLUENTBIT_PATCHED != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== fluent-bit was not patched after helm install, retrying now ===" | ||
| for i in $(seq 1 30); do | ||
| if kubectl get daemonset simplyblock-fluent-bit -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-fluent-bit -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"affinity":null}}}}' | ||
| echo "fluent-bit daemonset patched successfully" | ||
| # Also patch CSI node daemonset with client toleration | ||
| if kubectl get daemonset simplyblock-csi-node -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-csi-node -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role","operator":"Equal","value":"client","effect":"NoSchedule"}]}}}}' | ||
| echo "csi-node daemonset patched with client toleration" | ||
| fi | ||
| break | ||
| fi | ||
| echo "fluent-bit daemonset not found yet ($i/30), waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| # ── Run tests ─────────────────────────────────────────────────────────── | ||
|
|
||
| - name: Set RUN_DIR_FILE | ||
| run: echo "RUN_DIR_FILE=/tmp/sb_k8s_run_dir_${GITHUB_RUN_ID}_${GITHUB_RUN_ATTEMPT}.txt" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Record Test Start Time | ||
| run: echo "TEST_START_TIME=$(date +%s)" >> $GITHUB_ENV | ||
|
|
||
| - name: Setup and Run Tests | ||
| run: | | ||
| set -o pipefail | ||
| cd $GITHUB_WORKSPACE/e2e | ||
| python3 -m venv myenv | ||
| source myenv/bin/activate | ||
| python3 -m pip install -r requirements.txt | ||
|
|
||
| export CLUSTER_ID="${{ env.CLUSTER_ID }}" | ||
| export CLUSTER_SECRET="${{ env.CLUSTER_SECRET }}" | ||
| export K8S_LOCAL_KUBECTL=1 | ||
| export SBCLI_CMD=sbctl | ||
|
|
||
| python3 -u e2e.py \ | ||
| --testname K8sNativeNodeMigrationTest \ | ||
| --migrate_to_worker "${{ github.event.inputs.migrate_to_worker }}" \ | ||
| --ndcs $NDCS --npcs $NPCS --bs $BS --chunk_bs $CHUNK_BS \ | ||
| --run_k8s True 2>&1 | tee output.log | ||
| env: | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| BS: ${{ env.BS }} | ||
| CHUNK_BS: ${{ env.CHUNK_BS }} | ||
| SUPABASE_ANON_KEY: ${{ secrets.SUPABASE_ANON_KEY }} | ||
| SKIP_NFS: ${{ github.event.inputs.skip_nfs || 'false' }} | ||
| RUN_DIR_FILE: ${{ env.RUN_DIR_FILE }} | ||
|
|
||
| # ── Post-test: timing, logs, notifications ────────────────────────────── | ||
|
|
||
| - name: Export RUN_BASE_DIR from RUN_DIR_FILE | ||
| if: always() | ||
| run: | | ||
| test -f "${RUN_DIR_FILE}" || (echo "RUN_DIR_FILE not found — skipping"; exit 0) | ||
| RUN_BASE_DIR="$(cat "${RUN_DIR_FILE}" | tr -d '\r\n')" | ||
| [[ -n "${RUN_BASE_DIR}" ]] && echo "RUN_BASE_DIR=${RUN_BASE_DIR}" >> "$GITHUB_ENV" || true | ||
|
|
||
| - name: Record Test End Time | ||
| if: always() | ||
| run: echo "TEST_END_TIME=$(date +%s)" >> $GITHUB_ENV | ||
|
|
||
| - name: Calculate Total Time Taken | ||
| if: always() | ||
| run: | | ||
| TEST_TIME=$(($TEST_END_TIME - $TEST_START_TIME)) | ||
| TEST_TIME_HOURS=$(($TEST_TIME / 3600)) | ||
| TEST_TIME_MINS=$((($TEST_TIME % 3600) / 60)) | ||
| TEST_TIME_SECS=$(($TEST_TIME % 60)) | ||
| echo "Test runtime: ${TEST_TIME_HOURS}h ${TEST_TIME_MINS}m ${TEST_TIME_SECS}s" | ||
| echo "TEST_TIME_HOURS=$TEST_TIME_HOURS" >> $GITHUB_ENV | ||
| echo "TEST_TIME_MINS=$TEST_TIME_MINS" >> $GITHUB_ENV | ||
| echo "TEST_TIME_SECS=$TEST_TIME_SECS" >> $GITHUB_ENV | ||
|
|
||
| - name: Collect Graylog/OpenSearch logs | ||
| if: always() | ||
| timeout-minutes: 240 | ||
| run: | | ||
| set +e | ||
| echo "=== Collecting Graylog/OpenSearch logs (per-hour chunks) ===" | ||
| NAMESPACE=simplyblock | ||
|
|
||
| if [ -z "${TEST_START_TIME}" ] || [ -z "${TEST_END_TIME}" ]; then | ||
| echo "WARN: TEST_START_TIME or TEST_END_TIME not set, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| ELAPSED=$((TEST_END_TIME - TEST_START_TIME)) | ||
| if [ "${ELAPSED}" -le 0 ]; then | ||
| echo "WARN: Elapsed time is zero or negative, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Total window: 1h before test start → 1h after test end | ||
| WINDOW_START=$((TEST_START_TIME - 3600)) | ||
| WINDOW_END=$((TEST_END_TIME + 3600)) | ||
| TOTAL_SECONDS=$((WINDOW_END - WINDOW_START)) | ||
| TOTAL_HOURS=$(( (TOTAL_SECONDS + 3599) / 3600 )) | ||
| echo " Total window: ${TOTAL_HOURS} hour(s) to collect" | ||
|
|
||
| # Resolve admin pod | ||
| ADMIN_POD="" | ||
| for i in $(seq 1 12); do | ||
| ADMIN_POD=$(kubectl -n ${NAMESPACE} get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "${ADMIN_POD}" ]; then | ||
| PHASE=$(kubectl -n ${NAMESPACE} get pod "${ADMIN_POD}" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| [ "${PHASE}" = "Running" ] && break | ||
| ADMIN_POD="" | ||
| fi | ||
| sleep 10 | ||
| done | ||
| if [ -z "${ADMIN_POD}" ]; then | ||
| echo "ERROR: No running admin pod found, skipping" | ||
| exit 0 | ||
| fi | ||
| echo " Admin pod: ${ADMIN_POD}" | ||
|
|
||
| # Get Graylog service ClusterIP for mgmt-ip | ||
| MGMT_IP=$(kubectl get svc -n ${NAMESPACE} | grep graylog | awk '{print $3}') | ||
| echo " Graylog IP: ${MGMT_IP}" | ||
|
|
||
| # Determine output dir from RUN_BASE_DIR | ||
| OUTPUT_DIR="" | ||
| if [ -n "${RUN_BASE_DIR:-}" ] && [ -d "${RUN_BASE_DIR}" ]; then | ||
| OUTPUT_DIR="${RUN_BASE_DIR}/graylog_collected" | ||
| else | ||
| NFS_BASE="${NFS_MOUNTPOINT:-/mnt/nfs_share}" | ||
| TIMESTAMP=$(date -u "+%Y%m%d-%H%M%S") | ||
| OUTPUT_DIR="${NFS_BASE}/graylog_collected-${TIMESTAMP}" | ||
| fi | ||
| mkdir -p "${OUTPUT_DIR}" 2>/dev/null || true | ||
| echo " Output: ${OUTPUT_DIR}" | ||
|
|
||
| # Helper: convert epoch to ISO-8601 | ||
| epoch_to_iso() { | ||
| local ep=$1 | ||
| local iso | ||
| iso=$(date -u -d "@${ep}" "+%Y-%m-%dT%H:%M:%S" 2>/dev/null) | ||
| if [ -z "${iso}" ]; then | ||
| iso=$(python3 -c "from datetime import datetime,timezone; print(datetime.fromtimestamp(${ep},tz=timezone.utc).strftime('%Y-%m-%dT%H:%M:%S'))") | ||
| fi | ||
| echo "${iso}" | ||
| } | ||
|
|
||
| # Collect logs one hour at a time | ||
| CHUNK=0 | ||
| CURRENT=${WINDOW_START} | ||
| while [ ${CURRENT} -lt ${WINDOW_END} ]; do | ||
| CHUNK=$((CHUNK + 1)) | ||
| CHUNK_END=$((CURRENT + 3600)) | ||
| # Cap the last chunk at WINDOW_END | ||
| if [ ${CHUNK_END} -gt ${WINDOW_END} ]; then | ||
| CHUNK_END=${WINDOW_END} | ||
| fi | ||
| CHUNK_SECONDS=$((CHUNK_END - CURRENT)) | ||
| CHUNK_MINUTES=$(( (CHUNK_SECONDS + 59) / 60 )) | ||
| CHUNK_ISO=$(epoch_to_iso ${CURRENT}) | ||
|
|
||
| echo "" | ||
| echo "--- Chunk ${CHUNK}/${TOTAL_HOURS}: ${CHUNK_ISO} for ${CHUNK_MINUTES}m ---" | ||
|
|
||
| POD_OUTPUT_DIR="/tmp/graylog_collect_chunk${CHUNK}" | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- mkdir -p "${POD_OUTPUT_DIR}" 2>/dev/null || true | ||
|
|
||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| python3 /usr/local/lib/python3.12/site-packages/simplyblock_core/scripts/collect_logs.py \ | ||
| "${CHUNK_ISO}" "${CHUNK_MINUTES}" \ | ||
| --mode kubernetes \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --monitoring-secret "${MON_SECRET}" \ | ||
| --output-dir "${POD_OUTPUT_DIR}" \ | ||
| ${MGMT_IP:+--mgmt-ip "${MGMT_IP}"} \ | ||
| ${CLUSTER_ID:+--cluster-id "${CLUSTER_ID}"} \ | ||
| 2>&1 || { | ||
| echo "WARN: Graylog collect failed for chunk ${CHUNK}, retrying with --use-opensearch..." | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| python3 /usr/local/lib/python3.12/site-packages/simplyblock_core/scripts/collect_logs.py \ | ||
| "${CHUNK_ISO}" "${CHUNK_MINUTES}" \ | ||
| --mode kubernetes \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --monitoring-secret "${MON_SECRET}" \ | ||
| --output-dir "${POD_OUTPUT_DIR}" \ | ||
| --use-opensearch \ | ||
| ${MGMT_IP:+--mgmt-ip "${MGMT_IP}"} \ | ||
| ${CLUSTER_ID:+--cluster-id "${CLUSTER_ID}"} \ | ||
| 2>&1 || echo "WARN: OpenSearch fallback also failed for chunk ${CHUNK}" | ||
| } | ||
|
|
||
| # Copy tarballs from pod to NFS for this chunk | ||
| TARBALLS=$(kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| find "${POD_OUTPUT_DIR}" -name "*.tar.gz" -type f 2>/dev/null) || true | ||
|
|
||
| if [ -n "${TARBALLS}" ]; then | ||
| for TB in ${TARBALLS}; do | ||
| TB_NAME=$(basename "${TB}") | ||
| echo " Copying ${TB_NAME}..." | ||
| kubectl -n ${NAMESPACE} cp "${ADMIN_POD}:${TB}" "${OUTPUT_DIR}/${TB_NAME}" 2>&1 || true | ||
| done | ||
| for TB_FILE in "${OUTPUT_DIR}"/*.tar.gz; do | ||
| [ -f "${TB_FILE}" ] && tar -xzf "${TB_FILE}" -C "${OUTPUT_DIR}/" 2>/dev/null || true | ||
| done | ||
| else | ||
| echo "WARN: No tarballs found for chunk ${CHUNK}" | ||
| fi | ||
|
|
||
| # Cleanup this chunk in pod | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- rm -rf "${POD_OUTPUT_DIR}" 2>/dev/null || true | ||
|
|
||
| CURRENT=${CHUNK_END} | ||
| done | ||
|
|
||
| echo "" | ||
| echo "=== Graylog collection complete (${CHUNK} chunks): ${OUTPUT_DIR} ===" | ||
| ls -la "${OUTPUT_DIR}/" 2>/dev/null || true | ||
| env: | ||
| CLUSTER_ID: ${{ env.CLUSTER_ID }} | ||
| MON_SECRET: ${{ secrets.MON_SECRET }} | ||
|
|
||
| - name: Write Job Summary | ||
| if: always() | ||
| shell: bash | ||
| run: | | ||
| set +e | ||
| out_log="$GITHUB_WORKSPACE/e2e/output.log" | ||
|
|
||
| dur_fmt="${TEST_TIME_HOURS:-0}h ${TEST_TIME_MINS:-0}m ${TEST_TIME_SECS:-0}s" | ||
|
|
||
| total_cases="$(grep -E 'Number of Total Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| passed_cnt="$(grep -E 'Number of Passed Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| failed_cnt="$(grep -E 'Number of Failed Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| skipped_cnt="$(grep -E 'Number of Skipped Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '0')" | ||
| test_wise="$(grep -E '(PASSED|FAILED|SKIPPED) CASE' "${out_log}" 2>/dev/null \ | ||
| | sed 's/\x1b\[[0-9;]*m//g' | sed 's/.*INFO - //' || true)" | ||
|
|
||
| failure_summary="(unknown)" | ||
| log_tail="" | ||
| if [[ -f "${out_log}" ]]; then | ||
| failure_summary="$(grep -oE 'MultipleExceptions: .+|SkippedTestsException: .+' "${out_log}" \ | ||
| | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| if [[ -z "${failure_summary}" ]]; then | ||
| failure_summary="$(grep -E 'RuntimeError:|AssertionError:|Error:' "${out_log}" \ | ||
| | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| fi | ||
| [[ -z "${failure_summary}" ]] && failure_summary="(no exception line found)" | ||
| log_tail="$(tail -n 50 "${out_log}" | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| fi | ||
|
|
||
| conclusion="SUCCESS" | ||
| if [[ "${{ job.status }}" != "success" ]]; then | ||
| conclusion="FAILED" | ||
| fi | ||
|
|
||
| { | ||
| echo "## K8s Native Node Migration Summary" | ||
| echo "" | ||
| echo "**Result:** ${conclusion}" | ||
| echo "" | ||
| echo "### Run Info" | ||
| echo "- **Test class:** \`K8sNativeNodeMigrationTest\`" | ||
| echo "- **Cluster ID:** \`${CLUSTER_ID}\`" | ||
| echo "- **Branch:** \`${{ github.ref_name }}\`" | ||
| echo "- **Helm Charts Branch:** \`${{ github.event.inputs.helm_charts_branch }}\`" | ||
| echo "- **NDCS/NPCS:** \`${NDCS}/${NPCS}\`" | ||
| echo "- **BS/Chunk BS:** \`${BS}/${CHUNK_BS}\`" | ||
| echo "- **Workers:** \`${{ github.event.inputs.worker_nodes }}\`" | ||
| echo "- **Migrate To:** \`${{ github.event.inputs.migrate_to_worker }}\`" | ||
| echo "- **Duration:** ${dur_fmt}" | ||
| echo "" | ||
| echo "### Test Results" | ||
| echo "- **Total:** ${total_cases} | **Passed:** ${passed_cnt} | **Failed:** ${failed_cnt} | **Skipped:** ${skipped_cnt}" | ||
| if [[ -n "${test_wise}" ]]; then | ||
| echo '```' | ||
| printf '%s\n' "${test_wise}" | ||
| echo '```' | ||
| fi | ||
| echo "" | ||
| echo "### Failure Reason" | ||
| echo '```' | ||
| printf '%s\n' "${failure_summary}" | ||
| echo '```' | ||
| if [[ -n "${log_tail}" ]]; then | ||
| echo "" | ||
| echo "<details><summary>Last 50 lines of output.log</summary>" | ||
| echo "" | ||
| echo '```' | ||
| printf '%s\n' "${log_tail}" | ||
| echo '```' | ||
| echo "</details>" | ||
| fi | ||
| echo "" | ||
| echo "### NFS Log Locations" | ||
| echo "- **RUN_BASE_DIR:** \`${RUN_BASE_DIR:-not detected}\`" | ||
| echo "- Graylog logs: \`${RUN_BASE_DIR:-<run_dir>}/graylog_collected/\`" | ||
| if [[ -n "${RUN_BASE_DIR:-}" && -d "${RUN_BASE_DIR}" ]]; then | ||
| echo "" | ||
| echo "<details><summary>NFS directory listing</summary>" | ||
| echo "" | ||
| echo '```' | ||
| ls -la "${RUN_BASE_DIR}/" 2>/dev/null || echo "(empty)" | ||
| echo '```' | ||
| echo "</details>" | ||
| fi | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| env: | ||
| CLUSTER_ID: ${{ env.CLUSTER_ID }} | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| BS: ${{ env.BS }} | ||
| CHUNK_BS: ${{ env.CHUNK_BS }} | ||
| RUN_BASE_DIR: ${{ env.RUN_BASE_DIR }} | ||
| TEST_TIME_HOURS: ${{ env.TEST_TIME_HOURS }} | ||
| TEST_TIME_MINS: ${{ env.TEST_TIME_MINS }} | ||
| TEST_TIME_SECS: ${{ env.TEST_TIME_SECS }} | ||
|
|
||
| - name: Send Slack Notification | ||
| if: always() && (github.event.inputs.send_slack_notification || 'true') == 'true' | ||
| shell: bash | ||
| env: | ||
| SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} | ||
| JOB_STATUS: ${{ job.status }} | ||
| SLACK_RUN_URL: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | ||
| GITHUB_REF_NAME: ${{ github.ref_name }} | ||
| SLACK_WF_NAME: "K8s Native Node Migration" | ||
| TESTNAME: K8sNativeNodeMigrationTest | ||
| RUN_BASE_DIR: ${{ env.RUN_BASE_DIR }} | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| TEST_START_TIME: ${{ env.TEST_START_TIME }} | ||
| TEST_END_TIME: ${{ env.TEST_END_TIME }} | ||
| run: | | ||
| python3 - <<'PYEOF' | ||
| import json, os, re, sys, urllib.request, urllib.error | ||
| webhook = os.environ.get("SLACK_WEBHOOK_URL", "") | ||
| if not webhook: | ||
| print("No SLACK_WEBHOOK_URL set, skipping.") | ||
| sys.exit(0) | ||
| out_log = os.path.join(os.environ.get("GITHUB_WORKSPACE", "."), "e2e", "output.log") | ||
| total = passed = failed = skipped = 0 | ||
| if os.path.isfile(out_log): | ||
| content = open(out_log).read() | ||
| def px(pat): | ||
| m = re.search(pat, content) | ||
| return int(m.group(1)) if m else 0 | ||
| total = px(r'Number of Total Cases:\s*(\d+)') | ||
| passed = px(r'Number of Passed Cases:\s*(\d+)') | ||
| failed = px(r'Number of Failed Cases:\s*(\d+)') | ||
| skipped = px(r'Number of Skipped Cases:\s*(\d+)') | ||
| pass_pct = (passed * 100 // total) if total > 0 else 0 | ||
| s = int(os.environ.get("TEST_START_TIME", "0") or "0") | ||
| e = int(os.environ.get("TEST_END_TIME", "0") or "0") | ||
| secs = max(0, e - s) if e >= s > 0 else 0 | ||
| dur = f"{secs//3600}h {(secs%3600)//60}m {secs%60}s" | ||
| run_url = os.environ.get("SLACK_RUN_URL", "") | ||
| log_dir = os.environ.get("RUN_BASE_DIR", "N/A") | ||
| ndcs = os.environ.get("NDCS", "?") | ||
| npcs = os.environ.get("NPCS", "?") | ||
| test_cls = os.environ.get("TESTNAME", "") or "all" | ||
| branch = os.environ.get("GITHUB_REF_NAME", "?") | ||
| wf_name = os.environ.get("SLACK_WF_NAME", "Run") | ||
| ok = os.environ.get("JOB_STATUS", "") == "success" | ||
| icon = ":white_check_mark:" if ok else ":x:" | ||
| status = "SUCCESS" if ok else "FAILURE" | ||
| mention = "" if ok else " <!channel>" | ||
| lines = [ | ||
| f"{icon} *SimplyBlock {wf_name}*{mention}", | ||
| f"*Status:* {status} | *Duration:* {dur}", | ||
| f"*Branch:* `{branch}` | *NDCS/NPCS:* `{ndcs}/{npcs}` | *Test class:* `{test_cls}`", | ||
| "", | ||
| ] | ||
| if total > 0: | ||
| lines += [ | ||
| f":white_check_mark: *Passed:* {passed}/{total} ({pass_pct}%)", | ||
| f":x: *Failed:* {failed}", | ||
| f":fast_forward: *Skipped:* {skipped}", | ||
| ] | ||
| else: | ||
| lines.append("_(test counts not found in log)_") | ||
| lines += [ | ||
| "", | ||
| f":link: *Run:* <{run_url}|View on GitHub>", | ||
| f":file_folder: *Final Logs:* `{log_dir}`", | ||
| ] | ||
| payload = {"text": "\n".join(lines)} | ||
| req = urllib.request.Request( | ||
| webhook, | ||
| data=json.dumps(payload).encode(), | ||
| headers={"Content-Type": "application/json"}, | ||
| ) | ||
| try: | ||
| urllib.request.urlopen(req, timeout=15) | ||
| print("Slack notification sent.") | ||
| except Exception as exc: | ||
| print(f"WARN: Slack notification failed: {exc}", file=sys.stderr) | ||
| PYEOF | ||
|
|
||
| - name: Cleanup build folder | ||
| if: always() | ||
| run: | | ||
| rm -rf ./* || true | ||
| rm -rf ./.??* || true | ||
| rm -rf $HOME/.kube |
Comment on lines
+120
to
+1349
| runs-on: ${{ github.event.inputs.cluster_environment == 'aws-openshift' && 'vm-runner-43' || 'self-hosted' }} | ||
| timeout-minutes: 4320 | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.inputs.cluster_environment || 'local' }} | ||
| cancel-in-progress: false | ||
| env: | ||
| TESTNAME: ${{ github.event.inputs.testname || 'K8sNativeFailoverTest' }} | ||
| KEY_PATH: ${{ github.event.inputs.key_path || '/home/ec2-user/.ssh/simplyblock-us-east-2.pem' }} | ||
| CLIENT_IPS: ${{ github.event.inputs.client_ips || '' }} | ||
| NFS_MOUNTPOINT: ${{ github.event.inputs.nfs_mountpoint || '/mnt/nfs_share/' }} | ||
| SSH_USER: ${{ github.event.inputs.ssh_user || 'root' }} | ||
| SSH_PASSWORD: ${{ secrets.SSH_PASSWORD }} | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - uses: actions/checkout@master | ||
| name: Checkout helm-charts | ||
| with: | ||
| repository: simplyblock/helm-charts | ||
| ref: ${{ github.event.inputs.helm_charts_branch || 'main' }} | ||
| path: 'helm-charts' | ||
|
|
||
| - name: Resolve KEY_PATH (handles .ssh/, ~/.ssh/, quoted ~) and validate key exists | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
|
|
||
| kp="${KEY_PATH}" | ||
|
|
||
| # Strip wrapping quotes if user typed "~/.ssh/..." with quotes | ||
| kp="${kp%\"}"; kp="${kp#\"}" | ||
| kp="${kp%\'}"; kp="${kp#\'}" | ||
|
|
||
| # Normalize ".ssh/..." -> "$HOME/.ssh/..." | ||
| if [[ "$kp" == .ssh/* ]]; then | ||
| kp="${HOME}/${kp}" | ||
| fi | ||
|
|
||
| # Normalize "~/" -> "$HOME/" | ||
| if [[ "$kp" == ~/* ]]; then | ||
| kp="${HOME}/${kp#~/}" | ||
| fi | ||
|
|
||
| # Also handle "~.ssh/.." (unlikely, but safe) | ||
| if [[ "$kp" == "~.ssh/"* ]]; then | ||
| kp="${HOME}/.${kp#~.}" | ||
| fi | ||
|
|
||
| echo "Resolved KEY_PATH=$kp" | ||
| echo "KEY_PATH=$kp" >> "$GITHUB_ENV" | ||
|
|
||
| test -f "$kp" || (echo "ERROR: SSH key not found at $kp" && exit 1) | ||
| chmod 600 "$kp" || true | ||
|
|
||
| - name: Export KEY_NAME from KEY_PATH | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
| # KEY_PATH is already resolved and exported to $GITHUB_ENV in previous step | ||
| key_name="$(basename "${KEY_PATH}")" | ||
| echo "KEY_NAME=${key_name}" >> "$GITHUB_ENV" | ||
| echo "Exported KEY_NAME=${key_name}" | ||
|
|
||
| # ============================================================ | ||
| # PRE-BOOTSTRAP CLEANUP (remote ops only) | ||
| # Targets = CLIENTNODES | ||
| # ============================================================ | ||
| - name: Pre-clean kill fio/tmux and unmount NFS on clients | ||
| shell: bash | ||
| run: | | ||
| set -euxo pipefail | ||
|
|
||
| run_remote() { | ||
| local ip="$1" | ||
| local script="$2" | ||
| sshpass -p "${SSH_PASSWORD}" ssh \ | ||
| -o StrictHostKeyChecking=no \ | ||
| -o UserKnownHostsFile=/dev/null \ | ||
| "${SSH_USER}@${ip}" "bash -s" <<< "$script" | ||
| } | ||
|
|
||
| run_remote_with_retry() { | ||
| local ip="$1" | ||
| local script="$2" | ||
| local max=5 | ||
| for attempt in $(seq 1 $max); do | ||
| run_remote "$ip" "$script" && return 0 | ||
| echo "Attempt $attempt/$max failed for $ip, retrying in 5s..." | ||
| sleep 5 | ||
| done | ||
| echo "All $max attempts failed for $ip, continuing..." | ||
| return 0 | ||
| } | ||
|
|
||
| # Support comma-separated and/or space-separated input | ||
| normalized_targets="$(echo "${CLIENT_IPS}" | tr ',' ' ' | xargs -n1 | sort -u | xargs || true)" | ||
|
|
||
| if [ -z "${normalized_targets}" ]; then | ||
| echo "No client IPs provided, skipping client cleanup." | ||
| exit 0 | ||
| fi | ||
|
|
||
| for ip in ${normalized_targets}; do | ||
| echo "---- ${ip}: kill fio/tmux + disconnect NVMe + umount ${NFS_MOUNTPOINT} ----" | ||
| run_remote_with_retry "${ip}" "set -euxo pipefail; | ||
| pkill -9 fio || true; | ||
| pkill -9 tmux || true; | ||
| # Disconnect stale NVMe-oF connections from previous runs | ||
| for nqn in \$(nvme list-subsys -o json 2>/dev/null | jq -r '.Subsystems[]?.NQN // empty' | grep lvol || true); do | ||
| nvme disconnect -n \"\$nqn\" || true; | ||
| done; | ||
| mp='${NFS_MOUNTPOINT}'; | ||
| if mountpoint -q \"\$mp\"; then | ||
| umount -f \"\$mp\" || umount \"\$mp\" || true; | ||
| else | ||
| if mount | grep -q \" \$mp \"; then | ||
| umount -f \"\$mp\" || umount \"\$mp\" || true; | ||
| fi | ||
| fi" | ||
| done | ||
|
|
||
| - name: Setup kubeconfig | ||
| run: | | ||
| mkdir -p $HOME/.kube | ||
| echo "${KUBECONFIG_DATA}" > $HOME/.kube/config | ||
| chmod 600 $HOME/.kube/config | ||
| env: | ||
| KUBECONFIG_DATA: ${{ | ||
| github.event.inputs.cluster_environment == 'aws-openshift' && secrets.KUBECONFIG_AWS_OPENSHIFT || | ||
| github.event.inputs.cluster_environment == 'openshift-local' && secrets.KUBECONFIG_OPENSHIFT_LOCAL || | ||
| github.event.inputs.cluster_environment == 'gcp' && secrets.KUBECONFIG_GCP || | ||
| secrets.KUBECONFIG_LOCAL || secrets.KUBECONFIG_CONTENT | ||
| }} | ||
|
|
||
| - name: Install Helm v3.15.4 | ||
| run: | | ||
| if ! helm version 2>/dev/null | grep -q 'v3.15'; then | ||
| curl -L https://get.helm.sh/helm-v3.15.4-linux-amd64.tar.gz -o helm-v3.15.4-linux-amd64.tar.gz | ||
| tar -zxvf helm-v3.15.4-linux-amd64.tar.gz | ||
| sudo mv linux-amd64/helm /usr/local/bin/helm | ||
| rm -rf linux-amd64 helm-v3.15.4-linux-amd64.tar.gz | ||
| fi | ||
| helm version | ||
|
|
||
| - name: Install kubectl v1.31.0 | ||
| run: | | ||
| if ! kubectl version --client 2>/dev/null | grep -q 'v1.31'; then | ||
| curl -LO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl" | ||
| chmod +x kubectl | ||
| sudo mv kubectl /usr/local/bin/ | ||
| fi | ||
| kubectl version --client | ||
|
|
||
| - name: Set parameters | ||
| run: | | ||
| echo "NDCS=${{ github.event.inputs.ndcs || 2 }}" >> $GITHUB_ENV | ||
| echo "NPCS=${{ github.event.inputs.npcs || 1 }}" >> $GITHUB_ENV | ||
| echo "BS=${{ github.event.inputs.bs || 4096 }}" >> $GITHUB_ENV | ||
| echo "CHUNK_BS=${{ github.event.inputs.chunk_bs || 4096 }}" >> $GITHUB_ENV | ||
|
|
||
| - name: Verify kubectl connectivity | ||
| run: kubectl get nodes | ||
|
|
||
| # ── Cleanup old deployment ────────────────────────────────────────────── | ||
|
|
||
| - name: Cleanup old CSI deployment | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| set +e | ||
| NAMESPACE=simplyblock | ||
|
|
||
| echo "=== Phase 1: Helm uninstall ===" | ||
| helm uninstall spdk-csi -n $NAMESPACE 2>/dev/null || true | ||
|
|
||
| echo "=== Phase 2: Patch finalizers and delete CRs ===" | ||
| RESOURCES=( | ||
| "simplyblockpool.simplyblock.simplyblock.io simplyblock-pool" | ||
| "simplyblockpool.simplyblock.simplyblock.io simplyblock-pool2" | ||
| "simplyblocklvol.simplyblock.simplyblock.io simplyblock-lvol" | ||
| "simplyblocktask.simplyblock.simplyblock.io simplyblock-task" | ||
| "simplyblockdevices.simplyblock.simplyblock.io simplyblock-devices" | ||
| "simplyblockdevices.simplyblock.simplyblock.io simplyblock-device-action" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node2" | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io simplyblock-node-action" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster2" | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io simplyblock-cluster-activate" | ||
| "simplyblocksnapshotreplications.simplyblock.simplyblock.io simplyblock-snap-replication" | ||
| "simplyblocksnapshotreplications.simplyblock.simplyblock.io simplyblock-snap-replication-failback" | ||
| "pool.storage.simplyblock.io simplyblock-pool" | ||
| "pool.storage.simplyblock.io simplyblock-pool2" | ||
| "lvol.storage.simplyblock.io simplyblock-lvol" | ||
| "task.storage.simplyblock.io simplyblock-task" | ||
| "devices.storage.simplyblock.io simplyblock-devices" | ||
| "devices.storage.simplyblock.io simplyblock-device-action" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node2" | ||
| "storagenodes.storage.simplyblock.io simplyblock-node-action" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster2" | ||
| "storageclusters.storage.simplyblock.io simplyblock-cluster-activate" | ||
| "snapshotreplications.storage.simplyblock.io simplyblock-snap-replication" | ||
| "snapshotreplications.storage.simplyblock.io simplyblock-snap-replication-failback" | ||
| ) | ||
|
|
||
| patch_and_delete_crs() { | ||
| echo "Removing finalizers..." | ||
| for item in "${RESOURCES[@]}"; do | ||
| KIND=$(echo "$item" | awk '{print $1}') | ||
| NAME=$(echo "$item" | awk '{print $2}') | ||
| kubectl -n $NAMESPACE patch "$KIND" "$NAME" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "Deleting resources..." | ||
| for item in "${RESOURCES[@]}"; do | ||
| KIND=$(echo "$item" | awk '{print $1}') | ||
| NAME=$(echo "$item" | awk '{print $2}') | ||
| kubectl -n $NAMESPACE delete "$KIND" "$NAME" \ | ||
| --ignore-not-found --wait=false 2>/dev/null || true | ||
| done | ||
| } | ||
| patch_and_delete_crs | ||
|
|
||
| echo "=== Phase 3a: Delete VolumeSnapshots & VolumeSnapshotContents ===" | ||
| # Delete snapshots first (they block PVC deletion) | ||
| for VS in $(kubectl get volumesnapshot -n $NAMESPACE --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl -n $NAMESPACE patch volumesnapshot "$VS" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl -n $NAMESPACE delete volumesnapshot "$VS" --wait=false 2>/dev/null || true | ||
| done | ||
|
|
||
| # Delete VolumeSnapshotContents (cluster-scoped, may block snapshot deletion) | ||
| for VSC in $(kubectl get volumesnapshotcontent --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl patch volumesnapshotcontent "$VSC" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl delete volumesnapshotcontent "$VSC" --wait=false 2>/dev/null || true | ||
| done | ||
|
|
||
| # Delete VolumeSnapshotClasses | ||
| for VSCLASS in $(kubectl get volumesnapshotclass --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| kubectl delete volumesnapshotclass "$VSCLASS" --ignore-not-found 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3b: Delete PVCs (with timeout + retry) ===" | ||
| # First attempt: normal delete | ||
| kubectl -n $NAMESPACE delete pvc --all --wait=false 2>/dev/null || true | ||
|
|
||
| # Wait up to 60s for PVCs to go away | ||
| PVC_TIMEOUT=60 | ||
| while [ $PVC_TIMEOUT -gt 0 ]; do | ||
| REMAINING=$(kubectl -n $NAMESPACE get pvc --no-headers 2>/dev/null | wc -l) | ||
| if [ "$REMAINING" -eq 0 ]; then | ||
| echo "All PVCs deleted" | ||
| break | ||
| fi | ||
| echo "Waiting for $REMAINING PVCs to delete ($PVC_TIMEOUT s remaining)..." | ||
| sleep 5 | ||
| PVC_TIMEOUT=$((PVC_TIMEOUT - 5)) | ||
| done | ||
|
|
||
| # If PVCs are still stuck, patch finalizers and force delete | ||
| for PVC in $(kubectl -n $NAMESPACE get pvc --no-headers -o custom-columns=:metadata.name 2>/dev/null); do | ||
| echo "Force-deleting stuck PVC: $PVC" | ||
| kubectl -n $NAMESPACE patch pvc "$PVC" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl -n $NAMESPACE delete pvc "$PVC" --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3c: Delete PVs ===" | ||
| for PV in $(kubectl get pv --no-headers -o custom-columns=:metadata.name 2>/dev/null | grep -i simplyblock 2>/dev/null); do | ||
| kubectl patch pv "$PV" \ | ||
| --type=merge -p '{"metadata":{"finalizers":null}}' 2>/dev/null || true | ||
| kubectl delete pv "$PV" --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 3d: Force delete remaining namespaced resources ===" | ||
| for RTYPE in pod jobs service ds statefulset deployment replicaset secret sa configmap; do | ||
| kubectl -n $NAMESPACE delete $RTYPE --all --force --grace-period=0 2>/dev/null || true | ||
| done | ||
|
|
||
| # echo "=== Phase 4: Cleanup cluster-scoped resources ===" | ||
| # Delete StorageClasses | ||
| # for SC in $(kubectl get sc --no-headers -o custom-columns=:metadata.name 2>/dev/null | grep -i simplyblock 2>/dev/null); do | ||
| # kubectl delete sc "$SC" --ignore-not-found 2>/dev/null || true | ||
| # done | ||
| # kubectl delete clusterrole simplyblock-storage-node-role --ignore-not-found 2>/dev/null || true | ||
| # kubectl delete clusterrolebinding simplyblock-storage-node-binding --ignore-not-found 2>/dev/null || true | ||
|
|
||
| # echo "=== Phase 4b: Cleanup leftover secrets, SAs, and kube-system resources ===" | ||
| # kubectl -n $NAMESPACE delete secret simplyblock-csi-secret-v2 --ignore-not-found 2>/dev/null || true | ||
| # kubectl -n $NAMESPACE delete sa simplyblock-storage-node-sa --ignore-not-found 2>/dev/null || true | ||
|
|
||
| # # Clean up kube-system resources from previous helm installs | ||
| # for RES in sa clusterrole clusterrolebinding; do | ||
| # for NAME in $(kubectl get $RES -A --no-headers 2>/dev/null | grep -i simplyblock | awk '{print $1 ":" $2}' 2>/dev/null); do | ||
| # NS=$(echo "$NAME" | cut -d: -f1) | ||
| # RNAME=$(echo "$NAME" | cut -d: -f2) | ||
| # if [ "$RES" = "sa" ]; then | ||
| # kubectl -n "$NS" delete sa "$RNAME" --ignore-not-found 2>/dev/null || true | ||
| # else | ||
| # kubectl delete "$RES" "$RNAME" --ignore-not-found 2>/dev/null || true | ||
| # fi | ||
| # done | ||
| # done | ||
| # # Specifically clean numa-resource-plugin resources in kube-system | ||
| # kubectl -n kube-system delete ds simplyblock-numa-resource-plugin --ignore-not-found 2>/dev/null || true | ||
| # kubectl -n kube-system delete sa simplyblock-numa-resource-plugin --ignore-not-found 2>/dev/null || true | ||
| # kubectl -n kube-system delete cm simplyblock-numa-resource-plugin-config --ignore-not-found 2>/dev/null || true | ||
| # kubectl delete clusterrole simplyblock-numa-resource-plugin --ignore-not-found 2>/dev/null || true | ||
| # kubectl delete clusterrolebinding simplyblock-numa-resource-plugin --ignore-not-found 2>/dev/null || true | ||
|
|
||
| echo "=== Phase 5: Verify nothing remains ===" | ||
| echo "Namespaced resources:" | ||
| kubectl -n $NAMESPACE get all 2>/dev/null || echo "No resources found" | ||
| echo "" | ||
| echo "CRDs:" | ||
| for CR_TYPE in \ | ||
| "simplyblockpool.simplyblock.simplyblock.io" \ | ||
| "simplyblocklvol.simplyblock.simplyblock.io" \ | ||
| "simplyblockstoragenodes.simplyblock.simplyblock.io" \ | ||
| "simplyblockstorageclusters.simplyblock.simplyblock.io" \ | ||
| "pool.storage.simplyblock.io" \ | ||
| "lvol.storage.simplyblock.io" \ | ||
| "storagenodes.storage.simplyblock.io" \ | ||
| "storageclusters.storage.simplyblock.io"; do | ||
| kubectl -n $NAMESPACE get "$CR_TYPE" 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Phase 6: Delete namespace ===" | ||
| kubectl delete namespace $NAMESPACE --wait=false 2>/dev/null || true | ||
|
|
||
| for i in $(seq 1 36); do | ||
| if ! kubectl get namespace $NAMESPACE 2>/dev/null; then | ||
| echo "Namespace $NAMESPACE deleted" | ||
| break | ||
| fi | ||
|
|
||
| echo "Namespace still terminating, re-patching finalizers ($i/36)..." | ||
| patch_and_delete_crs | ||
|
|
||
| if [ "$i" -ge 6 ]; then | ||
| echo "Force-removing namespace finalizers..." | ||
| kubectl get namespace $NAMESPACE -o json 2>/dev/null | \ | ||
| jq '.spec.finalizers = []' | \ | ||
| kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f - 2>/dev/null || true | ||
| fi | ||
|
|
||
| sleep 5 | ||
| done | ||
|
|
||
| echo "=== Phase 7: Delete Released PVs from simplyblock ===" | ||
| for pv in $(kubectl get pv --no-headers 2>/dev/null | grep 'simplyblock/' | awk '{print $1}'); do | ||
| echo "Deleting PV $pv" | ||
| kubectl delete pv "$pv" --ignore-not-found 2>/dev/null || true | ||
| done | ||
|
|
||
| echo "=== Cleanup complete ===" | ||
|
|
||
|
|
||
| # ── Label nodes (after cluster is active) ─────────────────────────────── | ||
|
|
||
| - name: Label worker nodes | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| for NODE in "${NODES[@]}"; do | ||
| if [ "$CLUSTER_ENV" = "local" ] || [ "$CLUSTER_ENV" = "openshift-local" ]; then | ||
| echo "Labeling node $NODE with zone=default (local cluster)" | ||
| kubectl label node "$NODE" topology.kubernetes.io/zone=default --overwrite | ||
| else | ||
| echo "Skipping zone label for $NODE (cloud cluster: $CLUSTER_ENV)" | ||
| fi | ||
| kubectl label node "$NODE" simplyblock.io/role=mgmt-plane --overwrite | ||
| done | ||
|
|
||
| # ── Prepare namespace ───────────────────────────────────────────────── | ||
|
|
||
| - name: Create namespace and set pod-security labels | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| kubectl create namespace simplyblock --dry-run=client -o yaml | kubectl apply -f - | ||
| kubectl label namespace simplyblock \ | ||
| pod-security.kubernetes.io/enforce=privileged \ | ||
| pod-security.kubernetes.io/warn=privileged \ | ||
| pod-security.kubernetes.io/audit=privileged \ | ||
| --overwrite | ||
|
|
||
| - name: Configure OpenShift SCC policies | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| oc adm policy add-scc-to-user privileged -z default -n simplyblock | ||
| oc adm policy add-scc-to-user anyuid -z default -n simplyblock | ||
| oc label namespace simplyblock \ | ||
| pod-security.kubernetes.io/enforce=privileged \ | ||
| pod-security.kubernetes.io/audit=privileged \ | ||
| pod-security.kubernetes.io/warn=privileged \ | ||
| --overwrite | ||
|
|
||
| - name: Wait before helm install | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: sleep 30 | ||
|
|
||
| # ── Deploy CSI stack ──────────────────────────────────────────────────── | ||
|
|
||
| - name: Install Helm Chart for simplyblock-operator | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| cd $GITHUB_WORKSPACE/helm-charts/charts/simplyblock-operator/ | ||
|
|
||
| helm upgrade --install spdk-csi ./ \ | ||
| --namespace simplyblock \ | ||
| --create-namespace \ | ||
| --debug \ | ||
| --timeout 10m \ | ||
| --set controlplane.enabled=true \ | ||
| --set operator.enabled=true \ | ||
| --set controlplane.csiHostpathDriver.enabled=true \ | ||
| --set controlplane.storageclass.name=local-hostpath \ | ||
| --set image.simplyblock.repository="${{ github.event.inputs.simplyblock_repository || 'public.ecr.aws/simply-block/simplyblock' }}" \ | ||
| --set image.simplyblock.tag="${{ github.event.inputs.simplyblock_image }}" \ | ||
| --set image.operator.repository="${{ github.event.inputs.operator_repository || 'simplyblock/simplyblock-operator' }}" \ | ||
| --set image.operator.tag="${{ github.event.inputs.operator_tag || 'main' }}" \ | ||
| --set csiConfig.simplybk.ip="http://simplyblock-webappapi.simplyblock:5000" \ | ||
| --set prometheus.server.persistentVolume.storageClass=local-hostpath \ | ||
| --set controlplane.observability.enabled=true \ | ||
| --set opensearch.persistence.storageClass=local-hostpath | ||
|
|
||
| - name: Grant OpenShift SCC to all service accounts (post-helm) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| for sa in $(oc get sa -n simplyblock -o name | cut -d/ -f2); do | ||
| oc adm policy add-scc-to-user privileged -z $sa -n simplyblock | ||
| echo "Granted privileged SCC to $sa" | ||
| done | ||
|
|
||
| - name: Patch fluent-bit daemonset (post-helm) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== Waiting for simplyblock-fluent-bit daemonset (up to 5 min) ===" | ||
| PATCHED=false | ||
| for i in $(seq 1 30); do | ||
| if kubectl get daemonset simplyblock-fluent-bit -n $NAMESPACE &>/dev/null; then | ||
| echo "fluent-bit daemonset found, patching affinity..." | ||
| kubectl patch daemonset simplyblock-fluent-bit -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"affinity":null}}}}' | ||
| echo "fluent-bit daemonset patched successfully" | ||
| # Also patch CSI node daemonset with client toleration | ||
| if kubectl get daemonset simplyblock-csi-node -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-csi-node -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role","operator":"Equal","value":"client","effect":"NoSchedule"}]}}}}' | ||
| echo "csi-node daemonset patched with client toleration" | ||
| fi | ||
| PATCHED=true | ||
| break | ||
| fi | ||
| echo "fluent-bit daemonset not found yet ($i/30), waiting 10s..." | ||
| sleep 10 | ||
| done | ||
| echo "FLUENTBIT_PATCHED=$PATCHED" >> $GITHUB_ENV | ||
|
|
||
| # ── Apply custom resources ──────────────────────────────────────────── | ||
|
|
||
| - name: Wait for operator pod to be ready | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== Waiting for operator pod ===" | ||
| for i in $(seq 1 60); do | ||
| POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "$POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Operator pod $POD is Running" | ||
| break | ||
| fi | ||
| fi | ||
| echo "Waiting for operator pod ($i/60)..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| - name: Apply operator custom resources | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| MGMT_IFC="${{ github.event.inputs.mgmt_ifc || 'ens18' }}" | ||
| SB_REPO="${{ github.event.inputs.simplyblock_repository || 'public.ecr.aws/simply-block/simplyblock' }}" | ||
| SB_TAG="${{ github.event.inputs.simplyblock_image }}" | ||
| SPDK_IMAGE="${{ github.event.inputs.spdk_image }}" | ||
| DATA_NICS="${{ github.event.inputs.data_nics || 'enp1s0' }}" | ||
| PARTITIONS="${{ github.event.inputs.partitions || '0' }}" | ||
| JM_COUNT="${{ github.event.inputs.journal_manager_count || '3' }}" | ||
| OPENSHIFT_CLUSTER="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'true' || 'false' }}" | ||
| CORE_PERCENTAGE="${{ github.event.inputs.cluster_environment == 'aws-openshift' && '50' || github.event.inputs.cluster_environment == 'openshift-local' && '50' || github.event.inputs.cluster_environment == 'local' && '35' || '65' }}" | ||
| SKIP_KUBELET_CONFIG="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'false' || 'true' }}" | ||
| FORCE_FORMAT_4K="${{ (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') && 'true' || 'false' }}" | ||
|
|
||
| # Build workerNodes YAML list from comma-separated input | ||
| WORKER_YAML="" | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| for NODE in "${NODES[@]}"; do | ||
| WORKER_YAML="${WORKER_YAML} - ${NODE}"$'\n' | ||
| done | ||
|
|
||
| echo "=== Custom resource YAML to apply ===" | ||
| cat <<EOF | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageCluster | ||
| metadata: | ||
| name: simplyblock-cluster | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| mgmtIfname: ${MGMT_IFC} | ||
| fabricType: tcp | ||
| isSingleNode: false | ||
| enableNodeAffinity: true | ||
| strictNodeAntiAffinity: false | ||
| stripe: | ||
| dataChunks: ${NDCS} | ||
| parityChunks: ${NPCS} | ||
| warningThreshold: | ||
| capacity: 95 | ||
| provisionedCapacity: 97 | ||
| criticalThreshold: | ||
| capacity: 96 | ||
| provisionedCapacity: 98 | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: Pool | ||
| metadata: | ||
| name: simplyblock-pool | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| name: simplyblock-pool | ||
| clusterName: simplyblock-cluster | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageNode | ||
| metadata: | ||
| name: simplyblock-node | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| clusterImage: "${SB_REPO}:${SB_TAG}" | ||
| spdkImage: "${SPDK_IMAGE}" | ||
| mgmtIfname: ${MGMT_IFC} | ||
| dataIfname: | ||
| - ${DATA_NICS} | ||
| maxLogicalVolumeCount: 30 | ||
| partitions: ${PARTITIONS} | ||
| corePercentage: ${CORE_PERCENTAGE} | ||
| coreIsolation: false | ||
| journalManager: | ||
| count: ${JM_COUNT} | ||
| percentPerDevice: 3 | ||
| skipKubeletConfiguration: ${SKIP_KUBELET_CONFIG} | ||
| enableCpuTopology: true | ||
| openShiftCluster: ${OPENSHIFT_CLUSTER} | ||
| ubuntuHost: false | ||
| forceFormat4K: ${FORCE_FORMAT_4K} | ||
| workerNodes: | ||
| ${WORKER_YAML} | ||
| EOF | ||
|
|
||
| echo "=== Applying custom resources ===" | ||
| cat <<EOF | kubectl apply -f - | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageCluster | ||
| metadata: | ||
| name: simplyblock-cluster | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| mgmtIfname: ${MGMT_IFC} | ||
| fabricType: tcp | ||
| isSingleNode: false | ||
| enableNodeAffinity: true | ||
| strictNodeAntiAffinity: false | ||
| stripe: | ||
| dataChunks: ${NDCS} | ||
| parityChunks: ${NPCS} | ||
| warningThreshold: | ||
| capacity: 95 | ||
| provisionedCapacity: 97 | ||
| criticalThreshold: | ||
| capacity: 96 | ||
| provisionedCapacity: 98 | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: Pool | ||
| metadata: | ||
| name: simplyblock-pool | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| name: simplyblock-pool | ||
| clusterName: simplyblock-cluster | ||
| --- | ||
| apiVersion: storage.simplyblock.io/v1alpha1 | ||
| kind: StorageNode | ||
| metadata: | ||
| name: simplyblock-node | ||
| namespace: ${NAMESPACE} | ||
| spec: | ||
| clusterName: simplyblock-cluster | ||
| clusterImage: "${SB_REPO}:${SB_TAG}" | ||
| spdkImage: "${SPDK_IMAGE}" | ||
| mgmtIfname: ${MGMT_IFC} | ||
| dataIfname: | ||
| - ${DATA_NICS} | ||
| maxLogicalVolumeCount: 30 | ||
| partitions: ${PARTITIONS} | ||
| corePercentage: ${CORE_PERCENTAGE} | ||
| coreIsolation: false | ||
| journalManager: | ||
| count: ${JM_COUNT} | ||
| percentPerDevice: 3 | ||
| skipKubeletConfiguration: ${SKIP_KUBELET_CONFIG} | ||
| enableCpuTopology: true | ||
| openShiftCluster: ${OPENSHIFT_CLUSTER} | ||
| ubuntuHost: false | ||
| forceFormat4K: ${FORCE_FORMAT_4K} | ||
| workerNodes: | ||
| ${WORKER_YAML} | ||
| EOF | ||
|
|
||
| echo "Custom resources applied" | ||
| sleep 10 | ||
| env: | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
|
|
||
| - name: Grant OpenShift SCC and restart storage daemonset (post-CR) | ||
| if: ${{ github.event.inputs.use_existing_cluster != 'true' && (github.event.inputs.cluster_environment == 'aws-openshift' || github.event.inputs.cluster_environment == 'openshift-local') }} | ||
| run: | | ||
| echo "Waiting for simplyblock-storage-node-sa service account..." | ||
| for i in $(seq 1 60); do | ||
| if kubectl get sa simplyblock-storage-node-sa -n simplyblock &>/dev/null; then | ||
| echo "Service account simplyblock-storage-node-sa found" | ||
| break | ||
| fi | ||
| echo "Attempt $i/60: SA not found yet, waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "Granting privileged SCC to all service accounts..." | ||
| for sa in $(oc get sa -n simplyblock -o name | cut -d/ -f2); do | ||
| oc adm policy add-scc-to-user privileged -z $sa -n simplyblock | ||
| echo "Granted privileged SCC to $sa" | ||
| done | ||
|
|
||
| echo "Waiting for storage node daemonset to be created..." | ||
| for i in $(seq 1 60); do | ||
| if kubectl get daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock &>/dev/null; then | ||
| echo "Daemonset found, waiting for rollout to complete..." | ||
| kubectl rollout status daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock --timeout=300s | ||
| echo "Restarting daemonset..." | ||
| kubectl rollout restart daemonset simplyblock-storage-node-ds-simplyblock-cluster -n simplyblock | ||
| break | ||
| fi | ||
| echo "Attempt $i/60: daemonset not found yet, waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| # ── Wait for cluster readiness ────────────────────────────────────────── | ||
|
|
||
| - name: Wait for cluster to become active | ||
| timeout-minutes: ${{ github.event.inputs.cluster_environment == 'local' && 40 || 80 }} | ||
| id: cluster_status | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| ADMIN_POD="" | ||
| CLUSTER_ENV="${{ github.event.inputs.cluster_environment || 'local' }}" | ||
| if [ "$CLUSTER_ENV" = "local" ]; then | ||
| MAX_POLL=300 | ||
| else | ||
| MAX_POLL=720 | ||
| fi | ||
|
|
||
| echo "=== Waiting for admin pod to be ready ===" | ||
| for i in $(seq 1 60); do | ||
| ADMIN_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
|
|
||
| if [ -n "$ADMIN_POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$ADMIN_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Admin pod $ADMIN_POD is Running" | ||
| break | ||
| fi | ||
| echo "Admin pod $ADMIN_POD phase=$PHASE ($i/60)" | ||
| else | ||
| echo "Admin pod not found yet ($i/60)" | ||
| fi | ||
| sleep 10 | ||
| done | ||
|
|
||
| if [ -z "$ADMIN_POD" ]; then | ||
| echo "ERROR: Admin pod never appeared" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Expected number of snode-spdk pods = number of worker nodes | ||
| IFS=',' read -ra NODES <<< "${{ github.event.inputs.worker_nodes }}" | ||
| EXPECTED_SNODES=${#NODES[@]} | ||
| echo "=== Waiting for $EXPECTED_SNODES snode-spdk pods to be ready (MAX_POLL=$MAX_POLL) ===" | ||
|
|
||
| for i in $(seq 1 $MAX_POLL); do | ||
| TOTAL=$(kubectl -n $NAMESPACE get pods -l role=simplyblock-storage-node --no-headers 2>/dev/null | wc -l) | ||
| READY=$(kubectl -n $NAMESPACE get pods -l role=simplyblock-storage-node --no-headers 2>/dev/null | grep -c "Running" || true) | ||
|
|
||
| if [ "$READY" -ge "$EXPECTED_SNODES" ]; then | ||
| echo "All $READY/$EXPECTED_SNODES snode-spdk pods are Running" | ||
| break | ||
| fi | ||
|
|
||
| echo "snode-spdk pods: $READY/$EXPECTED_SNODES Running (total=$TOTAL) ($i/$MAX_POLL)" | ||
| if [ "$i" -eq "$MAX_POLL" ]; then | ||
| echo "ERROR: snode-spdk pods did not become ready within timeout" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "=== Re-resolving admin pod before cluster status polling ===" | ||
| ADMIN_POD="" | ||
| for i in $(seq 1 30); do | ||
| ADMIN_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "$ADMIN_POD" ]; then | ||
| PHASE=$(kubectl -n $NAMESPACE get pod "$ADMIN_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ "$PHASE" = "Running" ]; then | ||
| echo "Admin pod resolved: $ADMIN_POD (Running)" | ||
| break | ||
| fi | ||
| echo "Admin pod $ADMIN_POD phase=$PHASE, retrying ($i/30)..." | ||
| ADMIN_POD="" | ||
| else | ||
| echo "Admin pod not found, retrying ($i/30)..." | ||
| fi | ||
| sleep 10 | ||
| done | ||
| if [ -z "$ADMIN_POD" ]; then | ||
| echo "ERROR: Could not find running admin pod before polling" | ||
| kubectl -n $NAMESPACE get pods | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "=== Polling cluster status (MAX_POLL=$MAX_POLL) ===" | ||
| for i in $(seq 1 $MAX_POLL); do | ||
| # Re-resolve admin pod each iteration in case it gets recreated | ||
| NEW_POD=$(kubectl -n $NAMESPACE get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| NEW_PHASE=$(kubectl -n $NAMESPACE get pod "$NEW_POD" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| if [ -n "$NEW_POD" ] && [ "$NEW_PHASE" = "Running" ] && [ "$NEW_POD" != "$ADMIN_POD" ]; then | ||
| echo "Admin pod changed: $ADMIN_POD -> $NEW_POD" | ||
| ADMIN_POD="$NEW_POD" | ||
| fi | ||
|
|
||
| echo "[poll $i/$MAX_POLL] Using admin pod: $ADMIN_POD" | ||
| OUTPUT=$(kubectl -n $NAMESPACE exec "$ADMIN_POD" -- \ | ||
| sbctl cluster list 2>&1) || true | ||
| echo "[poll $i/$MAX_POLL] sbctl cluster list output:" | ||
| echo "$OUTPUT" | ||
|
|
||
| if echo "$OUTPUT" | grep -qi "active"; then | ||
| echo "Cluster is active!" | ||
|
|
||
| # Extract cluster ID and secret | ||
| # sbctl cluster list outputs a table; parse the data row | ||
| CLUSTER_ID=$(echo "$OUTPUT" | awk 'NR==4{print $2}') | ||
| CLUSTER_SECRET=$(echo "$OUTPUT" | awk 'NR==4{print $NF}') | ||
|
|
||
| # Fallback: try JSON format if table parsing fails | ||
| if [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" = "+" ]; then | ||
| echo "Table parsing failed (CLUSTER_ID=$CLUSTER_ID), trying JSON..." | ||
| JSON_OUT=$(kubectl -n $NAMESPACE exec "$ADMIN_POD" -- \ | ||
| sbctl cluster list --json 2>&1) || true | ||
| echo "JSON output: $JSON_OUT" | ||
| CLUSTER_ID=$(echo "$JSON_OUT" | jq -r '.[0].id // .[0].uuid // empty') | ||
| CLUSTER_SECRET=$(echo "$JSON_OUT" | jq -r '.[0].secret // empty') | ||
| fi | ||
|
|
||
| echo "Parsed CLUSTER_ID=${CLUSTER_ID}" | ||
| echo "CLUSTER_ID=${CLUSTER_ID}" >> $GITHUB_ENV | ||
| echo "CLUSTER_SECRET=${CLUSTER_SECRET}" >> $GITHUB_ENV | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo "Cluster not active yet ($i/$MAX_POLL)..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| echo "ERROR: Cluster did not become active within timeout" | ||
| kubectl -n $NAMESPACE get pods | ||
| kubectl -n $NAMESPACE exec "$ADMIN_POD" -- sbctl cluster list 2>&1 || true | ||
| exit 1 | ||
|
|
||
| - name: Patch fluent-bit daemonset (post-cluster-active) | ||
| if: ${{ env.FLUENTBIT_PATCHED != 'true' }} | ||
| run: | | ||
| NAMESPACE=simplyblock | ||
| echo "=== fluent-bit was not patched after helm install, retrying now ===" | ||
| for i in $(seq 1 30); do | ||
| if kubectl get daemonset simplyblock-fluent-bit -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-fluent-bit -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"affinity":null}}}}' | ||
| echo "fluent-bit daemonset patched successfully" | ||
| # Also patch CSI node daemonset with client toleration | ||
| if kubectl get daemonset simplyblock-csi-node -n $NAMESPACE &>/dev/null; then | ||
| kubectl patch daemonset simplyblock-csi-node -n $NAMESPACE \ | ||
| --type=merge -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role","operator":"Equal","value":"client","effect":"NoSchedule"}]}}}}' | ||
| echo "csi-node daemonset patched with client toleration" | ||
| fi | ||
| break | ||
| fi | ||
| echo "fluent-bit daemonset not found yet ($i/30), waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
|
||
| # ── Run tests ─────────────────────────────────────────────────────────── | ||
|
|
||
| - name: Set RUN_DIR_FILE | ||
| run: echo "RUN_DIR_FILE=/tmp/sb_k8s_run_dir_${GITHUB_RUN_ID}_${GITHUB_RUN_ATTEMPT}.txt" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Record Test Start Time | ||
| run: echo "TEST_START_TIME=$(date +%s)" >> $GITHUB_ENV | ||
|
|
||
| - name: Setup and Run Tests | ||
| run: | | ||
| set -o pipefail | ||
| cd $GITHUB_WORKSPACE/e2e | ||
| python3 -m venv myenv | ||
| source myenv/bin/activate | ||
| python3 -m pip install -r requirements.txt | ||
|
|
||
| export CLUSTER_ID="${{ env.CLUSTER_ID }}" | ||
| export CLUSTER_SECRET="${{ env.CLUSTER_SECRET }}" | ||
| export K8S_LOCAL_KUBECTL=1 | ||
| export SBCLI_CMD=sbctl | ||
|
|
||
| TESTNAME="" | ||
| if [ -n "${{ github.event.inputs.testname }}" ]; then | ||
| TESTNAME="--testname ${{ github.event.inputs.testname }}" | ||
| fi | ||
|
|
||
| python3 -u stress.py $TESTNAME \ | ||
| --ndcs $NDCS --npcs $NPCS --bs $BS --chunk_bs $CHUNK_BS \ | ||
| --run_k8s True 2>&1 | tee output.log | ||
| env: | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| BS: ${{ env.BS }} | ||
| CHUNK_BS: ${{ env.CHUNK_BS }} | ||
| SUPABASE_ANON_KEY: ${{ secrets.SUPABASE_ANON_KEY }} | ||
| CLIENT_IP: ${{ github.event.inputs.client_ips }} | ||
| SKIP_NFS: ${{ github.event.inputs.skip_nfs || 'false' }} | ||
| RUN_DIR_FILE: ${{ env.RUN_DIR_FILE }} | ||
|
|
||
| # ── Post-test: timing, logs, notifications ────────────────────────────── | ||
|
|
||
| - name: Export RUN_BASE_DIR from RUN_DIR_FILE | ||
| if: always() | ||
| run: | | ||
| test -f "${RUN_DIR_FILE}" || (echo "RUN_DIR_FILE not found — skipping"; exit 0) | ||
| RUN_BASE_DIR="$(cat "${RUN_DIR_FILE}" | tr -d '\r\n')" | ||
| [[ -n "${RUN_BASE_DIR}" ]] && echo "RUN_BASE_DIR=${RUN_BASE_DIR}" >> "$GITHUB_ENV" || true | ||
|
|
||
| - name: Record Test End Time | ||
| if: always() | ||
| run: echo "TEST_END_TIME=$(date +%s)" >> $GITHUB_ENV | ||
|
|
||
| - name: Calculate Total Time Taken | ||
| if: always() | ||
| run: | | ||
| TEST_TIME=$(($TEST_END_TIME - $TEST_START_TIME)) | ||
| TEST_TIME_HOURS=$(($TEST_TIME / 3600)) | ||
| TEST_TIME_MINS=$((($TEST_TIME % 3600) / 60)) | ||
| TEST_TIME_SECS=$(($TEST_TIME % 60)) | ||
| echo "Test runtime: ${TEST_TIME_HOURS}h ${TEST_TIME_MINS}m ${TEST_TIME_SECS}s" | ||
| echo "TEST_TIME_HOURS=$TEST_TIME_HOURS" >> $GITHUB_ENV | ||
| echo "TEST_TIME_MINS=$TEST_TIME_MINS" >> $GITHUB_ENV | ||
| echo "TEST_TIME_SECS=$TEST_TIME_SECS" >> $GITHUB_ENV | ||
|
|
||
| - name: Collect Graylog/OpenSearch logs | ||
| if: always() | ||
| timeout-minutes: 240 | ||
| run: | | ||
| set +e | ||
| echo "=== Collecting Graylog/OpenSearch logs (per-hour chunks) ===" | ||
| NAMESPACE=simplyblock | ||
|
|
||
| if [ -z "${TEST_START_TIME}" ] || [ -z "${TEST_END_TIME}" ]; then | ||
| echo "WARN: TEST_START_TIME or TEST_END_TIME not set, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| ELAPSED=$((TEST_END_TIME - TEST_START_TIME)) | ||
| if [ "${ELAPSED}" -le 0 ]; then | ||
| echo "WARN: Elapsed time is zero or negative, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Total window: 1h before test start → 1h after test end | ||
| WINDOW_START=$((TEST_START_TIME - 3600)) | ||
| WINDOW_END=$((TEST_END_TIME + 3600)) | ||
| TOTAL_SECONDS=$((WINDOW_END - WINDOW_START)) | ||
| TOTAL_HOURS=$(( (TOTAL_SECONDS + 3599) / 3600 )) | ||
| echo " Total window: ${TOTAL_HOURS} hour(s) to collect" | ||
|
|
||
| # Resolve admin pod | ||
| ADMIN_POD="" | ||
| for i in $(seq 1 12); do | ||
| ADMIN_POD=$(kubectl -n ${NAMESPACE} get pods \ | ||
| -l app=simplyblock-admin-control \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) || true | ||
| if [ -n "${ADMIN_POD}" ]; then | ||
| PHASE=$(kubectl -n ${NAMESPACE} get pod "${ADMIN_POD}" \ | ||
| -o jsonpath='{.status.phase}' 2>/dev/null) || true | ||
| [ "${PHASE}" = "Running" ] && break | ||
| ADMIN_POD="" | ||
| fi | ||
| sleep 10 | ||
| done | ||
| if [ -z "${ADMIN_POD}" ]; then | ||
| echo "ERROR: No running admin pod found, skipping" | ||
| exit 0 | ||
| fi | ||
| echo " Admin pod: ${ADMIN_POD}" | ||
|
|
||
| # Get Graylog service ClusterIP for mgmt-ip | ||
| MGMT_IP=$(kubectl get svc -n ${NAMESPACE} | grep graylog | awk '{print $3}') | ||
| echo " Graylog IP: ${MGMT_IP}" | ||
|
|
||
| # Determine output dir from RUN_BASE_DIR | ||
| OUTPUT_DIR="" | ||
| if [ -n "${RUN_BASE_DIR:-}" ] && [ -d "${RUN_BASE_DIR}" ]; then | ||
| OUTPUT_DIR="${RUN_BASE_DIR}/graylog_collected" | ||
| else | ||
| NFS_BASE="${NFS_MOUNTPOINT:-/mnt/nfs_share}" | ||
| TIMESTAMP=$(date -u "+%Y%m%d-%H%M%S") | ||
| OUTPUT_DIR="${NFS_BASE}/graylog_collected-${TIMESTAMP}" | ||
| fi | ||
| mkdir -p "${OUTPUT_DIR}" 2>/dev/null || true | ||
| echo " Output: ${OUTPUT_DIR}" | ||
|
|
||
| # Helper: convert epoch to ISO-8601 | ||
| epoch_to_iso() { | ||
| local ep=$1 | ||
| local iso | ||
| iso=$(date -u -d "@${ep}" "+%Y-%m-%dT%H:%M:%S" 2>/dev/null) | ||
| if [ -z "${iso}" ]; then | ||
| iso=$(python3 -c "from datetime import datetime,timezone; print(datetime.fromtimestamp(${ep},tz=timezone.utc).strftime('%Y-%m-%dT%H:%M:%S'))") | ||
| fi | ||
| echo "${iso}" | ||
| } | ||
|
|
||
| # Collect logs one hour at a time | ||
| CHUNK=0 | ||
| CURRENT=${WINDOW_START} | ||
| while [ ${CURRENT} -lt ${WINDOW_END} ]; do | ||
| CHUNK=$((CHUNK + 1)) | ||
| CHUNK_END=$((CURRENT + 3600)) | ||
| # Cap the last chunk at WINDOW_END | ||
| if [ ${CHUNK_END} -gt ${WINDOW_END} ]; then | ||
| CHUNK_END=${WINDOW_END} | ||
| fi | ||
| CHUNK_SECONDS=$((CHUNK_END - CURRENT)) | ||
| CHUNK_MINUTES=$(( (CHUNK_SECONDS + 59) / 60 )) | ||
| CHUNK_ISO=$(epoch_to_iso ${CURRENT}) | ||
|
|
||
| echo "" | ||
| echo "--- Chunk ${CHUNK}/${TOTAL_HOURS}: ${CHUNK_ISO} for ${CHUNK_MINUTES}m ---" | ||
|
|
||
| POD_OUTPUT_DIR="/tmp/graylog_collect_chunk${CHUNK}" | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- mkdir -p "${POD_OUTPUT_DIR}" 2>/dev/null || true | ||
|
|
||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| python3 /usr/local/lib/python3.12/site-packages/simplyblock_core/scripts/collect_logs.py \ | ||
| "${CHUNK_ISO}" "${CHUNK_MINUTES}" \ | ||
| --mode kubernetes \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --monitoring-secret "${MON_SECRET}" \ | ||
| --output-dir "${POD_OUTPUT_DIR}" \ | ||
| ${MGMT_IP:+--mgmt-ip "${MGMT_IP}"} \ | ||
| ${CLUSTER_ID:+--cluster-id "${CLUSTER_ID}"} \ | ||
| 2>&1 || { | ||
| echo "WARN: Graylog collect failed for chunk ${CHUNK}, retrying with --use-opensearch..." | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| python3 /usr/local/lib/python3.12/site-packages/simplyblock_core/scripts/collect_logs.py \ | ||
| "${CHUNK_ISO}" "${CHUNK_MINUTES}" \ | ||
| --mode kubernetes \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --monitoring-secret "${MON_SECRET}" \ | ||
| --output-dir "${POD_OUTPUT_DIR}" \ | ||
| --use-opensearch \ | ||
| ${MGMT_IP:+--mgmt-ip "${MGMT_IP}"} \ | ||
| ${CLUSTER_ID:+--cluster-id "${CLUSTER_ID}"} \ | ||
| 2>&1 || echo "WARN: OpenSearch fallback also failed for chunk ${CHUNK}" | ||
| } | ||
|
|
||
| # Copy tarballs from pod to NFS for this chunk | ||
| TARBALLS=$(kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- \ | ||
| find "${POD_OUTPUT_DIR}" -name "*.tar.gz" -type f 2>/dev/null) || true | ||
|
|
||
| if [ -n "${TARBALLS}" ]; then | ||
| for TB in ${TARBALLS}; do | ||
| TB_NAME=$(basename "${TB}") | ||
| echo " Copying ${TB_NAME}..." | ||
| kubectl -n ${NAMESPACE} cp "${ADMIN_POD}:${TB}" "${OUTPUT_DIR}/${TB_NAME}" 2>&1 || true | ||
| done | ||
| for TB_FILE in "${OUTPUT_DIR}"/*.tar.gz; do | ||
| [ -f "${TB_FILE}" ] && tar -xzf "${TB_FILE}" -C "${OUTPUT_DIR}/" 2>/dev/null || true | ||
| done | ||
| else | ||
| echo "WARN: No tarballs found for chunk ${CHUNK}" | ||
| fi | ||
|
|
||
| # Cleanup this chunk in pod | ||
| kubectl -n ${NAMESPACE} exec "${ADMIN_POD}" -- rm -rf "${POD_OUTPUT_DIR}" 2>/dev/null || true | ||
|
|
||
| CURRENT=${CHUNK_END} | ||
| done | ||
|
|
||
| echo "" | ||
| echo "=== Graylog collection complete (${CHUNK} chunks): ${OUTPUT_DIR} ===" | ||
| ls -la "${OUTPUT_DIR}/" 2>/dev/null || true | ||
| env: | ||
| CLUSTER_ID: ${{ env.CLUSTER_ID }} | ||
| MON_SECRET: ${{ secrets.MON_SECRET }} | ||
|
|
||
| - name: Write Job Summary | ||
| if: always() | ||
| shell: bash | ||
| run: | | ||
| set +e | ||
| out_log="$GITHUB_WORKSPACE/e2e/output.log" | ||
|
|
||
| dur_fmt="${TEST_TIME_HOURS:-0}h ${TEST_TIME_MINS:-0}m ${TEST_TIME_SECS:-0}s" | ||
|
|
||
| # --- test result counts --- | ||
| total_cases="$(grep -E 'Number of Total Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| passed_cnt="$(grep -E 'Number of Passed Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| failed_cnt="$(grep -E 'Number of Failed Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '?')" | ||
| skipped_cnt="$(grep -E 'Number of Skipped Cases:' "${out_log}" 2>/dev/null | tail -n 1 | grep -oE '[0-9]+$' || echo '0')" | ||
| test_wise="$(grep -E '(PASSED|FAILED|SKIPPED) CASE' "${out_log}" 2>/dev/null \ | ||
| | sed 's/\x1b\[[0-9;]*m//g' | sed 's/.*INFO - //' || true)" | ||
|
|
||
| # --- failure reason --- | ||
| failure_summary="(unknown)" | ||
| log_tail="" | ||
| if [[ -f "${out_log}" ]]; then | ||
| failure_summary="$(grep -oE 'MultipleExceptions: .+|SkippedTestsException: .+' "${out_log}" \ | ||
| | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| if [[ -z "${failure_summary}" ]]; then | ||
| failure_summary="$(grep -E 'RuntimeError:|AssertionError:|Error:' "${out_log}" \ | ||
| | tail -n 1 | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| fi | ||
| [[ -z "${failure_summary}" ]] && failure_summary="(no exception line found)" | ||
| log_tail="$(tail -n 50 "${out_log}" | sed 's/\x1b\[[0-9;]*m//g' || true)" | ||
| fi | ||
|
|
||
| conclusion="SUCCESS" | ||
| if [[ "${{ job.status }}" != "success" ]]; then | ||
| conclusion="FAILED" | ||
| fi | ||
|
|
||
| { | ||
| echo "## K8s Native Stress Run Summary" | ||
| echo "" | ||
| echo "**Result:** ${conclusion}" | ||
| echo "" | ||
| echo "### Run Info" | ||
| echo "- **Test class:** \`${TESTNAME}\`" | ||
| echo "- **Cluster ID:** \`${CLUSTER_ID}\`" | ||
| echo "- **Branch:** \`${{ github.ref_name }}\`" | ||
| echo "- **Helm Charts Branch:** \`${{ github.event.inputs.helm_charts_branch }}\`" | ||
| echo "- **NDCS/NPCS:** \`${NDCS}/${NPCS}\`" | ||
| echo "- **BS/Chunk BS:** \`${BS}/${CHUNK_BS}\`" | ||
| echo "- **Duration:** ${dur_fmt}" | ||
| echo "" | ||
| echo "### Test Results" | ||
| echo "- **Total:** ${total_cases} | **Passed:** ${passed_cnt} | **Failed:** ${failed_cnt} | **Skipped:** ${skipped_cnt}" | ||
| if [[ -n "${test_wise}" ]]; then | ||
| echo '```' | ||
| printf '%s\n' "${test_wise}" | ||
| echo '```' | ||
| fi | ||
| echo "" | ||
| echo "### Failure Reason" | ||
| echo '```' | ||
| printf '%s\n' "${failure_summary}" | ||
| echo '```' | ||
| if [[ -n "${log_tail}" ]]; then | ||
| echo "" | ||
| echo "<details><summary>Last 50 lines of output.log</summary>" | ||
| echo "" | ||
| echo '```' | ||
| printf '%s\n' "${log_tail}" | ||
| echo '```' | ||
| echo "</details>" | ||
| fi | ||
| echo "" | ||
| echo "### NFS Log Locations" | ||
| echo "- **RUN_BASE_DIR:** \`${RUN_BASE_DIR:-not detected}\`" | ||
| echo "- Graylog logs: \`${RUN_BASE_DIR:-<run_dir>}/graylog_collected/\`" | ||
| if [[ -n "${RUN_BASE_DIR:-}" && -d "${RUN_BASE_DIR}" ]]; then | ||
| echo "" | ||
| echo "<details><summary>NFS directory listing</summary>" | ||
| echo "" | ||
| echo '```' | ||
| ls -la "${RUN_BASE_DIR}/" 2>/dev/null || echo "(empty)" | ||
| echo '```' | ||
| echo "</details>" | ||
| fi | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| env: | ||
| TESTNAME: ${{ env.TESTNAME }} | ||
| CLUSTER_ID: ${{ env.CLUSTER_ID }} | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| BS: ${{ env.BS }} | ||
| CHUNK_BS: ${{ env.CHUNK_BS }} | ||
| RUN_BASE_DIR: ${{ env.RUN_BASE_DIR }} | ||
| TEST_TIME_HOURS: ${{ env.TEST_TIME_HOURS }} | ||
| TEST_TIME_MINS: ${{ env.TEST_TIME_MINS }} | ||
| TEST_TIME_SECS: ${{ env.TEST_TIME_SECS }} | ||
|
|
||
| - name: Send Slack Notification | ||
| if: always() && (github.event.inputs.send_slack_notification || 'true') == 'true' | ||
| shell: bash | ||
| env: | ||
| SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} | ||
| JOB_STATUS: ${{ job.status }} | ||
| SLACK_RUN_URL: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | ||
| GITHUB_REF_NAME: ${{ github.ref_name }} | ||
| SLACK_WF_NAME: "K8s Native Stress" | ||
| TESTNAME: ${{ env.TESTNAME }} | ||
| RUN_BASE_DIR: ${{ env.RUN_BASE_DIR }} | ||
| NDCS: ${{ env.NDCS }} | ||
| NPCS: ${{ env.NPCS }} | ||
| TEST_START_TIME: ${{ env.TEST_START_TIME }} | ||
| TEST_END_TIME: ${{ env.TEST_END_TIME }} | ||
| run: | | ||
| python3 - <<'PYEOF' | ||
| import json, os, re, sys, urllib.request, urllib.error | ||
| webhook = os.environ.get("SLACK_WEBHOOK_URL", "") | ||
| if not webhook: | ||
| print("No SLACK_WEBHOOK_URL set, skipping.") | ||
| sys.exit(0) | ||
| out_log = os.path.join(os.environ.get("GITHUB_WORKSPACE", "."), "e2e", "output.log") | ||
| total = passed = failed = skipped = 0 | ||
| if os.path.isfile(out_log): | ||
| content = open(out_log).read() | ||
| def px(pat): | ||
| m = re.search(pat, content) | ||
| return int(m.group(1)) if m else 0 | ||
| total = px(r'Number of Total Cases:\s*(\d+)') | ||
| passed = px(r'Number of Passed Cases:\s*(\d+)') | ||
| failed = px(r'Number of Failed Cases:\s*(\d+)') | ||
| skipped = px(r'Number of Skipped Cases:\s*(\d+)') | ||
| pass_pct = (passed * 100 // total) if total > 0 else 0 | ||
| s = int(os.environ.get("TEST_START_TIME", "0") or "0") | ||
| e = int(os.environ.get("TEST_END_TIME", "0") or "0") | ||
| secs = max(0, e - s) if e >= s > 0 else 0 | ||
| dur = f"{secs//3600}h {(secs%3600)//60}m {secs%60}s" | ||
| run_url = os.environ.get("SLACK_RUN_URL", "") | ||
| log_dir = os.environ.get("RUN_BASE_DIR", "N/A") | ||
| ndcs = os.environ.get("NDCS", "?") | ||
| npcs = os.environ.get("NPCS", "?") | ||
| test_cls = os.environ.get("TESTNAME", "") or "all" | ||
| branch = os.environ.get("GITHUB_REF_NAME", "?") | ||
| wf_name = os.environ.get("SLACK_WF_NAME", "Run") | ||
| ok = os.environ.get("JOB_STATUS", "") == "success" | ||
| icon = ":white_check_mark:" if ok else ":x:" | ||
| status = "SUCCESS" if ok else "FAILURE" | ||
| mention = "" if ok else " <!channel>" | ||
| lines = [ | ||
| f"{icon} *SimplyBlock {wf_name}*{mention}", | ||
| f"*Status:* {status} | *Duration:* {dur}", | ||
| f"*Branch:* `{branch}` | *NDCS/NPCS:* `{ndcs}/{npcs}` | *Test class:* `{test_cls}`", | ||
| "", | ||
| ] | ||
| if total > 0: | ||
| lines += [ | ||
| f":white_check_mark: *Passed:* {passed}/{total} ({pass_pct}%)", | ||
| f":x: *Failed:* {failed}", | ||
| f":fast_forward: *Skipped:* {skipped}", | ||
| ] | ||
| else: | ||
| lines.append("_(test counts not found in log)_") | ||
| lines += [ | ||
| "", | ||
| f":link: *Run:* <{run_url}|View on GitHub>", | ||
| f":file_folder: *Final Logs:* `{log_dir}`", | ||
| ] | ||
| payload = {"text": "\n".join(lines)} | ||
| req = urllib.request.Request( | ||
| webhook, | ||
| data=json.dumps(payload).encode(), | ||
| headers={"Content-Type": "application/json"}, | ||
| ) | ||
| try: | ||
| urllib.request.urlopen(req, timeout=15) | ||
| print("Slack notification sent.") | ||
| except Exception as exc: | ||
| print(f"WARN: Slack notification failed: {exc}", file=sys.stderr) | ||
| PYEOF | ||
|
|
||
| - name: Cleanup build folder | ||
| if: always() | ||
| run: | | ||
| rm -rf ./* || true | ||
| rm -rf ./.??* || true | ||
| rm -rf $HOME/.kube |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| scroll_id = data.get("_scroll_id") | ||
| hits = data.get("hits", {}).get("hits", []) | ||
| total = data.get("hits", {}).get("total", {}) | ||
| total = ( |
| scroll_id = data.get("_scroll_id") | ||
| hits = data.get("hits", {}).get("hits", []) | ||
| total = data.get("hits", {}).get("total", {}) | ||
| total = total.get("value", total) if isinstance(total, dict) else int(total or 0) |
| json={"scroll_id": scroll_id}, | ||
| timeout=10, | ||
| ) | ||
| except Exception: |
| if r.status_code == 200: | ||
| graylog_ok = True | ||
| self.logger.info("[graylog-export] Graylog is reachable") | ||
| except Exception: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.