Skip to content

For building a multi-nodes k8s cluster environment with infra services.

License

Notifications You must be signed in to change notification settings

jingtaozhang18/k8s-playground

Repository files navigation

k8s-playground

For building a multi-nodes k8s cluster environment with infra services.

Table Contents

Architecture

The overall structure is shown in the figure below.

Prepare the Host Environment

Install Ubuntu

Install Ubuntu22 on HP Z440 Workstation and install common command.

Install common commands through apt.

sudo apt update
sudo apt install -y \
  htop iftop \
  vim \
  curl wget \
  make

Refer:

Install KVM and Docker

Install KVM through apt command.

sudo apt -y install \
  bridge-utils \
  cpu-checker \
  libvirt-clients \
  libvirt-daemon \
  qemu \
  qemu-kvm

Install Docker through apt command. It is recommended to install according to the latest official website introduction.

sudo apt-get update
sudo apt-get -y install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get -y install \
  docker-ce docker-ce-cli \
  containerd.io \
  docker-compose-plugin

After installing docker, there is a quick way to allow a non-root user to use docker.

chown ${USER} /var/run/docker.sock

Refer:

Install Minikube and Helm

This repo uses a special version of minikube and helm.

Install Minikube:

curl -LO https://storage.googleapis.com/minikube/releases/v1.26.1/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
rm minikube-linux-amd64

Install Helm:

curl -fsSL -o ./get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 ./get_helm.sh
./get_helm.sh
rm ./get_helm.sh

Refer:

Config Network Environment

Config network environment, includes creating virtual bridge and kvm virtual network. They will be bridge br0 as shown in the architecture diagram.

Create virtual bridge by netplan command. Pls check network config file first, and the default config will use a static ip which may not be right for you.

mv /etc/netplan/01-network-manager-all.yaml /etc/netplan/01-network-manager-all.yaml.backup
cp configs/network/01-network-manager-all.yaml /etc/netplan/01-network-manager-all.yaml
netplan apply

Create KVM virtual network.

virsh net-define configs/network/kvm-bridged-network.xml
virsh net-start bridged-network
virsh net-autostart bridged-network
virsh net-list

Someday, the network is down in virtual machine, and the root cause is still unknown.

You can also choose to use NAT mode.

You cannot access specific network through changing route ip in nat mode

virsh net-define configs/network/kvm-nat-network.xml
virsh net-start nat-network
virsh net-autostart nat-network
virsh net-list

Refer:

Create K8S Cluster

Create Infra Standalone Services

It is need to create some infra to support k8s running well because this is a multi-nodes cluster.

Create NFS Server

Create NFS server through bash script, before running it, pls check NFS_DOMAIN variable which indicates the subnet that can access the NFS service.

Start K8S Cluster

Now, It is all ready for starting k8s cluster! Start it through below command. For setting route for k8s cluster, run scripts/k8s_set_route.sh script which will set route when node is ready.

PROFILE_NAME='playground'
SOFT_ROUTE_IP='192.168.1.41'
KVM_NETWORK='nat-network'
NODE_NUM=3
# bash ${WORKING_DIR}/scripts/k8s_set_route.sh ${PROFILE_NAME} ${NODE_NUM} ${SOFT_ROUTE_IP} &
# export HTTP_PROXY=http://${SOFT_ROUTE_IP}:1080
# export HTTPS_PROXY=https://${SOFT_ROUTE_IP}:1080
# export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.49.0/24,192.168.39.0/24
minikube config set WantUpdateNotification false
minikube \
  --profile ${PROFILE_NAME} \
  --driver=kvm2 \
  --install-addons=false \
  --kubernetes-version='v1.24.3' \
  --auto-update-drivers=false \
  --nodes=${NODE_NUM} \
  --cpus=8 \
  --memory=18g \
  --disk-size=40g \
  --kvm-network="${KVM_NETWORK}" \
  --image-mirror-country='cn' \
  --image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers' \
  start

To get nodes:

PROFILE_NAME="playground"
minikube kubectl --profile ${PROFILE_NAME} -- get pods -A

To access dashboard:

PROFILE_NAME="playground"
minikube dashboard --profile ${PROFILE_NAME} --url

Refer to k8s_start.sh.

Install Infra Service in K8S

Refer to k8s_infra_services_enable.sh for all deploy code.

Install standard Storage Class

Because this is a multi-nodes k8s, so the default storage class which using a certain host path can't satisfy the need. And the standard storage class can mount a nfs path which can be accessed by any node.

Deploy it using below command.

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
helm repo update
PROFILE_NAME="playground"
CONTEXT_NAME=${PROFILE_NAME}
NFS_STORAGE_NAMESPACE="storage-nfs"
IMAGE_MIRROR_SUFFIX=".registry.jingtao.fun"
# IMAGE_MIRROR_SUFFIX=""  # Leave blank to not apply mirror service
# get host ip
BR0_IP=$(ip addr show br0 | grep inet | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | tr -d "addr:")
BR0_IP=$(echo ${BR0_IP//\// } | awk '{print $1}')
echo "your host ip: ${BR0_IP}"
minikube kubectl --profile ${PROFILE_NAME} -- create namespace ${NFS_STORAGE_NAMESPACE} --dry-run=client -o yaml | minikube kubectl --profile ${PROFILE_NAME} -- apply -f -
helm upgrade --install nfs-subdir-external-provisioner \
  --kube-context ${CONTEXT_NAME} \
  --namespace ${NFS_STORAGE_NAMESPACE} \
  --values configs/charts_values/nfs-values.yaml \
  --set nfs.server=${BR0_IP} \
  --set image.repository="k8s.gcr.io${IMAGE_MIRROR_SUFFIX}/sig-storage/nfs-subdir-external-provisioner" \
  --wait \
  --timeout 10m0s \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

Docs

Refer:

About

For building a multi-nodes k8s cluster environment with infra services.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published