posts/k3s-etcd-ansible/

[giscus[bot]]

posts/k3s-etcd-ansible/

Documentation for Techno Tim YouTube videos for all your copy pasta needs!

https://technotim.live/posts/k3s-etcd-ansible/

[richardhughes260]

Hi Tim, big thanks for the videos and documentation! I have a number of RADXA 5B SBC's that I want to run a docker cluster on. I'm sure I'll need some help on my way!

[properlyabsurd]

Hi Tim, what's the best way to add more master nodes to the cluster? I started with three but want to add two more. Thanks for the video.

[bugaian]

Hi Tim, I tried to deploy your adapted 3ks cluster on small lab on a laptop 32Gb. 2Gb RAM per Vm and 60GB HDD deployed with multipass from canonical. The issue I am facing is this one after deploying 3 masters ansible playbook get stuck at nodes and not moving over...I tried it multiple times with no success. And its not a limitation of resources I think since if I try to deploy 5 masters it works fine... Will get back with more findings...
"ghcr.io/kube-vip/kube-vip:v0.5.12": failed to resolve reference "ghcr.io/kube-vip/kube-vip:v0.5.12": failed to do request: Head "https://ghcr.io/v2/kube-vip/kube-vip/manifests/v0.5.12": x509: certificate signed by unknown authority
Warning Failed 9m56s (x2 over 10m) kubelet Error: ErrImagePull
Normal BackOff 9m45s (x2 over 10m) kubelet Back-off pulling image "ghcr.io/kube-vip/kube-vip:v0.5.12"
Warning Failed 9m45s (x2 over 10m) kubelet Error: ImagePullBackOff
Normal SandboxChanged 8m57s kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulling 7m25s (x4 over 8m57s) kubelet Pulling image "ghcr.io/kube-vip/kube-vip:v0.5.12"
Warning Failed 7m24s (x4 over 8m56s) kubelet Failed to pull image "ghcr.io/kube-vip/kube-vip:v0.5.12": rpc error: code = Unknown desc = failed to pull and unpack image "ghcr.io/kube-vip/kube-vip:v0.5.12": failed to resolve reference "ghcr.io/kube-vip/kube-vip:v0.5.12": failed to do request: Head "https://ghcr.io/v2/kube-vip/kube-vip/manifests/v0.5.12": x509: certificate signed by unknown authority

Looks like it can not pull vip images...still investigating will et back to you with findings.

[bugaian]

OK I managed to fix it by pooling/pushing images to doker hub then changed image names and yaml url in ansible tasks.

[itsupport-s2i]

idk why, but:

apiserver_endpoint is virtual ip-address which will be configured on each master

apiserver_endpoint: "192.168.xxx.xxx"

i have it only in one master instead of each master.
have zero errors when i run ansible playbook and it's strange, have no idea where to investigate

[itsupport-s2i]

ok, i found, that all clusters need to be in same network.
when i put machines from another VLAN i have problems.

[kubert96]

TASK [k3s_server : Init cluster inside the transient k3s-init service] *********************************************************************************************
fatal: [172.16.30.30]: FAILED! => {"msg": "An unhandled exception occurred while templating '{% if groups[group_name_master | default('master')] | length > 1 %}\n {% if ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname'] %}\n --cluster-init\n {% else %}\n --server https://{{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip | split(",") | first | ansible.utils.ipwrap }}:6443\n {% endif %}\n --token {{ k3s_token }}\n{% endif %} {{ extra_server_args | default('') }}'. Error was a <class 'jinja2.exceptions.TemplateRuntimeError'>, original message: No filter named 'split' found."}

Geting this error every type on clean machines... :(

[kubert96]

After that :

TASK [k3s_server : Verify that all nodes actually joined (check k3s-init.service if this fails)] *******************************************************************
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (20 retries left).
FAILED - RETRYING: Verify that all nodes actually joined (check k3s-init.service if this fails) (19 retries left).

Bot when i go to one of the masters then i see 1 Master node but no other joined...

[kubert96]

Same problem every time :( So frustrated :(

fatal: [172.16.30.20]: FAILED! => {"msg": "An unhandled exception occurred while templating '{% if groups[group_name_master | default('master')] | length > 1 %}\n {% if ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname'] %}\n --cluster-init\n {% else %}\n --server https://{{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip | split(",") | first | ansible.utils.ipwrap }}:6443\n {% endif %}\n --token {{ k3s_token }}\n{% endif %} {{ extra_server_args | default('') }}'. Error was a <class 'jinja2.exceptions.TemplateRuntimeError'>, original message: No filter named 'split' found."}
fatal: [172.16.30.30]: FAILED! => {"msg": "An unhandled exception occurred while templating '{% if groups[group_name_master | default('master')] | length > 1 %}\n {% if ansible_hostname == hostvars[groups[group_name_master | default('master')][0]]['ansible_hostname'] %}\n --cluster-init\n {% else %}\n --server https://{{ hostvars[groups[group_name_master | default('master')][0]].k3s_node_ip | split(",") | first | ansible.utils.ipwrap }}:6443\n {% endif %}\n --token {{ k3s_token }}\n{% endif %} {{ extra_server_args | default('') }}'. Error was a <class 'jinja2.exceptions.TemplateRuntimeError'>, original message: No filter named 'split' found."}
changed: [172.16.30.10]

Even when i add another Master node it will get stuck...

[kubert96]

One wai to get around is to only have One Master node...
rmihalik@k3s-ctrl-pve-01:~$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3s-ctrl-pve-01 Ready control-plane,etcd,master 4m41s v1.25.12+k3s1
k3s-node-pve-01 Ready 48s v1.25.12+k3s1
k3s-node-pve-02 Ready 48s v1.25.12+k3s1
k3s-node-pve2-01 Ready 47s v1.25.12+k3s1
k3s-node-pve2-02 Ready 47s v1.25.12+k3s1
k3s-node-pve3-01 Ready 48s v1.25.12+k3s1
k3s-node-pve3-02 Ready 27s v1.25.12+k3s1

But this is no HA...

I am going to look at Ansible playbook to get to join another Control Plane's

[zeemlinux]

apt-get install -y python3-netaddr <<<< That worked for me after a series of tries.

[kubert96]

OK so multiple problems...

I dont know what is wrong at my first coment.

But second coment had Python3 something Netadd -

apt install --no-install-recommends python-netaddr -- This fixed everithing... Newer mentioned in tutorial of previsions ansible or this...

And Updated ansible to latest version...

[timothystewart6]

Sorry, it is mention in the github repo's docs. If you would like to open a PR to update this blog post, please do!

https://github.com/techno-tim/k3s-ansible#-system-requirements

[chris-deluca]

I was able to following the process and things seemed to work very well. I'm discovering that I don't seem to get consistent DNS resolution between pods in the same namespace. Has anyone else seen this or maybe have a resolution? Thanks in advance!

[B1GWH1T3]

Thanks for the guide !

getting the below error?

fatal: [k3smaster.home]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'k3s_version' is undefined. 'k3s_version' is undefined\n\nThe error appears to be in '/root/playbooks/k3s-ansible/roles/download/tasks/main.yml': line 3, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Download k3s binary x64\n ^ here\n"}

[timothystewart6]

No problem! It looks like you k3s version isn't defined in your all.yaml

what does it have for a version? The default is:

k3s_version: v1.25.12+k3s1

[B1GWH1T3]

use the latest version, but I noted the main.yml repo being referred does not exist if you open /downloads it gives you 404 error

[B1GWH1T3]

okay so If I change the k3s version I get an error, but just went with the version that you specified and it works 100%

[ghost]

How to use this role to install a standalone k3s ? Just install server?

[ghost]

I have a single VM and want to install both agent and server, what inventory structure should I use?

[AlphaInfamous]

# this is the user that has ssh access to these machines
ansible_user: <myuser>
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "America/Detroit"

# interface which will be used for flannel
# flannel_iface: "eth0"

# uncomment calico_iface to use tigera operator/calico cni instead of flannel https://docs.tigera.io/calico/latest/about
# calico_iface: "eth0"
calico_ebpf: false # use eBPF dataplane instead of iptables
calico_tag: "v3.27.2" # calico version tag

# uncomment cilium_iface to use cilium cni instead of flannel or calico
# ensure v4.19.57, v5.1.16, v5.2.0 or more recent kernel
cilium_iface: "eth0"
cilium_mode: "native" # native when nodes on same subnet or using bgp, else set routed
cilium_tag: "v1.15.4" # cilium version tag
# cilium_hubble: true # enable hubble observability relay and ui

# if using calico or cilium, you may specify the cluster pod cidr pool
cluster_cidr: "20.76.0.0/16"

# enable cilium bgp control plane for lb services and pod cidrs. disables metallb.
cilium_bgp: true

# bgp parameters for cilium cni. only active when cilium_iface is defined and cilium_bgp is true.
cilium_bgp_my_asn: "64513"
cilium_bgp_peer_asn: "64512"
cilium_bgp_peer_address: "20.76.6.101"
cilium_bgp_lb_cidr: "20.76.6.1/24" # cidr for cilium loadbalancer ipam

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "20.76.6.80"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "<token>"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: "{{ ansible_facts[(cilium_iface | default(calico_iface | default(flannel_iface)))]['ipv4']['address'] }}"

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  {{ '--flannel-iface=' + flannel_iface if calico_iface is not defined and cilium_iface is not defined else '' }}
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
# the contents of the if block is also required if using calico or cilium
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  {% if calico_iface is defined or cilium_iface is defined %}
  --flannel-backend=none
  --disable-network-policy
  --cluster-cidr={{ cluster_cidr | default('10.52.0.0/16') }}
  {% endif %}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik

extra_agent_args: >-
  {{ extra_args }}

# image tag for kube-vip
kube_vip_tag_version: "v0.7.2"

# tag for kube-vip-cloud-provider manifest
# kube_vip_cloud_provider_tag_version: "main"

# kube-vip ip range for load balancer
# (uncomment to use kube-vip for services instead of MetalLB)
# kube_vip_lb_ip_range: "20.76.6.80-20.76.6.90"

# metallb type frr or native
metal_lb_type: "native"

# metallb mode layer2 or bgp
metal_lb_mode: "layer2"

# bgp options
# metal_lb_bgp_my_asn: "64513"
# metal_lb_bgp_peer_asn: "64512"
# metal_lb_bgp_peer_address: "20.76.6.81"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.14.3"
metal_lb_controller_tag_version: "v0.14.3"

# metallb ip range for load balancer
# metal_lb_ip_range: "20.76.6.79-20.76.6.90"

# Only enable if your nodes are proxmox LXC nodes, make sure to configure your proxmox nodes
# in your hosts.ini file.
# Please read https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 before using this.
# Most notably, your containers must be privileged, and must not have nesting set to true.
# Please note this script disables most of the security of lxc containers, with the trade off being that lxc
# containers are significantly more resource efficient compared to full VMs.
# Mixing and matching VMs and lxc containers is not supported, ymmv if you want to do this.
# I would only really recommend using this if you have particularly low powered proxmox nodes where the overhead of
# VMs would use a significant portion of your available resources.
proxmox_lxc_configure: false
# the user that you would use to ssh into the host, for example if you run ssh some-user@my-proxmox-host,
# set this value to some-user
proxmox_lxc_ssh_user: root
# the unique proxmox ids for all of the containers in the cluster, both worker and master nodes
proxmox_lxc_ct_ids:
  - 200
  - 201
  - 202
  - 203
  - 204

# Only enable this if you have set up your own container registry to act as a mirror / pull-through cache
# (harbor / nexus / docker's official registry / etc).
# Can be beneficial for larger dev/test environments (for example if you're getting rate limited by docker hub),
# or air-gapped environments where your nodes don't have internet access after the initial setup
# (which is still needed for downloading the k3s binary and such).
# k3s's documentation about private registries here: https://docs.k3s.io/installation/private-registry
custom_registries: false
# The registries can be authenticated or anonymous, depending on your registry server configuration.
# If they allow anonymous access, simply remove the following bit from custom_registries_yaml
#   configs:
#     "registry.domain.com":
#       auth:
#         username: yourusername
#         password: yourpassword
# The following is an example that pulls all images used in this playbook through your private registries.
# It also allows you to pull your own images from your private registry, without having to use imagePullSecrets
# in your deployments.
# If all you need is your own images and you don't care about caching the docker/quay/ghcr.io images,
# you can just remove those from the mirrors: section.
custom_registries_yaml: |
  mirrors:
    docker.io:
      endpoint:
        - "https://registry.domain.com/v2/dockerhub"
    quay.io:
      endpoint:
        - "https://registry.domain.com/v2/quayio"
    ghcr.io:
      endpoint:
        - "https://registry.domain.com/v2/ghcrio"
    registry.domain.com:
      endpoint:
        - "https://registry.domain.com"

  configs:
    "registry.domain.com":
      auth:
        username: yourusername
        password: yourpassword

# On some distros like Diet Pi, there is no dbus installed. dbus required by the default reboot command.
# Uncomment if you need a custom reboot command
# custom_reboot_command: /usr/sbin/shutdown -r now

# Only enable and configure these if you access the internet through a proxy
# proxy_env:
#   HTTP_PROXY: "http://proxy.domain.local:3128"
#   HTTPS_PROXY: "http://proxy.domain.local:3128"
#   NO_PROXY: "*.domain.local,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

fatal: [20.76.6.217]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.129938", "end": "2024-05-16 19:13:18.436759", "msg": "non-zero return code", "rc": 1, "start": "2024-05-16 19:13:18.306821", "stderr": "E0516 19:13:18.427720    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.428171    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.429953    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.430482    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.432088    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nThe connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["E0516 19:13:18.427720    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.428171    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.429953    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.430482    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.432088    6163 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
fatal: [20.76.6.71]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.131616", "end": "2024-05-16 19:13:18.545570", "msg": "non-zero return code", "rc": 1, "start": "2024-05-16 19:13:18.413954", "stderr": "E0516 19:13:18.536880    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.537238    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.538793    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.539321    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nE0516 19:13:18.540921    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused\nThe connection to the server localhost:8080 was refused - did you specify the right host or port?", "stderr_lines": ["E0516 19:13:18.536880    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.537238    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.538793    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.539321    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "E0516 19:13:18.540921    6154 memcache.go:265] couldn't get current server API group list: Get \"http://localhost:8080/api?timeout=32s\": dial tcp 127.0.0.1:8080: connect: connection refused", "The connection to the server localhost:8080 was refused - did you specify the right host or port?"], "stdout": "", "stdout_lines": []}
fatal: [20.76.6.202]: FAILED! => {"attempts": 20, "changed": false, "cmd": ["k3s", "kubectl", "get", "nodes", "-l", "node-role.kubernetes.io/master=true", "-o=jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.140841", "end": "2024-05-16 19:13:19.292818", "msg": "", "rc": 0, "start": "2024-05-16 19:13:19.151977", "stderr": "", "stderr_lines": [], "stdout": "nuc1", "stdout_lines": ["nuc1"]}

what am I doing wrong??

[DaKaHa0]

Thanks Tim for your contribution.
I am using default variable, getting below error
I notice you have specified "v1.30.2+k3s2", I even tried changed to "v1.29.7+k3s1" still no luck. What am I doing wrong ?

FAILED! => {"argument_errors": ["missing required arguments: k3s_version"], "argument_spec_data": {"k3s_version": {"description": "The desired version of K3S", "required": true}}, "changed": false, "msg": "Validation of arguments failed:\nmissing required arguments: k3s_version", "validate_args_context": {"argument_spec_name": "main", "name": "download", "path"

[codedmayor]

I am getting the same error. Have you found a solution for it yet?

[codedmayor]

Never mind, found the solution. I was using an old ansible version. Minimum ansible version for it to work is 2.11