How to enable edgevpn and get a cluster with one k3 server and one agent?

[Ognian]

On one nodethe config has:

kairos:
  network_token: "..."
  role: "master"
k3s:
  enabled: true

on the other:

kairos:
  network_token: "..."    

k3s-agent:
  enabled: true

but edgevpn service is not started on any of the nodes.
What am I doing wrong?

[mudler]

Hi @Ognian take a look at : #531 (comment)

Good point in any case - our docs should have at least examples that are ready to copy-paste

[mudler]

I've added a full sample here: https://kairos.io/docs/installation/p2p/#configuration

[Ognian]

@mudler Hmm, as far as I understand from the documentation if there is a 'network_token' the p2p service should start( I assume that this service is called 'edgevpn' and there is a corresponding process named edgevpn), but there is no such process found in htop on my node(s). Neither on the x86-64 node:

cat /etc/os-release 
NAME="kairos-alpine-ubuntu"
VERSION="v1.3.0-k3sv1.25.4+k3s1"
ID="kairos"
ID_LIKE="kairos-alpine-ubuntu"
VERSION_ID="v1.3.0-k3sv1.25.4+k3s1"
PRETTY_NAME="kairos-alpine-ubuntu v1.3.0-k3sv1.25.4+k3s1"
ANSI_COLOR="0;32"
BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues/new/choose"
HOME_URL="https://github.com/kairos-io/provider-kairos"
IMAGE_REPO="quay.io/kairos/kairos-alpine-ubuntu"
IMAGE_LABEL="latest"
GITHUB_REPO="kairos-io/provider-kairos"
VARIANT="core"
FLAVOR="opensuse"

nor on the rpi node:

cat /etc/os-release 
NAME="kairos-opensuse-arm-rpi"
VERSION="v1.3.0-k3sv1.25.4+k3s1"
ID="kairos"
ID_LIKE="kairos-opensuse-arm-rpi"
VERSION_ID="v1.3.0-k3sv1.25.4+k3s1"
PRETTY_NAME="kairos-opensuse-arm-rpi v1.3.0-k3sv1.25.4+k3s1"
ANSI_COLOR="0;32"
BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues/new/choose"
HOME_URL="https://github.com/kairos-io/provider-kairos"
IMAGE_REPO="quay.io/kairos/kairos-opensuse-arm-rpi"
IMAGE_LABEL="latest"
GITHUB_REPO="kairos-io/provider-kairos"
VARIANT="core"
FLAVOR="opensuse"

Not sure what I'm missing...

[mudler]

can you check if you have a edgevpn@kairos systemd service? we run automated tests against it and it works fine here as well, can you show your config?

[Ognian]

on the rpi sudo systemctl list-unit-files --type=service does not show show a service with a name starting with 'edge'...
on the x86-64 the systemctl executable even does not exist (maybe this is a sign?)

rpi has a version build by building with build-arm-image and putting it on the rpi disk
x86-64 is booting from a local disk, installed from the usb drive booting the iso. When the iso image is booting I'm getting the QR code, so I assume that edgevpn was started, but when installing from the iso to the local mmc (local disk) and booting from there, the QR code doesn't come and edgevpn is non existent. (could this be the cause?)

rpi config: cat /usr/local/cloud-config/99_custom.yaml

# config options: https://kairos.io/reference/configuration
# and https://rancher.github.io/elemental-toolkit/docs/reference/cloud_init/


stages:
  initramfs:
    - name: "Setup hostname"
      hostname: "rpi4node"
    - name: "Setup users"
      # disable kairos password login is done via sshd
      authorized_keys:
        kairos:
          - "ssh-rsa xxx"
    - name: "Adjust sshd"
      commands:
        # for sed see https://www.gnu.org/software/sed/manual/sed.html 
        # enable sftp login via internal module since external one is depricated
        - sed -i -e  '/^\(Subsystem\s*sftp\).*/{s//\1 internal-sftp/;:a;n;ba;q}' -e '$aSubsystem sftp internal-sftp' /etc/ssh/sshd_config 
        # via ssh no passwords, only keys, no root login (only sudo) via ssh
        - sed -i -e  '/^\(PasswordAuthentication\).*/{s//\1 no/;:a;n;ba;q}' -e '$aPasswordAuthentication no' /etc/ssh/sshd_config  
        - sed -i -e  '/^\(KbdInteractiveAuthentication\).*/{s//\1 no/;:a;n;ba;q}' -e '$aKbdInteractiveAuthentication no' /etc/ssh/sshd_config  
        - sed -i -e  '/^\(PermitRootLogin\).*/{s//\1 no/;:a;n;ba;q}' -e '$aPermitRootLogin no' /etc/ssh/sshd_config  
    - name: "Set DNS servers" # fix for https://github.com/kairos-io/kairos/issues/335
      dns:
        nameservers:
        - 1.1.1.1
        - 8.8.8.8   
    - name: "load kernel driver for oled display"
      commands:
        - modprobe i2c-dev
   
kairos:
  network_token: yyy

k3s-agent:
  enabled: true
  args:
    # this node has an oled display
    - --node-label "ogi-it.com/oled-display=true"

x86-64: cat /oem/90_custom.yaml

# config options: https://kairos.io/reference/configuration
# and https://rancher.github.io/elemental-toolkit/docs/reference/cloud_init/


stages:
  initramfs:
    - name: "Setup hostname"
      hostname: "lp3dnode"
    - name: "Setup users"
      # looks like user MUST be defined!!
      users:
        kairos:
          # passwd: "!" lock user
          passwd: "kairos" #TEMP to get in

      # disable kairos password login is done via sshd
      authorized_keys:
        kairos:
          - "ssh-rsa xxx"
    - name: "Adjust sshd"
      commands:
        # for sed see https://www.gnu.org/software/sed/manual/sed.html 
        # enable sftp login via internal module since external one is depricated
        - sed -i -e  '/^\(Subsystem\s*sftp\).*/{s//\1 internal-sftp/;:a;n;ba;q}' -e '$aSubsystem sftp internal-sftp' /etc/ssh/sshd_config 
        # via ssh no passwords, only keys, no root login (only sudo) via ssh
        - sed -i -e  '/^\(PasswordAuthentication\).*/{s//\1 no/;:a;n;ba;q}' -e '$aPasswordAuthentication no' /etc/ssh/sshd_config  
        - sed -i -e  '/^\(KbdInteractiveAuthentication\).*/{s//\1 no/;:a;n;ba;q}' -e '$aKbdInteractiveAuthentication no' /etc/ssh/sshd_config  
        - sed -i -e  '/^\(PermitRootLogin\).*/{s//\1 no/;:a;n;ba;q}' -e '$aPermitRootLogin no' /etc/ssh/sshd_config  
    - name: "Set DNS servers" # fix for https://github.com/kairos-io/kairos/issues/335
      dns:
        nameservers:
        - 1.1.1.1
        - 8.8.8.8   
    

kairos:
  network_token: "yyy"
  role: "master"
k3s:
  enabled: true
  args:
    # this node has an oled display
    # - --node-label "ogi-it.com/oled-display=true"

[mudler]

@Ognian do not enable k3s manually with k3s.enable as it shadows the p2p configuration, can you try with the sample in the docs?

[mudler]

#cloud-config

hostname: "p2p-{{ trunc 4 .MachineID }}"

users:
- name: "kairos"
  passwd: "kairos"
  ssh_authorized_keys:
  - github:mudler

kairos:
 ## Generate a network token with the CLI as documented in https://kairos.io/docs/installation/p2p/#network_token
 network_token: ""

[Ognian]

OK, indeed if I remove any k3s and k3s-agent statements edgevpn ist started.
BUT I have now 2 more Problems:

• both start an k3s server so they are independent, not 2 nodes of the same cluster (one master and one agent)
• how shall I set a node label, when not allowed to use args...