[Release-1.26] - Fix rootless node password location

[dereknola]

Backport fix for Fix rootless node password location
Original Issue: #3636

• Fix rootless node password location #7887

[fmoral2]

Validated on Version:

-$ k3s version v1.26.6+k3s-6c6745b1 (6c6745b15e1e93fec257e75900b91d6e2de4d11c)

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu

Cluster Configuration:
1 node

Steps to validate the fix

1. Download and Install rootless k3s
2. Reload daemon
3. Apply net.ipv4.ip_forward and net.ipv6.conf.all.forwarding=1
4. Install uidmap and update-grub
5. Check for no errors related here - journalctl --user -u k3s-rootless|grep "unable to read node password file"
6. And here systemd-run --user -p Delegate=yes --tty k3s server --rootless

Docs:
https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation
https://docs.k3s.io/advanced#advanced-rootless-configuration

Validation Results:

    $ k3s -v
    k3s version v1.26.6+k3s-6c6745b1 (6c6745b1)
    go version go1.19.10



    ~$ cat /home/ubuntu/.config/systemd/user/k3s-rootless.service

    [Unit]
    Description=k3s (Rootless)


    ~$ grep cgroup /proc/mounts
    cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0


~$ stat -c %T -f /sys/fs/cgroup
    cgroup2fs

 ~$ cat /sys/fs/cgroup/cgroup.controllers
    cpuset cpu io memory hugetlb pids rdma misc

~$ sudo cat /etc/sysctl.conf


    # Uncomment the next line to enable packet forwarding for IPv4
    net.ipv4.ip_forward=1

    # Uncomment the next line to enable packet forwarding for IPv6
    #  Enabling this option disables Stateless Address Autoconfiguration
    #  based on Router Advertisements for this host
    net.ipv6.conf.all.forwarding=1


~$ sudo cat /etc/default/grub
    # If you change this file, run 'update-grub' afterwards to update
    # /boot/grub/grub.cfg.
    # For full documentation of the options in this file, see:
    #   info -f grub -n 'Simple configuration'

    GRUB_DEFAULT=0
    GRUB_TIMEOUT_STYLE=hidden
    GRUB_TIMEOUT=0
    GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
    GRUB_CMDLINE_LINUX=""


    ~$ grep cgroup /proc/mounts
    cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0


  ~$ stat -c %T -f /sys/fs/cgroup
    cgroup2fs


 ~$ cat /sys/fs/cgroup/cgroup.controllers
    cpuset cpu io memory hugetlb pids rdma misc



$ sudo sysctl -p
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 1


 ~$ systemctl --user status k3s-rootless
    ● k3s-rootless.service - k3s (Rootless)
    Loaded: loaded (/home/ubuntu/.config/systemd/user/k3s-rootless.service; enabled; vendor preset: enabled)
    Active: active (running) since Wed 2023-07-12 15:07:19 UTC; 4s ago
    Main PID: 998 (k3s-server)
    Tasks: 37
    Memory: 498.5M
    CPU: 5.230s
    CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/k3s-rootless.service
    └─k3s_evac
    ├─ 998 "/usr/local/bin/k3s server" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
    ├─1010 "/proc/self/exe init" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
    ├─1020 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --cidr 10.41.0.0/16 1010 tap0
    ├─1024 "k3s server" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
    ├─1044 "k3s server" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
    └─1064 "containerd " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ">

    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=info msg="certificate CN=system:node:ip-172-31-21-12,O=system:nodes signed by CN=k3s-client-ca@1689174441: notBefore=2023-07-12 >
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=warning msg="Failed to load kernel module overlay with modprobe"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=warning msg="Failed to load kernel module nf_conntrack with modprobe"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=warning msg="Failed to load kernel module br_netfilter with modprobe"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=warning msg="Failed to load kernel module iptable_nat with modprobe"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=warning msg="Failed to load kernel module iptable_filter with modprobe"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=info msg="Set sysctl 'net/bridge/bridge-nf-call-iptables' to 1"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=error msg="Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=info msg="Logging containerd to /home/ubuntu/.rancher/k3s/agent/containerd/containerd.log"
    Jul 12 15:07:23 ip-172-31-21-12 k3s[1044]: time="2023-07-12T15:07:23Z" level=info msg="Running containerd -c /home/ubuntu/.rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.soc>
    lines 1-26/26 (END)


~$ journalctl --user -u k3s-rootless|grep "unable to read node password file"
    ubuntu@ip-172-31-21-12:~$



 ~$KUBECONFIG=~/.kube/k3s.yaml kubectl get pods -A
    NAMESPACE     NAME                                      READY   STATUS      RESTARTS        AGE
    kube-system   helm-install-traefik-crd-qp5l6            0/1     Completed   0               7m55s
    kube-system   helm-install-traefik-4tlqr                0/1     Completed   2               7m55s
    kube-system   coredns-59b4f5bbd5-cbntj                  1/1     Running     1 (2m51s ago)   7m54s
    kube-system   local-path-provisioner-76d776f6f9-4tgw6   1/1     Running     1 (2m51s ago)   7m55s
    kube-system   svclb-traefik-73d1c75a-22qkh              2/2     Running     2 (2m51s ago)   7m25s
    kube-system   traefik-57c84cf78d-sctx6                  1/1     Running     1 (2m51s ago)   7m25s
    kube-system   metrics-server-68cf49699b-k5gkc           1/1     Running     1 (2m51s ago)   7m55s



$ systemd-run --user -p Delegate=yes --tty k3s server --rootless

    Running as unit: run-u2.service
    Press ^] three times within 1s to disconnect TTY.
    INFO[0000] Starting k3s v1.26.6+k3s-6c6745b1 (6c6745b1)
    INFO[0000] Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s
    INFO[0000] Configuring database table schema and indexes, this may take a moment...
    INFO[0000] Database tables and indexes are up to date
    INFO[0000] Kine available at unix://kine.sock
    INFO[0000] Reconciling bootstrap data between datastore and disk
    INFO[0000] Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anon

    e-status-update-frequency=1m0s --profiling=false
    I0712 15:18:18.176858      28 server.go:172] Version: v1.26.6+k3s-6c6745b1
    I0712 15:18:18.176896      28 server.go:174] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
    INFO[0000] Server node token is available at /home/ubuntu/.rancher/k3s/server/token
    INFO[0000] To join server node to cluster: k3s server -s https://10.41.0.100:6443 -t ${SERVER_NODE_TOKEN}
    INFO[0000] Agent node token is available at /home/ubuntu/.rancher/k3s/server/agent-token
    INFO[0000] To join agent node to cluster: k3s agent -s https://10.41.0.100:6443 -t ${AGENT_NODE_TOKEN}
    INFO[0000] Wrote kubeconfig /home/ubuntu/.kube/k3s.yaml
    INFO[0000] Run: k3s kubectl
    INFO[0000] Waiting for API server to become available
    I0712 15:18:18.245033      28 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
    I0712 15:18:18.245067      28 plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota.
    I0712 15:18:18.252303      28 shared_informer.go:270] Waiting for caches to sync for node_authorizer
    W0712 15:18:18.300136      28 genericapiserver.go:660] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
    I0712 15:18:18.301681      28 instance.go:277] Using reconciler: lease
    I0712 15:18:18.554954      28 instance.go:621] API group "internal.apiserver.k8s.io" is not enabled, skipping.
    I0712 15:18:18.781255      28 instance.go:621] API group "resource.k8s.io" is not enabled, skipping.
    W0712 15:18:18.944194      28 genericapiserver.go:660] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.944227      28 genericapiserver.go:660] Skipping API authentication.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:18.947081      28 genericapiserver.go:660] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.954020      28 genericapiserver.go:660] Skipping API autoscaling/v2beta1 because it has no resources.
    W0712 15:18:18.954048      28 genericapiserver.go:660] Skipping API autoscaling/v2beta2 because it has no resources.
    W0712 15:18:18.957821      28 genericapiserver.go:660] Skipping API batch/v1beta1 because it has no resources.
    W0712 15:18:18.960969      28 genericapiserver.go:660] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.963785      28 genericapiserver.go:660] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.963856      28 genericapiserver.go:660] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.970698      28 genericapiserver.go:660] Skipping API networking.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.970736      28 genericapiserver.go:660] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:18.973435      28 genericapiserver.go:660] Skipping API node.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.973463      28 genericapiserver.go:660] Skipping API node.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:18.973516      28 genericapiserver.go:660] Skipping API policy/v1beta1 because it has no resources.
    W0712 15:18:18.980554      28 genericapiserver.go:660] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.980586      28 genericapiserver.go:660] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:18.983281      28 genericapiserver.go:660] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.983309      28 genericapiserver.go:660] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:18.990790      28 genericapiserver.go:660] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:18.997901      28 genericapiserver.go:660] Skipping API flowcontrol.apiserver.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:18.997933      28 genericapiserver.go:660] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:19.004566      28 genericapiserver.go:660] Skipping API apps/v1beta2 because it has no resources.
    W0712 15:18:19.004596      28 genericapiserver.go:660] Skipping API apps/v1beta1 because it has no resources.
    W0712 15:18:19.007933      28 genericapiserver.go:660] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:19.007962      28 genericapiserver.go:660] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
    W0712 15:18:19.010813      28 genericapiserver.go:660] Skipping API events.k8s.io/v1beta1 because it has no resources.
    W0712 15:18:19.046600      28 genericapiserver.go:660] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
    INFO[0002] Password verified locally for node ip-172-31-21-12


    ```