[BUG] K3d cannot start K3s 1.16 & 1.17 in GitHub Actions

[nolar]

Hello. I am the maintainer of a GitHub Actions step for setting up K3s & K3d — https://github.com/nolar/setup-k3d-k3s. In this project and in one of the other projects I test K3s versions down to 1.16 inclusive.

Since recently (26.06.2021 – 30.06.2021), something has changed, and K3d versions 1.17 and 1.16 cannot start properly now. It was fine in GitHub Actions on 26.06.2021 with K3d 1.17 & 1.16. It remains fine with K3s 1.18-1.21. It is all fine on Macbook with all the recent and old versions of K3d. Versions of K3d seems to have no effect on the issue (in either way), but maybe I tried it wrong.

I.e., between Jun 26 and Jun 30, something has broken specifically in GitHub Actions and specifically with K3s 1.16-1.17.

Can you please hint what can that be? And how to debug it? Thanks.

What did you do

• How was the cluster created?

  • k3d cluster create --wait --image=rancher/k3s:v1.17.17-k3s1 (as here)

• What did you do afterwards?

  • k3d commands?
  • docker commands?
  • OS operations (e.g. shutdown/reboot)?

Nothing.

What did you expect to happen

It starts the cluster and the cluster is accessible to kubectl get serviceaccount default.

Screenshots or terminal output

E.g., here:

k3d installed into /usr/local/bin/k3d
Run 'k3d --help' to see what you can do with it.
k3d version v4.4.3
k3s version v1.20.6-k3s1 (default)

INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default' (d7c8ddf7787d2ce7e4c64f1389b86352cc9bb447e33fc7b468ac2e70f4521930) 
INFO[0000] Created volume 'k3d-k3s-default-images'      
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0001] Pulling image 'rancher/k3s:v1.17.17-k3s1'    
INFO[0008] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0008] Pulling image 'docker.io/rancher/k3d-proxy:v4.4.3' 
INFO[0011] Starting cluster 'k3s-default'               
INFO[0011] Starting servers...                          
INFO[0011] Starting Node 'k3d-k3s-default-server-0'     
INFO[0018] Starting agents...                           
INFO[0018] Starting helpers...                          
INFO[0018] Starting Node 'k3d-k3s-default-serverlb'     
INFO[0019] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
WARN[0022] Failed to patch CoreDNS ConfigMap to include entry '172.18.0.1 host.k3d.internal': Exec process in node 'k3d-k3s-default-server-0' failed with exit code '1' 
INFO[0022] Successfully added host record to /etc/hosts in 2/2 nodes 
INFO[0022] Cluster 'k3s-default' created successfully!  
INFO[0022] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 
INFO[0022] You can now use it like this:                
kubectl config use-context k3d-k3s-default
kubectl cluster-info

Then, cluster polling begins with kubectl get serviceaccount default every 1 second:

  Error from server (NotFound): serviceaccounts "default" not found
  Error from server (ServiceUnavailable): the server is currently unable to handle the request (get serviceaccounts default)
  Error from server (ServiceUnavailable): the server is currently unable to handle the request (get serviceaccounts default)
  Error from server (ServiceUnavailable): the server is currently unable to handle the request (get serviceaccounts default)
  Error from server (ServiceUnavailable): the server is currently unable to handle the request (get serviceaccounts default)
  Error from server (NotFound): serviceaccounts "default" not found
…………

And so until the build times out.

The only essential difference I have found is this:

In newly broken builds:

INFO[0017] Starting Node 'k3d-k3s-default-serverlb'     
INFO[0018] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 

WARN[0021] Failed to patch CoreDNS ConfigMap to include entry '172.18.0.1 host.k3d.internal': Exec process in node 'k3d-k3s-default-server-0' failed with exit code '1' 

INFO[0021] Successfully added host record to /etc/hosts in 2/2 nodes 
INFO[0021] Cluster 'k3s-default' created successfully!  

In old & new successful builds:

INFO[0014] Starting Node 'k3d-k3s-default-serverlb'     
INFO[0015] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
INFO[0018] Successfully added host record to /etc/hosts in 2/2 nodes and to the CoreDNS ConfigMap 
INFO[0018] Cluster 'k3s-default' created successfully!  

Which OS & Architecture

GitHub Actions, Ubuntu 20.04.2.

Which version of k3d

4.4.6, 4.4.4, 4.4.3.

Which version of docker

Client:
 Version:           20.10.6+azure
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c28948e3c12dce3d1df60b6f184990618553f
 Built:             Fri Apr  9 17:01:36 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.6+azure
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd246c3ab53105434eef8ffe997b6fd14dc6
  Built:            Fri Apr  9 22:06:18 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6+azure
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        

(e.g. here)

[iwilltry42]

Hi @nolar, thanks for opening this issue!
And great job with the GitHub Action :) (feel free to create a PR to add it as a related project to the README and k3d.io),

The warning shouldn't cause an issue here.
Please note though, that 1.16 and 1.17 are EOL already and 1.18 will be EOL in less than a month if I remember correctly.

Do you have any logs from the k3s containers that k3d created?
Or maybe kubectl output other than that regarding the serviceaccount? 🤔

[nolar]

You are right. They are deprecated for a long time. I usually keep the old versions supported as long as it requires near-zero effort (I thought maybe you know a quick solution). This is not the case, it seems — the effort will be high. So I've dropped 1.16 & 1.17.

Thanks for the response!

[iwilltry42]

@nolar we can still try to debug it 😁
Didn't mean that we shouldn't try 👍 I mean from your comments it doesn't look like it's an issue with K3s or K3d, so maybe it's something else.
If you think it's worth the effort, feel free to send some more info 👍

[nolar]

@iwilltry42 Extracting that info from GitHub Actions' runners is problematic. I simply do not know how to debug that interactively, so I will have to script it in a fake PR — which seems like overkill. It is easier to drop the old K8s versions.

I probably needed an excuse to drop 1.16 (and thus CRD v1beta support) from Kopf — and this looks like one. So, no worries. Thanks for the offer.