[BUG] host.k3d.internal breaks on system reboot

[mdshack]

What did you do

• How was the cluster created?

  • k3d cluster create [clustername] --volume ... --volume ... --registry-config [local path to registry config] --agents 1 --servers 1 --port 8081:8081@loadbalancer --port ... --port ... --port ... --k3s-arg --disable=traefik@server:0 --k3s-arg --disable=metrics-server@server:0

• What did you do afterwards?

1. Shell into the cluster agent: docker exec -it k3d-relay-agent-0 /bin/sh
2. wget a running registry that exists on the host machine successfully: wget host.k3d.internal:5000
   [refer to Screenshots or terminal output -> Successful wget below]
3. Restart host machine (an actual power cycle)
4. Shell into the cluster agent: docker exec -it k3d-relay-agent-0 /bin/sh
5. wget a running registry that exists on the host machine without success: wget host.k3d.internal:5000
   [refer to Screenshots or terminal output -> Unsuccessful wget below]
6. Stop your cluster: k3d cluster stop [clustername]
7. Start your cluster: k3d cluster start [clustername]
8. Repeat steps 1-2 and ensure you get another successful wget

What did you expect to happen

Expect host.k3d.internal:5000 to be reachable on machine restart

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Successful wget

Connecting to host.k3d.internal:5000 (172.20.0.1:5000)
saving to 'index.html'
'index.html' saved

Unsuccessful wget

wget: bad address 'host.k3d.internal:5000'

Which OS & Architecture

• output of k3d runtime-info

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "1"
endpoint: /var/run/docker.sock
filesystem: extfs
name: docker
os: Ubuntu 20.04.5 LTS
ostype: linux
version: 20.10.23

Which version of k3d

• output of k3d version

k3d version v5.4.6
k3s version v1.24.4-k3s1 (default)

Which version of docker

• output of docker version and docker info

Client: Docker Engine - Community
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:36:25 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.10
  Git commit:       6051f14
  Built:            Thu Jan 19 17:34:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.10.0-docker)
  compose: Docker Compose (Docker Inc., v2.15.1)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
 Containers: 11
  Running: 6
  Paused: 0
  Stopped: 5
 Images: 56
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b842e528e99d4d4c1686467debf2bd4b88ecd86
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.15.0-60-generic
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.42GiB
 Name: ....
 ID: ...
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: ...
 Registry: ...
 Labels:
 Experimental: false
 Insecure Registries:
 ...:5001
  ....:5002
  localhost:32000
  ....:5003
  ...:5000
  127.0.0.0/8
 Registry Mirrors:
  ...:5001/
 Live Restore Enabled: false

[jtele2]

I am having the same problem. Was working fine up until about a week ago.

[bruciebruce]

I have also been hit with this issue. Cluster is up on AWS. Builds fine with all pods running. When EC2 instance is rebooted most of the pods enter a crash loop.

ubuntu@ip-10-1-1-102:~$ k3d runtime-info
arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "1"
endpoint: /var/run/docker.sock
filesystem: extfs
name: docker
os: Ubuntu 20.04.5 LTS
ostype: linux
version: 20.10.12

ubuntu@ip-10-1-1-102:~$ k3d version
k3d version v5.4.7
k3s version v1.25.6-k3s1 (default)

ubuntu@ip-10-1-1-102:~$ docker info
Client:
Context: default
Debug Mode: false

Server:
Containers: 5
Running: 5
Paused: 0
Stopped: 0
Images: 3
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.15.0-1028-aws
Operating System: Ubuntu 20.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 61.81GiB
Name: ip-10-1-1-102
ID: KB2D:4ZRM:F5IX:G6ZP:JVCV:ORCW:D3FO:GG4J:N5RH:ZJVR:J2QL:TAPO
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[bruciebruce]

Trying to use Kind instead.

[CormacLennon]

Has anyone found a workaround for this yet? its a pain starting and stopping the cluster to get this back. Any way we can query the k3d instance to find out what it ought to be?

[DanHorrocksBurgess]

Has anyone found a workaround for this yet? its a pain starting and stopping the cluster to get this back. Any way we can query the k3d instance to find out what it ought to be?

Did you happen to find a workaround for this?

We're experiencing the same problem, our developers locally have to stop and start the cluster every time they reboot their machines to fix DNS resolution.

[CormacLennon]

Has anyone found a workaround for this yet? its a pain starting and stopping the cluster to get this back. Any way we can query the k3d instance to find out what it ought to be?

Did you happen to find a workaround for this?

We're experiencing the same problem, our developers locally have to stop and start the cluster every time they reboot their machines to fix DNS resolution.

I wrote a powershell function for our developers to run to fix the issue

function Repair-ClusterCoreDns()
{
$servero = docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' k3d-energy-server-0
$serverlb = docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' k3d-energy-serverlb
$registry = docker inspect --format='{{with index .NetworkSettings.Networks \"k3d-energy\"}}{{.IPAddress}}{{end}}' k3d-myregistry.localhost
$hostK3dInternal = Get-HostK3dInternal

$ips = "|
    $hostK3dInternal host.k3d.internal
    $servero k3d-energy-server-0
    $serverlb k3d-energy-serverlb
    $registry k3d-myregistry.localhost
"
    $patch = 'data:
  NodeHosts: ' + $ips
    Write-Output "Adding following members to the coredns config map:"
    Write-Output $patch
    kubectl patch configmap/coredns -n kube-system --type merge --patch $patch
}
Set-Alias -Name fixdns -Value Repair-ClusterCoreDns -Force
Export-ModuleMember -Function Repair-ClusterCoreDns -Alias fixdns

function Get-HostK3dInternal()
{
  $hostIp = ""
  $dnsEntries = docker exec k3d-energy-tools /bin/sh -c "getent ahostsv4 host.k3d.internal"

  foreach ( $dnsEntry in $dnsEntries) {
      $chunks = $dnsEntry.Split(" ") | Where-Object {$_}

      if($chunks[2] -eq "host.k3d.internal")
      {
        $hostIp = $chunks[0]
      }
  }

  if($hostIp -eq "")
  {
    Write-Host 'FAILURE: Could not resolve host.k3d.internal, Please ensure k3d-energy-tools container is running'
  }
  return $hostIp
}

its not perfect but it works, the important line to figure out what the host.k3d.internal ip should be is

docker exec k3d-energy-tools /bin/sh -c "getent ahostsv4 host.k3d.internal"

Which I only figured out reading the source and, as its undocumented, is liable to change. but it works for now

[Lanchez]

This is also a problem with local registries and it can easily be replicated. Local registries break when used from the cluster after a cluster restart.

1. Create cluster with a local registry
2. Check that coredns ConfigMap has proper entries
3. Restart docker daemon
4. Local registries break when accessed from the cluster and coredns ConfigMap is missing the entries