[Bug]: /etc/cloud/rename_interface.sh: No such file or directory

[Dani4kor]

Description

Following README Getting started instructions , facing this problem

module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Host: x.x.x.x
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   User: root
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Password: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Private key: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Certificate: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   SSH Agent: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Checking Host Key: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Target Platform: unix
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connected!
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + /etc/cloud/rename_interface.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): /tmp/terraform_1897964418.sh: line 3: /etc/cloud/rename_interface.sh: No such file or directory

╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.null_resource.first_control_plane,
│   on .terraform/modules/kube-hetzner/init.tf line 70, in resource "null_resource" "first_control_plane":
│   70:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_1897964418.sh": Process exited with status 127
╵

no /etc/cloud/rename_interface.sh is located on server

k3s-control-plane-fsn1-rjh:/etc/cloud # ls -la
total 8
drwxr-xr-x. 1 root root   72 Jun  7 17:20 .
drwxr-xr-x. 1 root root  278 Jun  8 14:54 ..
drwxr-xr-x. 1 root root   12 Jun  7 17:20 clean.d
-rw-r--r--. 1 root root 1088 Jun 20  2023 cloud.cfg
drwxr-xr-x. 1 root root   40 Jun  7 17:20 cloud.cfg.d
drwxr-xr-x. 1 root root 1420 Jun  7 17:20 templates


k3s-control-plane-fsn1-rjh:/tmp # cat /tmp/terraform_1897964418.sh
#!/bin/sh
set -ex
/etc/cloud/rename_interface.sh
mkdir -p /etc/rancher/k3s
[ -f /tmp/config.yaml ] && mv /tmp/config.yaml /etc/rancher/k3s/config.yaml
chmod 0600 /etc/rancher/k3s/config.yaml
[ -e /etc/rancher/k3s/k3s.yaml ] && exit 0
cat >> /etc/environment <<EOF

EOF
set -a; source /etc/environment; set +a;

cat > /etc/profile.d/00-alias.sh <<EOF
alias k=kubectl
EOF

cat > /etc/bash_completion.d/kubectl <<EOF
if command -v kubectl >/dev/null; then
  source <(kubectl completion bash)
  complete -o default -F __start_kubectl k
fi
EOF

timeout 180s /bin/sh -c 'while ! ping -c 1 1.1.1.1 >/dev/null 2>&1; do echo "Ready for k3s installation, waiting for a successful connection to the internet..."; sleep 5; done; echo Connected'
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_CHANNEL=v1.29 INSTALL_K3S_EXEC='server ' sh -
/sbin/semodule -v -i /usr/share/selinux/packages/k3s.pp
restorecon -v /usr/local/bin/k3s

builded image with packer as described in readme

k3s-control-plane-fsn1-rjh:/# cat /etc/os-release 
NAME="openSUSE MicroOS"
# VERSION="20240607"
ID="opensuse-microos"
ID_LIKE="suse opensuse opensuse-tumbleweed"
VERSION_ID="20240607"
PRETTY_NAME="openSUSE MicroOS"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:microos:20240607"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:MicroOS"
LOGO="distributor-logo-MicroOS"

Kube.tf file

locals {
  hcloud_token = ""
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
  source = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key = file("~/.ssh/id_rsa.pub")
  ssh_private_key = file("~/.ssh/id_rsa")
  ssh_max_auth_tries = 5
  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cx32",
      location    = "fsn1",
      labels      = ["control-plane", "master", "fsn1"],
      taints      = [],
      count       = 3,
      placement_group = "k3s-control-plane",
      backups = true
    }
  ]

agent_nodepools = [
    {
      name        = "agent-medium",
      server_type = "cx32",
      location    = "fsn1",
      labels      = ["agent", "slave", "fsn1", "medium"],
      taints      = [],
      count       = 0,
      placement_group = "k3s-agent-medium"
    }
  ]

  autoscaler_nodepools = [
    {
      name        = "agent-medium",
      server_type = "cx32",
      location    = "fsn1",
      min_nodes   = 1,
      max_nodes   = 2,
      placement_group = "k3s-agent-medium"
    }
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"
  load_balancer_disable_ipv6 = true
  load_balancer_health_check_interval = "5s"
  load_balancer_health_check_timeout = "3s"

  cluster_autoscaler_image = "registry.k8s.io/autoscaling/cluster-autoscaler"
  cluster_autoscaler_version = "v1.27.3"
  cluster_autoscaler_log_level = 4
  cluster_autoscaler_log_to_stderr = true
  cluster_autoscaler_stderr_threshold = "INFO"
  cluster_autoscaler_server_creation_timeout = 15

  enable_delete_protection = {
    floating_ip   = true
    load_balancer = true
    volume        = true
  }

  enable_csi_driver_smb = true
  ingress_controller = "nginx"
  enable_local_storage = true

  enable_cert_manager = true

  export_values = true
  create_kubeconfig = true
  enable_metrics_server = true
  restrict_outbound_traffic = false


  dns_servers = [
    "1.1.1.1",
    "8.8.8.8"
  ]
}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

No response

Platform

mac m1

[dalosa14]

same here

[mysticaltech]

Interesting, I'll have a look later today.

[mysticaltech]

@Dani4kor Can you terraform destroy, terraform init -upgrade and try again please?

[Dani4kor]

terraform destory

Destroy complete! Resources: X destroyed.

terraform init --upgrade

Initializing the backend...
Upgrading modules...
Downloading git::https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner.git for kube-hetzner...
- kube-hetzner in .terraform/modules/kube-hetzner
- kube-hetzner.agents in .terraform/modules/kube-hetzner/modules/host
- kube-hetzner.control_planes in .terraform/modules/kube-hetzner/modules/host

Initializing provider plugins...
- Finding hashicorp/local versions matching ">= 2.4.0"...
- Finding tenstad/remote versions matching ">= 0.1.2"...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/random...
- Finding latest version of hashicorp/cloudinit...
- Finding hetznercloud/hcloud versions matching ">= 1.43.0"...
- Finding integrations/github versions matching ">= 5.44.0"...
- Using previously-installed hashicorp/local v2.5.1
- Using previously-installed tenstad/remote v0.1.3
- Using previously-installed hashicorp/null v3.2.2
- Using previously-installed hashicorp/random v3.6.2
- Using previously-installed hashicorp/cloudinit v2.3.4
- Using previously-installed hetznercloud/hcloud v1.47.0
- Installing integrations/github v6.2.2...
- Installed integrations/github v6.2.2 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)

Result

....
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): Backing up /etc/rancher/k3s/registries.yaml to /tmp/registries_2024-06-21_17-35-53.yaml
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot stat '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): Updated registries.yaml detected, restart of k3s service required
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot create regular file '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries (remote-exec): k3s service or k3s-agent service restarted successfully
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): Backing up /etc/rancher/k3s/registries.yaml to /tmp/registries_2024-06-21_17-35-53.yaml
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot stat '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): Updated registries.yaml detected, restart of k3s service required
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot create regular file '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.registries: Creation complete after 3s [id=3551722489441731771]
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries (remote-exec): k3s service or k3s-agent service restarted successfully
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): Backing up /etc/rancher/k3s/registries.yaml to /tmp/registries_2024-06-21_17-35-53.yaml
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot stat '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): Updated registries.yaml detected, restart of k3s service required
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): cp: cannot create regular file '/etc/rancher/k3s/registries.yaml': No such file or directory
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): No active k3s or k3s-agent service found
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries (remote-exec): k3s service or k3s-agent service restarted successfully
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.registries: Creation complete after 3s [id=6349535723117724841]
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   Host: x.x.x.x
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   User: root
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   Password: false
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   Private key: true
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   Certificate: false
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   SSH Agent: true
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec):   Target Platform: unix
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   Host: x.x.x.x
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   User: root
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   Password: false
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   Private key: true
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   Certificate: false
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   SSH Agent: true
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec):   Target Platform: unix
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.registries: Creation complete after 3s [id=1060475260337499317]
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram: Provisioning with 'remote-exec'...
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   Host: x.x.x.x
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   User: root
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   Password: false
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   Private key: true
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   Certificate: false
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   SSH Agent: true
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   Checking Host Key: false
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec):   Target Platform: unix
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].hcloud_server_network.server: Creation complete after 4s [id=49163361-4361117]
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram (remote-exec): Connected!
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].hcloud_server_network.server: Creation complete after 5s [id=49163360-4361117]
module.kube-hetzner.module.control_planes["0-1-control-plane-fsn1"].null_resource.zram: Creation complete after 5s [id=6697467964240100583]
module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].null_resource.zram: Creation complete after 5s [id=6815029547267037962]
module.kube-hetzner.module.control_planes["0-2-control-plane-fsn1"].null_resource.zram: Creation complete after 6s [id=689469212275291105]
module.kube-hetzner.null_resource.first_control_plane: Creating...
module.kube-hetzner.null_resource.first_control_plane: Provisioning with 'file'...
module.kube-hetzner.data.cloudinit_config.autoscaler_config[0]: Reading...
module.kube-hetzner.data.cloudinit_config.autoscaler_config[0]: Read complete after 0s [id=3577989300]
module.kube-hetzner.null_resource.first_control_plane: Provisioning with 'remote-exec'...
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connecting to remote host via SSH...
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Host: x.x.x.x
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   User: root
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Password: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Private key: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Certificate: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   SSH Agent: true
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Checking Host Key: false
module.kube-hetzner.null_resource.first_control_plane (remote-exec):   Target Platform: unix
module.kube-hetzner.null_resource.first_control_plane (remote-exec): Connected!
module.kube-hetzner.null_resource.first_control_plane (remote-exec): + /etc/cloud/rename_interface.sh
module.kube-hetzner.null_resource.first_control_plane (remote-exec): /tmp/terraform_752420511.sh: line 3: /etc/cloud/rename_interface.sh: No such file or directory
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube-hetzner.null_resource.first_control_plane,
│   on .terraform/modules/kube-hetzner/init.tf line 70, in resource "null_resource" "first_control_plane":
│   70:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_752420511.sh": Process exited with status 127

kube.tf stay same

terraform --version

Terraform v1.8.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.4
+ provider registry.terraform.io/hashicorp/local v2.5.1
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.2
+ provider registry.terraform.io/hetznercloud/hcloud v1.47.0
+ provider registry.terraform.io/integrations/github v6.2.2
+ provider registry.terraform.io/tenstad/remote v0.1.3

[mysticaltech]

@Dani4kor Try placing the autoscaler in another placement group, who knows, and remove the version for autoscaler, use the latest instead (default now) of v1.30.1.

Also remove or increase ssh_max_auth_tries by a lot.

For the nodepool labels, carefully with the format, you might want to have an equal sign in there, see kube.tf.example. Not sure if it's really needed, but at least it's good practice, try it.

Last but not least, if all of the above do not work, proceed by first trying without all of those, just comment them out:

  cluster_autoscaler_server_creation_timeout = 15

  enable_delete_protection = {
    floating_ip   = true
    load_balancer = true
    volume        = true
  }

  enable_csi_driver_smb = true
  ingress_controller = "nginx"
  enable_local_storage = true

  enable_cert_manager = true

  export_values = true
  create_kubeconfig = true
  enable_metrics_server = true
  restrict_outbound_traffic = false

And then proceed by elimination, intuition, also looking at the k3s logs (see debug section in the readme). Also executing the fail script helps, it shows you the error.

[mysticaltech]

But, /etc/cloud/rename_interface.sh: No such file or directory indicates a probable SSH failure, so increasing the retry by a factor of 10x or 100x should help, try that first.

[Dani4kor]

Following your suggetions i start removing settings one-by-one, and its not helps:(

But you pointed on "ssh" problem and i start dive

TL;DR

Problem was in ssh key, i change the key- and all goes good

  ssh_public_key = file("~/.ssh/id_rsa_self.pub")
  ssh_private_key = file("~/.ssh/id_rsa_self")

#to 

  ssh_public_key = file("~/.ssh/id_rsa.pub")
  ssh_private_key = file("~/.ssh/id_rsa")

Whats wrong with key?

As im able to connect to servers with key deployed by module(checked path/logs whatever) - i dont pointing my attention for the key.

Trying to find difference in keys i found thats "wrong"

➜  k3s-hetzner git:(main) ✗ ssh-keygen -lf ~/.ssh/id_rsa_self.pub
3072 SHA256:xxxx no comment (RSA)
➜  k3s-hetzner git:(main) ✗ ssh-keygen -lf ~/.ssh/id_rsa.pub
3072 SHA256:xxxx name@pc (RSA)

type, lenght etc was the same except comment

In "bad key" comment was on new line ("\n")

#bad key
ssh-rsa ….
here@comment

#good key
ssh-rsa ... here@comment

Thats means:

• module added key to authorized_keys from file you source file you provided
• module copy it "as it"
• module not able to connect OR "do changes" # says its connected, but failed trying executing command
• we get error

openssh-client is able to connect even if authorized_keys contains bad format:
- seems its check it by pattern or by line, whatever

# openssh-client will able to connect with this authorized_keys on server
ssh-rsa …. [comment] <- checked
ssh-rsa ….  <- checked


# below bad data <- ignored
somedata <- ignored
hello <- ignored
asdfsadfsadfsad <- ignored
cat /etc/passwd <- ignored

ssh-rsa ..bad_format..  [comment] <- ignored

ssh-rsa …. [comment] <- checked

Possbile iam wrong somewhere in my research, feel free to fix/point where iam wrong
Anyway, after changing it to right format(remove comment) on id_rsa_self - all comes good

Thanks for help and your time @mysticaltech