Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k3sup not working on fcos with testing channel #407

Open
1 of 2 tasks
adrecord opened this issue May 30, 2023 · 6 comments
Open
1 of 2 tasks

k3sup not working on fcos with testing channel #407

adrecord opened this issue May 30, 2023 · 6 comments

Comments

@adrecord
Copy link

I use k3sup to setup a k3s cluster on fcos nodes for personal cluster (libvirt vms). This used to work. Per this issue the stable channel doesn't have some rpms, but using the testing channel works. I verified that manually ssh'ing to the fcos node and running curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=testing sh - does work, as suggested in this comment. However, the equivalent version of this, to pass --k3s-channel testing to k3sup, does not work.

Expected Behaviour

I'm expecting k3sup installs k3s successfully.

Current Behaviour

$ ./k3sup install --host coreos01.adr.net --user root --k3s-channel testing --ssh-key ~/.ssh/id_ecdsa
Running: k3sup install
2023/05/30 00:26:20 coreos01.adr.net
Public IP: coreos01.adr.net
[INFO]  Finding release for channel testing
[INFO]  Using v1.27.2-rc3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Finding available k3s-selinux versions
Checking out tree e7109be...done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rancher-k3s-common-testing updates-archive
Updating metadata for 'fedora-cisco-openh264'...done
Updating metadata for 'fedora-modular'...done
Updating metadata for 'updates-modular'...done
Updating metadata for 'updates'...done
Updating metadata for 'fedora'...done
Updating metadata for 'rancher-k3s-common-testing'...done
Updating metadata for 'updates-archive'...done
Importing rpm-md...done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:56:46Z solvables: 4
rpm-md repo 'fedora-modular'; generated: 2023-04-13T20:30:47Z solvables: 1082
rpm-md repo 'updates-modular'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'updates'; generated: 2023-05-30T01:55:50Z solvables: 13522
rpm-md repo 'fedora'; generated: 2023-04-13T20:37:10Z solvables: 69222
rpm-md repo 'rancher-k3s-common-testing'; generated: 2023-05-25T20:15:06Z solvables: 8
rpm-md repo 'updates-archive'; generated: 2023-05-30T02:11:55Z solvables: 13778
Resolving dependencies...done
Will download: 1 package (21.2 kB)
Downloading from 'rancher-k3s-common-testing'...done
Importing packages...done
Checking out packages...done
Running pre scripts...done
Running post scripts...done
Running posttrans scripts...done
Writing rpmdb...done
Writing OSTree commit...done
Staging deployment...done
Freed: 2.1 MB (pkgcache branches: 0)
Added:
  k3s-selinux-1.3-10.coreos.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[WARN]  Failed to find the k3s-selinux policy, please install:
    rpm-ostree install -y container-selinux
    rpm-ostree install -y https://rpm-testing.rancher.io/k3s/testing/common/coreos/noarch/

[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Result: [INFO]  Finding release for channel testing
[INFO]  Using v1.27.2-rc3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Finding available k3s-selinux versions
Checking out tree e7109be...done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rancher-k3s-common-testing updates-archive
Updating metadata for 'fedora-cisco-openh264'...done
Updating metadata for 'fedora-modular'...done
Updating metadata for 'updates-modular'...done
Updating metadata for 'updates'...done
Updating metadata for 'fedora'...done
Updating metadata for 'rancher-k3s-common-testing'...done
Updating metadata for 'updates-archive'...done
Importing rpm-md...done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:56:46Z solvables: 4
rpm-md repo 'fedora-modular'; generated: 2023-04-13T20:30:47Z solvables: 1082
rpm-md repo 'updates-modular'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'updates'; generated: 2023-05-30T01:55:50Z solvables: 13522
rpm-md repo 'fedora'; generated: 2023-04-13T20:37:10Z solvables: 69222
rpm-md repo 'rancher-k3s-common-testing'; generated: 2023-05-25T20:15:06Z solvables: 8
rpm-md repo 'updates-archive'; generated: 2023-05-30T02:11:55Z solvables: 13778
Resolving dependencies...done
Will download: 1 package (21.2 kB)
Downloading from 'rancher-k3s-common-testing'...done
Importing packages...done
Checking out packages...done
Running pre scripts...done
Running post scripts...done
Running posttrans scripts...done
Writing rpmdb...done
Writing OSTree commit...done
Staging deployment...done
Freed: 2.1 MB (pkgcache branches: 0)
Added:
  k3s-selinux-1.3-10.coreos.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
 [WARN]  Failed to find the k3s-selinux policy, please install:
    rpm-ostree install -y container-selinux
    rpm-ostree install -y https://rpm-testing.rancher.io/k3s/testing/common/coreos/noarch/

Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

Error: error received processing command: Process exited with status 1

Possible Solution

not sure

Steps to Reproduce

  1. install fcos on a node
  2. run ./k3sup install --host coreos01.adr.net --user root --k3s-channel testing --ssh-key ~/.ssh/id_ecdsa

Your Environment

  • k3sup version:
Version: 0.12.12
Git Commit: 02c7a775b9914b9dcf3b90fa7935eb347b7979e7
  • What OS or type or VM are you using for your cluster? Where is it hosted? (for k3sup install/join):

FCOS

[root@coreos01 ~]# uname -a
Linux coreos01 6.2.13-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 27 01:33:30 UTC 2023 x86_64 GNU/Linux

[root@coreos01 ~]# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="38.20230430.3.1 (CoreOS)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora CoreOS 38.20230430.3.1"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='38.20230430.3.1'

Do you want to work on this?

Subject to design approval, are you willing to work on a Pull Request for this issue or feature request?

  • Yes
  • No
@adrecord
Copy link
Author

For comparison here's output of me manually running k3s installer, with the testing channel, on that fcos node..

[root@coreos01 ~]# curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=testing sh -
[INFO]  Finding release for channel testing
[INFO]  Using v1.27.2-rc3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Finding available k3s-selinux versions
Checking out tree e7109be... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rancher-k3s-common-testing updates-archive
Updating metadata for 'rancher-k3s-common-testing'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:56:46Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2023-04-13T20:30:47Z solvables: 1082
rpm-md repo 'updates-modular' (cached); generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'updates' (cached); generated: 2023-05-30T01:55:50Z solvables: 13522
rpm-md repo 'fedora' (cached); generated: 2023-04-13T20:37:10Z solvables: 69222
rpm-md repo 'rancher-k3s-common-testing'; generated: 2023-05-25T20:15:06Z solvables: 8
rpm-md repo 'updates-archive'; generated: 2023-05-30T02:11:55Z solvables: 13778
Resolving dependencies... done
Will download: 1 package (21.2 kB)
Downloading from 'rancher-k3s-common-testing'... done
Importing packages... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 2.1 MB (pkgcache branches: 0)
Added:
  k3s-selinux-1.3-10.coreos.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[WARN]  Failed to find the k3s-selinux policy, please install:
    rpm-ostree install -y container-selinux
    rpm-ostree install -y https://rpm-testing.rancher.io/k3s/testing/common/coreos/noarch/

[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[root@coreos01 ~]# echo $?
0

@alexellis
Copy link
Owner

Hi @adrecord

I've unfortunately not heard of fcos, can you please give us a link to the project you are referring to? I assume that's a shortened version of some other project?

What makes you believe that K3s should work on fcos?

This does seem like an issue for the k3s project and not k3sup, since k3sup just installs k3s over SSH.

If K3s fails to install, it implies a compatibility issue that must be solved upstream K3s itself.

Thoughts?

If you think that bash is working, but the k3sup installation may run with different parameters, you could also try adding --print-command to the command, such as:

k3sup install --print-command

Alex

@adrecord
Copy link
Author

adrecord commented Jun 1, 2023 •
edited
Loading

Thanks for your response, and thanks for this great project. I appreciate both.

I've unfortunately not heard of fcos, can you please give us a link to the project you are referring to? I assume that's a shortened version of some other project?

Sorry for the shorthand. By fcos I meant Fedora CoreOS.

This does seem like an issue for the k3s project and not k3sup, since k3sup just installs k3s over SSH.

So the text in the body of this issue included the snippet of output of trying to run k3sup. The first comment in this issue was me running k3s installer directly on the fcos host, for comparison. That did work, which is why I thought this was a k3sup issue.

If you think that bash is working, but the k3sup installation may run with different parameters, you could also try adding --print-command to the command, such as:

Thanks for this great advice. Seeing the commands led me to figure out what's going on. I can see that the k3s install line looks great and k3s installs fine. The error occurs after trying to sudo cat /etc/rancher/k3s/k3s.yaml. Sshing to that host, that file doesn't exist. It looks like k3s installed but didn't automatically start the service.

[root@coreos01 ~]# systemctl status k3s
○ k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
       Docs: https://k3s.io

I did not set INSTALL_K3S_SKIP_START to true, so I'm not sure why k3s didn't start automatically. Had k3s started automatically that file would have existed, allowing k3sup to continue. Here's the full output of the k3sup install with --print-command

$ ./k3sup install --host coreos01.adr.net --user root --k3s-channel testing --ssh-key ~/.ssh/id_ecdsa --print-command
Running: k3sup install
2023/05/31 21:36:48 coreos01.adr.net
Public IP: coreos01.adr.net
ssh: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='server --tls-san coreos01.adr.net' INSTALL_K3S_CHANNEL='testing' sh -

[INFO]  Finding release for channel testing
[INFO]  Using v1.27.2-rc3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Finding available k3s-selinux versions
Checking out tree e7109be...done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rancher-k3s-common-testing updates-archive
Updating metadata for 'fedora-cisco-openh264'...done
Updating metadata for 'fedora-modular'...done
Updating metadata for 'updates-modular'...done
Updating metadata for 'updates'...done
Updating metadata for 'fedora'...done
Updating metadata for 'rancher-k3s-common-testing'...done
Updating metadata for 'updates-archive'...done
Importing rpm-md...done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:56:46Z solvables: 4
rpm-md repo 'fedora-modular'; generated: 2023-04-13T20:30:47Z solvables: 1082
rpm-md repo 'updates-modular'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'updates'; generated: 2023-05-31T17:25:42Z solvables: 13915
rpm-md repo 'fedora'; generated: 2023-04-13T20:37:10Z solvables: 69222
rpm-md repo 'rancher-k3s-common-testing'; generated: 2023-05-25T20:15:06Z solvables: 8
rpm-md repo 'updates-archive'; generated: 2023-05-31T18:06:06Z solvables: 14547
Resolving dependencies...done
Will download: 1 package (21.2 kB)
Downloading from 'rancher-k3s-common-testing'...done
Importing packages...done
Checking out packages...done
Running pre scripts...done
Running post scripts...done
Running posttrans scripts...done
Writing rpmdb...done
Writing OSTree commit...done
Staging deployment...done
Freed: 2.1 MB (pkgcache branches: 0)
Added:
  k3s-selinux-1.3-10.coreos.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[WARN]  Please reboot your machine to activate the changes and avoid data loss.
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Result: [INFO]  Finding release for channel testing
[INFO]  Using v1.27.2-rc3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.2-rc3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Finding available k3s-selinux versions
Checking out tree e7109be...done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rancher-k3s-common-testing updates-archive
Updating metadata for 'fedora-cisco-openh264'...done
Updating metadata for 'fedora-modular'...done
Updating metadata for 'updates-modular'...done
Updating metadata for 'updates'...done
Updating metadata for 'fedora'...done
Updating metadata for 'rancher-k3s-common-testing'...done
Updating metadata for 'updates-archive'...done
Importing rpm-md...done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:56:46Z solvables: 4
rpm-md repo 'fedora-modular'; generated: 2023-04-13T20:30:47Z solvables: 1082
rpm-md repo 'updates-modular'; generated: 2018-02-20T19:18:14Z solvables: 0
rpm-md repo 'updates'; generated: 2023-05-31T17:25:42Z solvables: 13915
rpm-md repo 'fedora'; generated: 2023-04-13T20:37:10Z solvables: 69222
rpm-md repo 'rancher-k3s-common-testing'; generated: 2023-05-25T20:15:06Z solvables: 8
rpm-md repo 'updates-archive'; generated: 2023-05-31T18:06:06Z solvables: 14547
Resolving dependencies...done
Will download: 1 package (21.2 kB)
Downloading from 'rancher-k3s-common-testing'...done
Importing packages...done
Checking out packages...done
Running pre scripts...done
Running post scripts...done
Running posttrans scripts...done
Writing rpmdb...done
Writing OSTree commit...done
Staging deployment...done
Freed: 2.1 MB (pkgcache branches: 0)
Added:
  k3s-selinux-1.3-10.coreos.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
 [WARN]  Please reboot your machine to activate the changes and avoid data loss.
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

ssh: sudo cat /etc/rancher/k3s/k3s.yaml

Error: error received processing command: Process exited with status 1

@adrecord
Copy link
Author

adrecord commented Jun 1, 2023 •
edited
Loading

found it.. the k3s install script is defaulting INSTALL_K3S_SKIP_START to true if it's not explicitly set.

        coreos)
            rpm_installer="rpm-ostree"
            # rpm_install_extra_args="--apply-live"
            : "${INSTALL_K3S_SKIP_START:=true}"
            ;;

edit: I looked at the k3sup options, and I'm not seeing how to get k3sup to set that env variable on the install of k3s. Is that possible?

Where k3sup was calling..

ssh: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='server --tls-san coreos01.adr.net' INSTALL_K3S_CHANNEL='testing' sh -

it should look like..

ssh: curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START='false' INSTALL_K3S_EXEC='server --tls-san coreos01.adr.net' INSTALL_K3S_CHANNEL='testing' sh -

is there any way to call k3sup to make that happen?

Thanks!

@alexellis
Copy link
Owner

I think they have INSTALL_K3S_SKIP_START set to true because the logs say that you have to reboot your machine after installing K3s. Is that correct?

Freed: 2.1 MB (pkgcache branches: 0)
Added:
  k3s-selinux-1.3-10.coreos.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[WARN]  Please reboot your machine to activate the changes and avoid data loss.

If that's the case, you'd still want INSTALL_K3S_SKIP_START to remain as true, because it would fail without the reboot.

@adrecord
Copy link
Author

adrecord commented Jun 3, 2023

Perhaps having the node reboot to let systemctl start k3s the way it will on every subsequent reboot is cleaner, but it seems to work just fine without reboot. It also immediately creates the /etc/rancher/k3s/k3s.yaml file that k3sup is looking for to move back to the calling host, which would allow k3sup to complete successfully.

I guess this question is now just a generic question of... is there a way to pass environment variables thru k3sup such that they'll be seen in the environment of the k3s installer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexellis @adrecord