Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Latest install.sh exits without error message on airgapped nodes #7247

Closed
djpbessems opened this issue Apr 7, 2023 · 19 comments
Closed

Latest install.sh exits without error message on airgapped nodes #7247

djpbessems opened this issue Apr 7, 2023 · 19 comments
Assignees
Labels
area/airgap kind/bug Something isn't working
Milestone

Comments

@djpbessems
Copy link

djpbessems commented Apr 7, 2023

Environmental Info:
K3s Version:
master-branch

Node(s) CPU architecture, OS, and Version:
Ubuntu 20.04 LTS

Cluster Configuration:
N/A

Describe the bug:
When running the latest install.sh with below command, it exits right after showing [INFO] Skipping k3s download and verify without an error message (exit code is not 0 though):

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="server --cluster-init --token <token> --tls-san <ipaddress> --disable local-storage --config /etc/rancher/k3s/config.yaml" ./install.sh

Steps To Reproduce:
See above

  • Installed K3s:
    N/A

Expected behavior:
Installation starts.

Actual behavior:
See above

Additional context / logs:
The install.sh from tag v1.26.3+k3s1 still works correctly.

/kind bug

@brandond
Copy link
Contributor

brandond commented Apr 7, 2023

cc @galal-hussein

@galal-hussein
Copy link
Contributor

galal-hussein commented Apr 7, 2023

@djpbessems I am not able to reproduce with these following steps:

  • on an ubuntu 20.04 server download k3s binary:
wget https://github.com/k3s-io/k3s/releases/download/v1.26.3%2Bk3s1/k3s
chmod +x k3s
cp k3s /usr/local/bin
  • create the config.yaml file
mkdir -p /etc/rancher/k3s
echo "selinux: false" >> /etc/rancher/k3s/config.yaml
  • run the install script
# curl https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="server --cluster-init --token test --tls-san xxxxx --disable local-storage --config /etc/rancher/k3s/config.yaml" sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29027  100 29027    0     0   383k      0 --:--:-- --:--:-- --:--:--  388k
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

can you make sure that the steps are the same, or am I missing something

@brandond
Copy link
Contributor

brandond commented Apr 7, 2023

@djpbessems You might try running the script with -x to show what command is failing:

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="server --cluster-init --token <token> --tls-san <ipaddress> --disable local-storage --config /etc/rancher/k3s/config.yaml" sh -x ./install.sh

@djpbessems
Copy link
Author

djpbessems commented Apr 7, 2023

I realised I forgot to share that this is happening on an air-gapped system.

Since k3s (and kube-vip) needs a default route, I've configured the system with a bogus default route. When I then run the latest version of the install script, it stops with a silent error (last few lines of output when running with -x):

[...]
+ curl+ grep -oP (?<="browser_download_url": ")[^"]*
 -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest
+ grep -oE [^\/]+el8\.noarch\.rpm
+ available_version=

When I then replace the default route with a working gateway, the command succeeds again. So the latest changes make k3s unable to be installed in an air-gapped environment despite the INSTALL_K3S_SKIP_DOWNLOAD setting.

@brandond
Copy link
Contributor

brandond commented Apr 7, 2023

@galal-hussein it looks like the change at 027cc18#diff-043df5bdbf6639d7a77e1d44c5226fd7371e5259a1e4df3a0dd5d64c30dca44fR500-R504 makes it call out to the GitHub API to retrieve the latest release of the k3s-selinux package, even if it's not going to be shown.

While I do like the idea of having the script call out to get the latest version to suggest, we probably need to make that an optional check with a short timeout, and fall back to a known-good default value. It also shouldn't happen when we're not downloading anything.

@brandond brandond added this to the v1.26.4+k3s1 milestone Apr 7, 2023
@brandond brandond added kind/bug Something isn't working area/airgap labels Apr 7, 2023
@Rudedog9d
Copy link

+1 to this issue on CoreOS on versions 37.20230218.3.0 (CoreOS) and 37.20230322.3.0.

Full OS-Release for our primary image

NAME="Fedora Linux"
VERSION="37.20230218.3.0 (CoreOS)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora CoreOS 37.20230218.3.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=37
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=37
SUPPORT_END=2023-11-14
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='37.20230218.3.0'

We're also seeing a silent failure at the available_versions step. We were confused by the grep working outside of the script but not in the script - and we realized it's the rpm_target variable causing it to fail.

Here's a bunch of debug info. Let me know if you need something additional.

Last lines of script with `-x`

[INFO]  Installing k3s to /usr/local/bin/k3s
+ chown root:root /tmp/k3s-install.dkFHc26FAG/k3s.bin
+ mv -f /tmp/k3s-install.dkFHc26FAG/k3s.bin /usr/local/bin/k3s
+ setup_selinux
+ case ${INSTALL_K3S_CHANNEL} in
+ rpm_channel=stable
+ rpm_site=rpm.rancher.io
+ '[' stable = testing ']'
+ '[' -r /etc/os-release ']'
+ . /etc/os-release
++ NAME='Fedora Linux'
++ VERSION='37.20230218.3.0 (CoreOS)'
++ ID=fedora
++ VERSION_ID=37
++ VERSION_CODENAME=
++ PLATFORM_ID=platform:f37
++ PRETTY_NAME='Fedora CoreOS 37.20230218.3.0'
++ ANSI_COLOR='0;38;2;60;110;180'
++ LOGO=fedora-logo-icon
++ CPE_NAME=cpe:/o:fedoraproject:fedora:37
++ HOME_URL=https://getfedora.org/coreos/
++ DOCUMENTATION_URL=https://docs.fedoraproject.org/en-US/fedora-coreos/
++ SUPPORT_URL=https://github.com/coreos/fedora-coreos-tracker/
++ BUG_REPORT_URL=https://github.com/coreos/fedora-coreos-tracker/
++ REDHAT_BUGZILLA_PRODUCT=Fedora
++ REDHAT_BUGZILLA_PRODUCT_VERSION=37
++ REDHAT_SUPPORT_PRODUCT=Fedora
++ REDHAT_SUPPORT_PRODUCT_VERSION=37
++ SUPPORT_END=2023-11-14
++ VARIANT=CoreOS
++ VARIANT_ID=coreos
++ OSTREE_VERSION=37.20230218.3.0
++ expr '' : '.*suse.*'
+ '[' 0 '!=' 0 ']'
+ '[' 37 = 7 ']'
+ '[' '' = coreos ']'
+ '[' coreos = coreos ']'
+ rpm_target=coreos
+ rpm_site_infix=coreos
+ package_installer=rpm-ostree
+ '[' rpm-ostree = rpm-ostree ']'
+ '[' -x /bin/yum ']'
+ '[' rpm-ostree = yum ']'
+ '[' stable = testing ']'
++ curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest
++ grep -oP '(?<="browser_download_url": ")[^"]*'
++ grep -oE '[^\/]+coreos\.noarch\.rpm'
+ available_version=
+ cleanup
+ code=1
+ set +e
+ trap - EXIT
+ rm -rf /tmp/k3s-install.dkFHc26FAG
+ exit 1

curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest

{
  "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/74469641",
  "assets_url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/74469641/assets",
  "upload_url": "https://uploads.github.com/repos/k3s-io/k3s-selinux/releases/74469641/assets{?name,label}",
  "html_url": "https://github.com/k3s-io/k3s-selinux/releases/tag/v1.2.stable.2",
  "id": 74469641,
  "author": {
    "login": "brandond",
    "id": 370103,
    "node_id": "MDQ6VXNlcjM3MDEwMw==",
    "avatar_url": "https://avatars.githubusercontent.com/u/370103?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/brandond",
    "html_url": "https://github.com/brandond",
    "followers_url": "https://api.github.com/users/brandond/followers",
    "following_url": "https://api.github.com/users/brandond/following{/other_user}",
    "gists_url": "https://api.github.com/users/brandond/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/brandond/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/brandond/subscriptions",
    "organizations_url": "https://api.github.com/users/brandond/orgs",
    "repos_url": "https://api.github.com/users/brandond/repos",
    "events_url": "https://api.github.com/users/brandond/events{/privacy}",
    "received_events_url": "https://api.github.com/users/brandond/received_events",
    "type": "User",
    "site_admin": false
  },
  "node_id": "RE_kwDODnar2c4EcFEJ",
  "tag_name": "v1.2.stable.2",
  "target_commitish": "master",
  "name": "v1.2.stable.2",
  "draft": false,
  "prerelease": false,
  "created_at": "2022-06-10T21:43:16Z",
  "published_at": "2022-08-16T00:40:26Z",
  "assets": [
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818004",
      "id": 74818004,
      "node_id": "RA_kwDODnar2c4EdaHU",
      "name": "k3s-selinux-1.2-2.el7.noarch.rpm",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-rpm",
      "state": "uploaded",
      "size": 16240,
      "download_count": 1014,
      "created_at": "2022-08-16T00:41:49Z",
      "updated_at": "2022-08-16T00:41:50Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.noarch.rpm"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818006",
      "id": 74818006,
      "node_id": "RA_kwDODnar2c4EdaHW",
      "name": "k3s-selinux-1.2-2.el7.src.rpm",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-rpm",
      "state": "uploaded",
      "size": 18725,
      "download_count": 69,
      "created_at": "2022-08-16T00:41:50Z",
      "updated_at": "2022-08-16T00:41:50Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.src.rpm"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818087",
      "id": 74818087,
      "node_id": "RA_kwDODnar2c4EdaIn",
      "name": "k3s-selinux-1.2-2.el8.noarch.rpm",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-rpm",
      "state": "uploaded",
      "size": 20696,
      "download_count": 3931,
      "created_at": "2022-08-16T00:43:04Z",
      "updated_at": "2022-08-16T00:43:04Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.noarch.rpm"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818088",
      "id": 74818088,
      "node_id": "RA_kwDODnar2c4EdaIo",
      "name": "k3s-selinux-1.2-2.el8.src.rpm",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-rpm",
      "state": "uploaded",
      "size": 23061,
      "download_count": 116,
      "created_at": "2022-08-16T00:43:04Z",
      "updated_at": "2022-08-16T00:43:04Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.src.rpm"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818174",
      "id": 74818174,
      "node_id": "RA_kwDODnar2c4EdaJ-",
      "name": "k3s-selinux-1.2-2.sle.noarch.rpm",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-rpm",
      "state": "uploaded",
      "size": 20958,
      "download_count": 96,
      "created_at": "2022-08-16T00:44:42Z",
      "updated_at": "2022-08-16T00:44:42Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.noarch.rpm"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818196",
      "id": 74818196,
      "node_id": "RA_kwDODnar2c4EdaKU",
      "name": "k3s-selinux-1.2-2.sle.src.rpm",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "application/x-rpm",
      "state": "uploaded",
      "size": 23404,
      "download_count": 60,
      "created_at": "2022-08-16T00:44:42Z",
      "updated_at": "2022-08-16T00:44:43Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.src.rpm"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818007",
      "id": 74818007,
      "node_id": "RA_kwDODnar2c4EdaHX",
      "name": "sha256sum-centos7-noarch.txt",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "text/plain; charset=utf-8",
      "state": "uploaded",
      "size": 195,
      "download_count": 185,
      "created_at": "2022-08-16T00:41:50Z",
      "updated_at": "2022-08-16T00:41:51Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos7-noarch.txt"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818089",
      "id": 74818089,
      "node_id": "RA_kwDODnar2c4EdaIp",
      "name": "sha256sum-centos8-noarch.txt",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "text/plain; charset=utf-8",
      "state": "uploaded",
      "size": 195,
      "download_count": 83,
      "created_at": "2022-08-16T00:43:04Z",
      "updated_at": "2022-08-16T00:43:05Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos8-noarch.txt"
    },
    {
      "url": "https://api.github.com/repos/k3s-io/k3s-selinux/releases/assets/74818199",
      "id": 74818199,
      "node_id": "RA_kwDODnar2c4EdaKX",
      "name": "sha256sum-microos-noarch.txt",
      "label": "",
      "uploader": {
        "login": "ks3serviceaccount",
        "id": 74435438,
        "node_id": "MDQ6VXNlcjc0NDM1NDM4",
        "avatar_url": "https://avatars.githubusercontent.com/u/74435438?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/ks3serviceaccount",
        "html_url": "https://github.com/ks3serviceaccount",
        "followers_url": "https://api.github.com/users/ks3serviceaccount/followers",
        "following_url": "https://api.github.com/users/ks3serviceaccount/following{/other_user}",
        "gists_url": "https://api.github.com/users/ks3serviceaccount/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/ks3serviceaccount/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/ks3serviceaccount/subscriptions",
        "organizations_url": "https://api.github.com/users/ks3serviceaccount/orgs",
        "repos_url": "https://api.github.com/users/ks3serviceaccount/repos",
        "events_url": "https://api.github.com/users/ks3serviceaccount/events{/privacy}",
        "received_events_url": "https://api.github.com/users/ks3serviceaccount/received_events",
        "type": "User",
        "site_admin": false
      },
      "content_type": "text/plain; charset=utf-8",
      "state": "uploaded",
      "size": 195,
      "download_count": 62,
      "created_at": "2022-08-16T00:44:43Z",
      "updated_at": "2022-08-16T00:44:43Z",
      "browser_download_url": "https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-microos-noarch.txt"
    }
  ],
  "tarball_url": "https://api.github.com/repos/k3s-io/k3s-selinux/tarball/v1.2.stable.2",
  "zipball_url": "https://api.github.com/repos/k3s-io/k3s-selinux/zipball/v1.2.stable.2",
  "body": "## What's Changed\r\n* Use SHA256 to sign packages instead of default SHA1 by @brandond in https://github.com/k3s-io/k3s-selinux/pull/32\r\n* Bump pip/setuptools version; switch to https for git clone by @brandond in https://github.com/k3s-io/k3s-selinux/pull/33\r\n\r\n**Full Changelog**: https://github.com/k3s-io/k3s-selinux/compare/v1.1.stable.1...v1.2.stable.2",
  "mentions_count": 1
}

grep -oP '(?<="browser_download_url": ")[^"]*'

curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest | grep -oP '(?<="browser_download_url": ")[^"]*'
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.noarch.rpm
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el7.src.rpm
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.noarch.rpm
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.src.rpm
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.noarch.rpm
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.sle.src.rpm
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos7-noarch.txt
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-centos8-noarch.txt
https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/sha256sum-microos-noarch.txt

All together now

curl -s https://api.github.com/repos/k3s-io/k3s-selinux/releases/latest | grep -oP '(?<="browser_download_url": ")[^"]*' |  grep -oE '[^/]+coreos.noarch.rpm'
# no return, exit status 1

@Rudedog9d
Copy link

We fell back to the previous version of the script for now, as we need to spin up a cluster for a competition this weekend (new cluster each week). Will look out for the fix on this ticket!

Link to previous version of script for anyone looking to do the same:
https://raw.githubusercontent.com/k3s-io/k3s/7e59376bb91d451d3eaf16b9a3f80ae4d711b2bc/install.sh

@Rudedog9d
Copy link

Rudedog9d commented Apr 7, 2023

Also as a note - the script appears to work as is if exporting the variable INSTALL_K3S_CHANNEL=testing, although our automated cluster join step didn't appear to work.

Logs from script

[INFO]  Using v1.25.4+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt
[INFO]  Skipping binary downloaded, installed k3s matches hash
Inactive requests:
  k3s-selinux (already provided by k3s-selinux-0.5-1.el8.noarch)
Checking out tree d34ff10... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rancher-k3s-common-testing updates-archive
Updating metadata for 'rancher-k3s-common-testing'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2022-10-06T11:01:40Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2022-11-05T07:58:03Z solvables: 1454
rpm-md repo 'updates-modular' (cached); generated: 2023-02-22T11:28:37Z solvables: 1464
rpm-md repo 'updates' (cached); generated: 2023-04-07T02:52:47Z solvables: 28764
rpm-md repo 'fedora' (cached); generated: 2022-11-05T08:04:38Z solvables: 66822
rpm-md repo 'rancher-k3s-common-testing'; generated: 2023-02-09T22:23:18Z solvables: 2
rpm-md repo 'updates-archive' (cached); generated: 2023-04-07T03:57:22Z solvables: 36528
Resolving dependencies... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Changes queued for next boot. Run "systemctl reboot" to start a reboot
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.

@fieldju
Copy link

fieldju commented Apr 7, 2023

Not sure if this is realated but I am seeing today that curl -sfL https://get.k3s.io | bash - started to fail intermittently but its not exiting 1

When I SSH'd on to the EC2 instance I had to run the install script by hand a few times before it worked
image

I'm trying to work around it with some retries in my script, but if this is round robin'ing to some old script thats in cache or what not then retry'ing will start failing

echo "INSTALLING_K3S" > /home/admin/status

install_k3s() {
  echo "installing k3s"
  curl -sfL https://get.k3s.io | bash -
  sudo chown -R admin:admin /etc/rancher/k3s
  echo "finished installing k3s"
}

# Install K3s with 20 retries
for i in {1..20}; do install_k3s && break || sleep 1; done

echo "INSTALLING_ADMISSION_CONTROLLER" > /home/admin/status

I might look into #7247 (comment) and see if I can pin to a known working script.

@fieldju
Copy link

fieldju commented Apr 7, 2023

We fell back to the previous version of the script for now, as we need to spin up a cluster for a competition this weekend (new cluster each week). Will look out for the fix on this ticket!

Link to previous version of script for anyone looking to do the same: https://raw.githubusercontent.com/k3s-io/k3s/7e59376bb91d451d3eaf16b9a3f80ae4d711b2bc/install.sh

FWIW I tried pinning to this version and was still experiencing my issue.

@Rudedog9d
Copy link

@fieldju sounds like you're having a different issue. Our experienced of it has been completely consistent thus far.

@brandond
Copy link
Contributor

brandond commented Apr 8, 2023

@fieldju that is unrelated. The K3s channel server is on infrastructure that is occasionally overloaded when Rancher releases chart updates.

@brandond brandond changed the title Latest install.sh exits without error message Latest install.sh exits without error message on airgapped nodes Apr 8, 2023
@terrytangyuan
Copy link

terrytangyuan commented Apr 8, 2023

Not sure if this is realated but I am seeing today that curl -sfL https://get.k3s.io | bash - started to fail intermittently but its not exiting 1

I am seeing the same issue. https://github.com/argoproj/argo-workflows/actions/runs/4641097132/jobs/8217213836

Any suggestions?

What I did was as usual:

Run curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION= INSTALL_K3S_CHANNEL=stable INSTALL_K3S_EXEC=--docker K3S_KUBECONFIG_MODE=644 sh -
[INFO]  Finding release for channel stable
[INFO]  Using stable as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/stable/sha256sum-amd64.txt
Error: Process completed with exit code 22.

@brandond
Copy link
Contributor

brandond commented Apr 8, 2023

@terrytangyuan see my message directly above yours.

@sradigan
Copy link

sradigan commented Apr 8, 2023

I just pushed up a commit with some changes that I think will help with some if the issues here.

I'll test it and make a pull request if this at least resolves the coreos issue.

1b1594b

Edit: Note that the boolean on the "testing" check is != (not equals) rather than = (equals). The block was nested in the else so that the ENV var could be set on the air-gapped system to skip the curl to the net.

@terrytangyuan
Copy link

terrytangyuan commented Apr 10, 2023

Thanks. Looks like stable is no longer available. I pinned the CI to a specific version.

@jgbright
Copy link

I am seeing an error that may have the same root cause, related to using the -P option with grep. I started seeing this on Friday, April 7. The timing of when the error started seems to coincide with the code change referenced earlier in the thread.

root@qemuarm64:~# INSTALL_K3S_EXEC="--snapshotter=native" INSTALL_K3S_SKIP_DOWNLOAD=true ./k3s_install.sh
[INFO]  Skipping k3s download and verify
grep: invalid option -- 'P'
BusyBox v1.31.1 (2022-12-13 23:09:41 UTC) multi-call binary.

Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...

@brandond
Copy link
Contributor

brandond commented Apr 10, 2023

cc @galal-hussein we should also not expect that any non-posix grep options are available, such as -P which is a gnu extension to support PCREs.

@bguzman-3pillar
Copy link

Validated on v1.26.4-rc1+k3s1

$ k3s -v
k3s version v1.26.4-rc1+k3s1 (257fa2c5)
go version go1.19.8

Environment Details

Infrastructure

  • Cloud
  • Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.1 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.1 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

Cluster Configuration:

1 server

Config.yaml:

selinux: true

Testing Steps

  1. Create Airgapped environment
  2. Get all the relevant files from bastion:
- Install script: wget -O k3s_install.sh [get.k3s.io](http://get.k3s.io/)
- Version used: wget https://github.com/k3s-io/k3s/releases/download/v1.26.4-rc1%2Bk3s1/k3s-airgap-images-amd64.tar.gz
- K3S binary: wget https://github.com/k3s-io/k3s/releases/download/v1.26.4-rc1%2Bk3s1/k3s
  1. Copy all of that to airgapped node
  2. Run this (Steps here https://docs.k3s.io/installation/airgap#installing-k3s-in-an-air-gapped-environment)
sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp ./k3s-airgap-images-amd64.tar.gz /var/lib/rancher/k3s/agent/images/
  1. Copy the config.yaml file
sudo mkdir -p /etc/rancher/k3s
sudo cp config.yaml /etc/rancher/k3s/config.yaml
  1. Install k3s using this INSTALL_K3S_SKIP_DOWNLOAD=true ./k3s_install.sh
  2. Check everything is up and running kubectl get node -A -o wide

Validation Results:

$ cat /etc/rancher/k3s/config.yaml
selinux: true
$ INSTALL_K3S_SKIP_DOWNLOAD=true ./k3s_install.sh
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
$ kubectl get node,all -A 
NAME                                               STATUS   ROLES                  AGE   VERSION
node/ip-172-31-34-209.us-east-2.compute.internal   Ready    control-plane,master   20m   v1.26.4-rc1+k3s1

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-59b4f5bbd5-c9nhj                  1/1     Running     0          20m
kube-system   pod/local-path-provisioner-76d776f6f9-qf9tb   1/1     Running     0          20m
kube-system   pod/helm-install-traefik-crd-tfsq2            0/1     Completed   0          20m
kube-system   pod/svclb-traefik-0ef1a7a1-fk2pq              2/2     Running     0          20m
kube-system   pod/traefik-56b8c5fb5c-8bb4z                  1/1     Running     0          20m
kube-system   pod/helm-install-traefik-sc8sz                0/1     Completed   1          20m
kube-system   pod/metrics-server-7b67f64457-hnkjp           1/1     Running     0          20m

NAMESPACE     NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default       service/kubernetes       ClusterIP      10.43.0.1      <none>          443/TCP                      20m
kube-system   service/kube-dns         ClusterIP      10.43.0.10     <none>          53/UDP,53/TCP,9153/TCP       20m
kube-system   service/metrics-server   ClusterIP      10.43.13.87    <none>          443/TCP                      20m
kube-system   service/traefik          LoadBalancer   10.43.97.234   172.31.34.209   80:31066/TCP,443:30999/TCP   20m

NAMESPACE     NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik-0ef1a7a1   1         1         1       1            1           <none>          20m

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns                  1/1     1            1           20m
kube-system   deployment.apps/local-path-provisioner   1/1     1            1           20m
kube-system   deployment.apps/traefik                  1/1     1            1           20m
kube-system   deployment.apps/metrics-server           1/1     1            1           20m

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-59b4f5bbd5                  1         1         1       20m
kube-system   replicaset.apps/local-path-provisioner-76d776f6f9   1         1         1       20m
kube-system   replicaset.apps/traefik-56b8c5fb5c                  1         1         1       20m
kube-system   replicaset.apps/metrics-server-7b67f64457           1         1         1       20m

NAMESPACE     NAME                                 COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik-crd   1/1           18s        20m
kube-system   job.batch/helm-install-traefik       1/1           21s        20m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/airgap kind/bug Something isn't working
Projects
Archived in project
Development

No branches or pull requests

9 participants