Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for deploying k3s #682

Merged
merged 6 commits into from
Mar 14, 2023
Merged

Add support for deploying k3s #682

merged 6 commits into from
Mar 14, 2023

Conversation

tserong
Copy link
Member

@tserong tserong commented Mar 3, 2023

This does a couple of things:

  1. Lets us use sesdev to deploy k3s, just for the sake of having an easy way to create little VM clusters running k3s (think: arbitrary k3s dev/test playground)
  2. Adds the ability to deploy rook on top of k3s, for the sake of testing rook/ceph for ongoing SES maintenance.

@tserong
Copy link
Member Author

tserong commented Mar 3, 2023

AFAICT those remaining lint failures aren't actually in code I touched in this PR.

trociny
trociny reviewed Mar 3, 2023
View reviewed changes
Copy link
Contributor

@trociny trociny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sesdev create k3s --deploy-ses failed for me with:

    master: ++ curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
    master: In order to verify checksum, openssl must first be installed.
    master: Please install openssl or set VERIFY_CHECKSUM=false in your environment.
    master: Failed to install helm

After I added export VERIFY_CHECKSUM=false into seslib/templates/k3s/provision.sh.j2 the deploy succeeded and the cluster is working.

Great job! Thanks!

@trociny
Copy link
Contributor

trociny commented Mar 3, 2023

Just a note. It might be useful to have the toolbox pod installed during deploy or have a hint message at the end of the deployment :

kubectl create -f ~/cluster/rook-ceph/examples/toolbox.yaml

@kshtsk
Copy link
Contributor

kshtsk commented Mar 3, 2023

AFAICT those remaining lint failures aren't actually in code I touched in this PR.

even though they must be fixed

@kshtsk
Copy link
Contributor

kshtsk commented Mar 3, 2023

@tserong btw, the deployment test is still failing how about to fix the repo issues or/and switch to the newer ses version in self test.

@tserong
Copy link
Member Author

tserong commented Mar 6, 2023 •
edited
Loading

@trociny I've updated this to explicitly install openssl to fix helm's VERIFY_CHECKSUM thing. I've also added toolbox deployment, because I agree that seems pretty useful. I've done that as separate commits so it's easy to see changes, but can squash later.

@kshtsk I've cherry picked your commits from #683 here to hopefully get a clean jenkins and tox run, but I'm happy to rebase this if you want to merge your PR before merging this one.

@kshtsk
Copy link
Contributor

kshtsk commented Mar 6, 2023

@tserong I gonna try this PR pretty soon

@kshtsk
Copy link
Contributor

kshtsk commented Mar 6, 2023

Looks interesting, could you please rebase and merge.

tserong added 5 commits March 7, 2023 17:37
@tserong
Add support for deploying k3s
9ddf6e2 
This adds `sesdev create k3s`, which by default will deploy a
master with four workers on openSUSE Tumbleweed.  It will also
deploy on SLE 15 SP3 if you specify `--os=sles-15-sp3`.

k3s is installed by running `curl https://get.k3s.io`.  Helm
is also installed on the master, again using `curl`.  If you
don't want the latest stable k3s, you can install a specific
version with the `--k3s-version` option.  If you want more or
less workers, use `--roles='[master],[worker],[...]`.

Signed-off-by: Tim Serong <tserong@suse.com>
@tserong
Add ability to deploy Ceph via Rook on k3s
1ede85a 
This adds the `--deploy-ses` option to `sesdev create k3s`,
which will pull registry.suse.com/ses/7.1/charts/rook-ceph,
then deploy Rook, which in turn will deploy Ceph, and have
it eat all the disks on all the workers.

Signed-off-by: Tim Serong <tserong@suse.com>
@tserong
README.md: Add k3s notes
5fdffe5 
Signed-off-by: Tim Serong <tserong@suse.com>
@tserong
k3s: install openssl for helm checksum verification
a3750a8 
When deploying on SLE 15, openssl is already installed as
it's a dependency of ca-certificates.  This is not true on
Tumblweed though, so let's just always install it.

Signed-off-by: Tim Serong <tserong@suse.com>
@tserong
k3s: deploy ceph toolbox container
96b696a 
Signed-off-by: Tim Serong <tserong@suse.com>
@tserong
Copy link
Member Author

tserong commented Mar 7, 2023

Rebased

@votdev votdev self-requested a review March 7, 2023 07:03
@kshtsk
Copy link
Contributor

kshtsk commented Mar 7, 2023 

    master: ++ ceph status
    master: 2023-03-07T10:11:23.938+0000 7faa9004e700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
    master: 2023-03-07T10:11:23.938+0000 7faa9004e700 -1 AuthRegistry(0x7faa8805f158) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
    master: 2023-03-07T10:11:24.046+0000 7faa9004e700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
    master: 2023-03-07T10:11:24.046+0000 7faa9004e700 -1 AuthRegistry(0x7faa9004cf50) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
    master: 2023-03-07T10:11:24.246+0000 7faa8ddea700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1]
    master: 2023-03-07T10:11:24.250+0000 7faa9004e700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
    master: [errno 13] RADOS permission denied (error connecting to the cluster)
    master: +++ err_report 638
    master: +++ local hn
    master: +++ set +x
    master: Error in provisioner script trapped!

@kshtsk
Copy link
Contributor

kshtsk commented Mar 7, 2023

Just wonder why it wasn't failing before rebase, and btw, it is not reproducible on master now.

@kshtsk kshtsk self-requested a review March 7, 2023 10:31
kshtsk
kshtsk reviewed Mar 7, 2023
View reviewed changes
@tserong
k3s: add k3s_deploy_ses setting
be0bfab 
This avoids causing confusion by overloading the caasp_deploy_ses setting.

Signed-off-by: Tim Serong <tserong@suse.com>
@kshtsk
Copy link
Contributor

kshtsk commented Mar 13, 2023

so, gonna merge this?

@tserong tserong merged commit 1cf4353 into SUSE:master Mar 14, 2023
@tserong tserong deleted the wip-k3s branch March 14, 2023 02:04
@tserong
Copy link
Member Author

tserong commented Mar 14, 2023

so, gonna merge this?

Yep :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@trociny trociny trociny left review comments

@kshtsk kshtsk kshtsk approved these changes

@shaas shaas Awaiting requested review from shaas

@mgfritch mgfritch Awaiting requested review from mgfritch

@m-ildefons m-ildefons Awaiting requested review from m-ildefons

@votdev votdev Awaiting requested review from votdev

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tserong @trociny @kshtsk