Cannot install on k3s

[coopstah13]

Bug Report

What did you do?
Installed Operator Lifecycle Manager on k3s.

What did you expect to see?
Successful installation.

What did you see instead? Under which circumstances?
Does not install. Not all pods start.

NAME                                READY   STATUS             RESTARTS   AGE
catalog-operator-7b788c597d-jkb52   1/1     Running            0          2d
olm-operator-946bd977f-cjh45        0/1     CrashLoopBackOff   415        2d

Environment

• operator-lifecycle-manager version:

0.13.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3-k3s.2", GitCommit:"e7e6a3c4e9a7d80b87793612730d10a863a25980", GitTreeState:"clean", BuildDate:"2019-11-18T18:31:23Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3-k3s.2", GitCommit:"e7e6a3c4e9a7d80b87793612730d10a863a25980", GitTreeState:"clean", BuildDate:"2019-11-18T18:31:23Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

k3s version v1.0.0 (18bd921c)

Possible Solution

Additional context
Pod logs from crashing pod

time="2019-11-22T17:06:04Z" level=info msg="log level info"
time="2019-11-22T17:06:04Z" level=info msg="TLS keys not set, using non-https for metrics"
time="2019-11-22T17:06:04Z" level=info msg="Using in-cluster kube client config"
time="2019-11-22T17:06:04Z" level=info msg="Using in-cluster kube client config"
time="2019-11-22T17:06:06Z" level=fatal msg="couldn't clean previous release" error="Delete https://10.43.0.1:443/apis/operators.coreos.com/v1alpha1/namespaces/olm/catalogsources/olm-operators: dial tcp 10.43.0.1:443: connect: no route to host"

[coopstah13]

Link to issue I opened with k3s: k3s-io/k3s#765

[njhale]

Hi @coopstah13! Thanks for bringing this over.

From your logs, it looks like there's some trouble connecting to the kube apiserver, which is causing olm-operator's bootstrapping to fail. While olm-operator probably shouldn't crash outright, based on the number of restarts, I would guess the root cause isn't transient networking issues. Do you know of any other kube controllers that are able to connect to the apiserver on this cluster?

Also, could you tail the logs for the catalog-operator deployment? It would help to know if it's having trouble too. In general, the more info that you can get us (deployments, pods, nodes, etc...), the better we can triage the problem.

[coopstah13]

yeah that has the same error

time="2019-11-25T02:19:26Z" level=info msg="log level info"
time="2019-11-25T02:19:26Z" level=info msg="TLS keys not set, using non-https for metrics"
time="2019-11-25T02:19:26Z" level=info msg="Using in-cluster kube client config"
time="2019-11-25T02:19:26Z" level=info msg="Using in-cluster kube client config"
time="2019-11-25T02:19:26Z" level=info msg="Using in-cluster kube client config"
time="2019-11-25T02:19:27Z" level=info msg="operator not ready: communicating with server failed: Get https://10.43.0.1:443/version?timeout=32s: dial tcp 10.43.0.1:443: connect: no route to host"

I have nothing else deployed on this, it is just a clean installation of k3s and me trying to get OLM on it.

These are my nodes

NAME                   STATUS   ROLES    AGE     VERSION
oc3447428006.xxx.com   Ready    master   5d23h   v1.16.3-k3s.2

These are my services

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   5d23h

There are no pods or deployments.

[njhale]

Yeah, I suspect there is some network or cluster configuration issue preventing OLM's controllers from communicating with the apiserver. I'm not sure if k3s explicitly disallows pods from connecting to the apiserver or if there's some configuration needed to enable it, but it doesn't look like there's anything wrong on OLM's side (besides bad error handling in olm-operator). To confirm this is the case, you could find another custom controller (something built using client-go) and see if it has the same problem.

Aside from that, would ask the k3s folks about what their policies are around control plane access for workloads.

[exdx]

We would like to support k3s, if we can, and are reviewing currently @coopstah13

[clyang82]

I just tried to install olm on k3s. it works well. Here are my steps:

1. follow the document https://k3s.io/ to install on ubuntu
   Machine info

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic

 curl -sfL https://get.k3s.io | sh -
[INFO]  Finding latest release
[INFO]  Using v1.0.0 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.0.0/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.0.0/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
root@cyang21:~# k3s kubectl get nodes
NAME                   STATUS   ROLES    AGE   VERSION
cyang21.com   Ready    master   12s   v1.16.3-k3s.2

2. install olm

curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.11.0/install.sh | bash -s 0.11.0
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
namespace/olm created
namespace/operators created
clusterrole.rbac.authorization.k8s.io/system:controller:operator-lifecycle-manager created
serviceaccount/olm-operator-serviceaccount created
clusterrolebinding.rbac.authorization.k8s.io/olm-operator-binding-olm created
deployment.apps/olm-operator created
deployment.apps/catalog-operator created
clusterrole.rbac.authorization.k8s.io/aggregate-olm-edit created
clusterrole.rbac.authorization.k8s.io/aggregate-olm-view created
operatorgroup.operators.coreos.com/global-operators created
operatorgroup.operators.coreos.com/olm-operators created
clusterserviceversion.operators.coreos.com/packageserver created
catalogsource.operators.coreos.com/operatorhubio-catalog created
Waiting for deployment "olm-operator" rollout to finish: 0 of 1 updated replicas are available...
deployment "olm-operator" successfully rolled out
Waiting for deployment "catalog-operator" rollout to finish: 0 of 1 updated replicas are available...
deployment "catalog-operator" successfully rolled out
Package server phase: Installing
Package server phase: Succeeded
deployment "packageserver" successfully rolled out

check the status of olm-operator

root@cyang21:~# k3s kubectl get pod -n olm
NAME                                READY   STATUS    RESTARTS   AGE
olm-operator-7cb7d95799-9rf6w       1/1     Running   0          65s
catalog-operator-776d59f97d-dk2f5   1/1     Running   0          65s
packageserver-6bbbfc846c-77t9j      1/1     Running   0          47s
operatorhubio-catalog-dlvlx         1/1     Running   0          50s
packageserver-6bbbfc846c-qn2lb      1/1     Running   0          41s

3. run an example to ensure olm is running normally. follow this document - https://github.com/operator-framework/operator-sdk-samples/tree/master/go/memcached-operator
   results is

root@cyang21:~/go/src/github.com/operator-framework/operator-sdk-samples/go/memcached-operator# k3s kubectl get pod -n memcached
NAME                                  READY   STATUS    RESTARTS   AGE
memcached-operator-84cd957484-wnhsc   1/1     Running   0          75s
example-memcached-7c4df9b7b4-t5fkq    1/1     Running   0          60s
example-memcached-7c4df9b7b4-b2hs2    1/1     Running   0          60s
example-memcached-7c4df9b7b4-4g6mm    1/1     Running   0          60s

[coopstah13]

I am not using ubuntu, so it could be related.
I am using RHEL 7.7.
These are the exact steps I ran and it does not come up on RHEL.

[clyang82]

I have tried on RHEL 7.7 with above steps. It works well. Maybe the issue was fixed in k3s side. You can have a try with latest k3s to see if it is working or not.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.