Issue when installing Istio for Knative #90

duglin · 2019-02-28T13:49:00Z

Describe the bug
When installing Istio as part of Knative, I see error like this show up in the k3s console:

ERRO[2019-02-27T07:51:03.743701454-08:00] ServiceController istio-system/istio-ingressgateway [svccontroller-change] failed with : failed to create istio-system/svclb-istio-ingressgateway apps/v1, Kind=Deployment for svccontroller istio-system/istio-ingressgateway: Deployment.apps "svclb-istio-ingressgateway" is invalid: [spec.template.spec.containers[3].ports[0].name: Invalid value: "tcp-pilot-grpc-tls": must be no more than 15 characters, spec.template.spec.containers[4].ports[0].name: Invalid value: "tcp-citadel-grpc-tls": must be no more than 15 characters, spec.template.spec.containers[6].ports[0].name: Invalid value: "http2-prometheus": must be no more than 15 characters]

I can't seem to reproduce this on my other K8s type of clusters (IKS and hack/local-up-cluster.sh). Does K3s introduce some new restrictions in this space?

To Reproduce
Steps to reproduce the behavior:

run K3s
install Istio: https://github.com/knative/docs/blob/master/install/Knative-with-IKS.md#installing-istio
step 1

Expected behavior
No errors on the K3s console

The text was updated successfully, but these errors were encountered:

ibuildthecloud · 2019-02-28T14:21:48Z

@duglin I believe this is just standard k8s behavior: https://github.com/kubernetes/kubernetes/blob/6902f3112d98eb6bd0894886ff9cd3fbd03a7f79/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go#L253

I'll do some more research, I have for sure ran Istio on k3s before.

duglin · 2019-02-28T15:19:28Z

yeah, I'm trying to understand why I don't see this on my other clusters when it seems like I should. I don't see any obvious flag to disable this check either.... very odd

southwolf · 2019-03-05T01:40:42Z

@duglin Have you tried this
https://github.com/knative/docs/blob/master/install/Knative-with-Minikube.md

I believe minikube might be similar to k3s. I used this one and managed to launch it.

erikwilson · 2019-03-07T23:39:55Z

I am unable to reproduce the Step 1 failure on 0.1.0 or the newest RC.

The only message I see on Step 1 is:

unable to recognize "https://github.com/knative/serving/releases/download/v0.4.0/istio.yaml": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta1"

And when installing Knative Step 2:

unable to recognize "https://github.com/knative/eventing/releases/download/v0.4.0/release.yaml": no matches for kind "ClusterChannelProvisioner" in version "eventing.knative.dev/v1alpha1"

All of the pods start except a few:

root@erik-k3s-dev:~# kubectl get pods --all-namespaces
NAMESPACE            NAME                                            READY   STATUS             RESTARTS   AGE
istio-system         cluster-local-gateway-547467ccf6-7xbxw          1/1     Running            0          7m58s
istio-system         istio-citadel-7d64db8bcf-vvvc5                  1/1     Running            0          7m59s
istio-system         istio-cleanup-secrets-hfwww                     0/1     Completed          0          8m
istio-system         istio-egressgateway-6ddf4c8bd6-k8l94            1/1     Running            0          8m
istio-system         istio-galley-7dd996474-kdxj2                    1/1     Running            0          8m
istio-system         istio-ingressgateway-84b89d647f-th8bk           1/1     Running            0          8m
istio-system         istio-pilot-86bb4fcbbd-p7s6c                    2/2     Running            0          7m59s
istio-system         istio-policy-5c4d9ff96b-xzkbl                   2/2     Running            0          7m59s
istio-system         istio-sidecar-injector-6977b5cf5b-g85j4         1/1     Running            0          7m59s
istio-system         istio-statsd-prom-bridge-b44b96d7b-fzk8l        1/1     Running            0          8m
istio-system         istio-telemetry-7676df547f-5xmpd                2/2     Running            0          7m59s
istio-system         zipkin-5495cf4cf-jxf2k                          1/1     Running            0          6m54s
knative-build        build-controller-7b8987d675-mvtkv               1/1     Running            0          6m56s
knative-build        build-webhook-74795c8696-564zr                  1/1     Running            0          6m56s
knative-eventing     eventing-controller-864657d8d4-wqnch            1/1     Running            0          6m56s
knative-eventing     in-memory-channel-controller-f794cc9d8-g2qv4    1/1     Running            0          6m55s
knative-eventing     in-memory-channel-dispatcher-8595c7f8d7-wtvjs   1/2     CrashLoopBackOff   6          6m55s
knative-eventing     webhook-5d76776d55-z9wpm                        1/1     Running            0          6m56s
knative-monitoring   elasticsearch-logging-0                         1/1     Running            0          6m55s
knative-monitoring   elasticsearch-logging-1                         1/1     Running            0          5m50s
knative-monitoring   grafana-568674f4f9-p257v                        1/1     Running            0          6m54s
knative-monitoring   kibana-logging-7698db4f94-frdmb                 1/1     Running            0          6m55s
knative-monitoring   kube-state-metrics-58b9779756-bbvsd             4/4     Running            0          5m46s
knative-monitoring   node-exporter-ftfzq                             2/2     Running            0          6m55s
knative-monitoring   prometheus-system-0                             1/1     Running            0          6m54s
knative-monitoring   prometheus-system-1                             1/1     Running            0          6m54s
knative-serving      activator-7c8b59d78-gkhnw                       1/2     CrashLoopBackOff   6          6m57s
knative-serving      autoscaler-666c9bfcc6-5q6z7                     1/2     CrashLoopBackOff   6          6m57s
knative-serving      controller-799cd5c6dc-sr9md                     1/1     Running            0          6m57s
knative-serving      webhook-5b66fdf6b9-ssg84                        1/1     Running            0          6m57s
knative-sources      controller-manager-0                            1/1     Running            0          6m55s
kube-system          coredns-7748f7f6df-m7jm9                        1/1     Running            0          8m42s
kube-system          helm-install-traefik-d8j6d                      0/1     Completed          0          8m42s
kube-system          svclb-traefik-8f7c97697-s5wqd                   2/2     Running            1          8m31s
kube-system          traefik-6876857645-dl88l                        1/1     Running            0          8m31s

root@erik-k3s-dev:~# kubectl logs -n knative-eventing in-memory-channel-dispatcher-8595c7f8d7-wtvjs dispatcher
{"level":"fatal","ts":1552001051.4944558,"caller":"fanoutsidecar/main.go:91","msg":"Unable to create configMap noticer.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:91\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}

root@erik-k3s-dev:~# kubectl logs -n knative-serving activator-7c8b59d78-gkhnw activator
{"level":"info","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.4\" is not a valid GitHub commit ID"}
{"level":"info","ts":"2019-03-07T23:25:49.551Z","logger":"activator","caller":"activator/main.go:131","msg":"Starting the knative activator","knative.dev/controller":"activator"}
{"level":"fatal","ts":"2019-03-07T23:25:51.644Z","logger":"activator","caller":"activator/main.go:147","msg":"Version check failed: Get https://10.43.0.1:443/version?timeout=32s: EOF","knative.dev/controller":"activator","stacktrace":"main.main\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:147\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}

root@erik-k3s-dev:~# kubectl logs -n knative-serving autoscaler-666c9bfcc6-5q6z7 autoscaler
{"level":"info","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.4\" is not a valid GitHub commit ID"}
{"level":"fatal","ts":"2019-03-07T23:25:33.647Z","logger":"autoscaler","caller":"autoscaler/main.go:95","msg":"Version check failed: an error on the server (\"\") has prevented the request from succeeding","stacktrace":"main.main\n\t/go/src/github.com/knative/serving/cmd/autoscaler/main.go:95\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}

root@erik-k3s-dev:~# head -3 /proc/meminfo
MemTotal:       32939856 kB
MemFree:        16705668 kB
MemAvailable:   23599904 kB

This is on a DigitalOcean Ubuntu 18.04 droplet. Could you provide more information on your setup @duglin?

duglin · 2019-03-08T01:54:55Z

just rerun the kubectl cmd again when you see unable to recognize... errors. It's a timing issue.

I was running this on my macbook pro 13", ubuntu 15.10 VM, Vmware Fusion.

duglin · 2019-03-08T02:14:56Z

Just tried it again and I see it.
In one window I ran: k3s server
and then in another window I ran:

KUBECONFIG=/etc/rancher/k3s/k3s.yaml  kubectl apply --filename https://github.com/knative/serving/releases/download/v0.4.0/istio-crds.yaml

And that ran ok, no errors/issues. Then I ran:

KUBECONFIG=/etc/rancher/k3s/k3s.yaml  kubectl apply --filename https://github.com/knative/serving/releases/download/v0.4.0/istio.yaml

and I see the errors in the first window - the k3s server output.

erikwilson · 2019-03-12T23:51:51Z

Thanks for the info, I am able to reproduce @duglin.

The unable to recognize... errors are probably caused by us removing autoscaling/v2beta1 features in 16bd458

Are you able to verify that istio-system/svclb-istio-ingressgateway is successfully created on other k8s systems? I tried on an RKE system and kubectl get pods --all-namespaces | grep svclb produced no results, the pod is never created.

duglin · 2019-03-13T00:34:40Z

@erikwilson no I don't see that one, but I do see istio-system/istio-ingressgateway-84b89d647f-xqmkx on k3s and istio-system/istio-ingressgateway-84b89d647f-ggbdr on my IKS cluster. Where do you see the svclb.... one in the yamls? I don't see that.

erikwilson · 2019-03-13T00:54:02Z

I don't think it is in the yaml, from the log it appears that istio-system/istio-ingressgateway is trying to create istio-system/svclb-istio-ingressgateway but it is unable to do so. I think k3s is making the error more visible here, but the same thing is happening on other k8s clusters.

erikwilson · 2019-03-13T04:42:54Z

Sorry, we are actually creating something with the svclb- prefix: https://github.com/rancher/k3s/blob/master/pkg/servicelb/controller.go#L213

But to get a better idea on what is happening with the load balancer you can use a command like kubectl get services -n istio-system istio-ingressgateway, both k3s and RKE show as pending for external-ip, is this working in IKS?

duglin · 2019-03-13T12:20:52Z

I do see that it is "pending" for K3s, but for IKS it is not, it has an IP assigned. Could it be that K3s, while creating the LB/external-IP is creating a new Kube resource and that name is too long?

If a port name is longer than 15 characters we are unable to create the associated service load balancer containers. Use our own short name of `lb-port-{port}` to avoid naming issues. For k3s-io/issues/90

erikwilson · 2019-03-14T19:28:39Z

@duglin it is related to the naming of the ports, I have a PR in to use shorter port names.

Istio will try to use the same ports as the traefik load balancer, so I removed the traefik service like kubectl delete service -n kube-system traefik, and now with that PR I see the Istio lb come up:

erik@lubuntu:~/go/src/github.com/rancher/k3s$ kubectl get services -n istio-system istio-ingressgateway
NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                                                                                                                   AGE
istio-ingressgateway   LoadBalancer   10.43.74.55   10.0.2.15     80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31737/TCP,8060:30679/TCP,853:30732/TCP,15030:31027/TCP,15031:30080/TCP   25m
erik@lubuntu:~/go/src/github.com/rancher/k3s$ kubectl get pods --all-namespaces | grep svclb
istio-system   svclb-istio-ingressgateway-69ff8bc4dc-v8dxs   8/8     Running     0          25m

There may still be some issues with the missing autoscaling/v2beta1 HorizontalPodAutoscaler tho.

erikwilson · 2019-03-15T05:38:11Z

It is easy to update istio.yaml to use autoscaling/v2beta2, still seeing the CrashLoopBackOff errors with Knative tho.

erikwilson · 2019-03-22T22:41:51Z

The load balancer issue related to istio has been fixed, but there is probably more stuff to figure out. Perhaps it is best to open new issues for Knative or other istio stuff so we can close out this port naming problem.

duglin · 2019-03-26T23:02:58Z

Getting closer... now I get this:

unable to recognize "https://github.com/knative/serving/releases/download/v0.4.0/istio-lean.yaml": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta1"

I believe it's because K3s has apiversions:

autoscaling/v1
autoscaling/v2beta2

but not autoscaling/v2beta1. I don't know enough about that area of the code to know if it's ok to not have the version or if people use it a lot... but FYI since Istio seems to use it, and therefore Knative.

erikwilson · 2019-03-26T23:24:05Z

We try to remove old or obsolete API versions, so autoscaling/v2beta1 was dropped. The difference in v2beta2 is pretty small for this application tho, so it is possible to update istio yaml to use v2beta2 config.

duglin · 2019-03-27T02:17:56Z

The problem is that's under Istio's control. @linsun do you know if Istio can (or will) upgrade to v2beta2 any time soon?

erikwilson · 2019-03-29T20:58:17Z

Thank you for reporting this @duglin, I am closing out this issue for the service load balancer port name fix and have opened up a few more regarding autoscaling/v2beta2 and knative errors.

puneetmathur-87 · 2019-04-17T07:05:37Z

How this was resolved ? I am also facing the same for the activator and dispatcher.

erikwilson · 2019-04-17T07:27:46Z

There are still issues with install Knative that are being tracked at /issues/286, @puneetmathur-87. Although those errors are for Knative v0.4.0 there are still issues with v0.5.0 that may be related to permission changes in Kubernetes v1.14, the changelog provides some potential work-arounds that I have not tried yet.

ibuildthecloud added this to Backlog in K3S Development Feb 28, 2019

erikwilson mentioned this issue Mar 14, 2019

Short port names for service load balancer #232

Merged

erikwilson moved this from Backlog to Needs review in K3S Development Mar 14, 2019

erikwilson added the status/ready-for-review label Mar 15, 2019

ghost mentioned this issue Mar 17, 2019

k3s support projectriff/riff#1171

Closed

erikwilson added the status/to-test label Mar 22, 2019

ibuildthecloud added this to the v0.3.0 milestone Mar 25, 2019

erikwilson moved this from Needs review to Testing in K3S Development Mar 25, 2019

This was referenced Mar 27, 2019

Upgrade to autoscaling/v2beta2 istio/istio#12823

Closed

Upgrade istio to use autoscaling/v2beta2 #285

Closed

Resolve errors with installing Knative #286

Closed

erikwilson closed this as completed Mar 29, 2019

K3S Development automation moved this from Testing to Done Mar 29, 2019

erikwilson added the kind/bug Something isn't working label Mar 29, 2019

ohthehugemanatee mentioned this issue Jul 29, 2021

Agent fails to update lease, 127.0.0.1:6444 timeout and "object has been modified" #3726

Closed

1 task

samip5 mentioned this issue Aug 10, 2021

Unable to create pods on k3s-agent running on Gentoo (openrc, arm64) #3815

Closed

IceeMC mentioned this issue Dec 21, 2021

[BUG] K3S fails to authenticate completely valid service account tokens #4820

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue when installing Istio for Knative #90

Issue when installing Istio for Knative #90

duglin commented Feb 28, 2019

ibuildthecloud commented Feb 28, 2019

duglin commented Feb 28, 2019

southwolf commented Mar 5, 2019

erikwilson commented Mar 7, 2019

duglin commented Mar 8, 2019

duglin commented Mar 8, 2019 •

edited

erikwilson commented Mar 12, 2019

duglin commented Mar 13, 2019

erikwilson commented Mar 13, 2019

erikwilson commented Mar 13, 2019

duglin commented Mar 13, 2019

erikwilson commented Mar 14, 2019

erikwilson commented Mar 15, 2019

erikwilson commented Mar 22, 2019

duglin commented Mar 26, 2019

erikwilson commented Mar 26, 2019

duglin commented Mar 27, 2019

erikwilson commented Mar 29, 2019

puneetmathur-87 commented Apr 17, 2019

erikwilson commented Apr 17, 2019

Issue when installing Istio for Knative #90

Issue when installing Istio for Knative #90

Comments

duglin commented Feb 28, 2019

ibuildthecloud commented Feb 28, 2019

duglin commented Feb 28, 2019

southwolf commented Mar 5, 2019

erikwilson commented Mar 7, 2019

duglin commented Mar 8, 2019

duglin commented Mar 8, 2019 • edited

erikwilson commented Mar 12, 2019

duglin commented Mar 13, 2019

erikwilson commented Mar 13, 2019

erikwilson commented Mar 13, 2019

duglin commented Mar 13, 2019

erikwilson commented Mar 14, 2019

erikwilson commented Mar 15, 2019

erikwilson commented Mar 22, 2019

duglin commented Mar 26, 2019

erikwilson commented Mar 26, 2019

duglin commented Mar 27, 2019

erikwilson commented Mar 29, 2019

puneetmathur-87 commented Apr 17, 2019

erikwilson commented Apr 17, 2019

duglin commented Mar 8, 2019 •

edited