Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net-kourier-controller component liveliness probe Failing #840

Closed
junydania opened this issue May 10, 2022 · 9 comments
Closed

net-kourier-controller component liveliness probe Failing #840

junydania opened this issue May 10, 2022 · 9 comments
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@junydania
Copy link

Having an issue with net-kourier-knative liveliness probe failing. Describe the pod gives this data

  Normal   Started    65s   kubelet            Started container controller
  Warning  Unhealthy  64s   kubelet            Readiness probe failed: 2022/05/10 14:25:56 failed to connect to service at ":18000": context deadline exceeded
  Warning  Unhealthy  56s   kubelet            Readiness probe failed: 2022/05/10 14:26:04 failed to connect to service at ":18000": context deadline exceeded
  Warning  Unhealthy  46s   kubelet            Readiness probe failed: 2022/05/10 14:26:14 failed to connect to service at ":18000": context deadline exceeded
  Warning  Unhealthy  36s   kubelet            Readiness probe failed: 2022/05/10 14:26:24 failed to connect to service at ":18000": context deadline exceeded
  Warning  Unhealthy  26s   kubelet            Readiness probe failed: 2022/05/10 14:26:34 failed to connect to service at ":18000": context deadline exceeded
  Warning  Unhealthy  16s   kubelet            Readiness probe failed: 2022/05/10 14:26:44 failed to connect to service at ":18000": context deadline exceeded
  Warning  Unhealthy  6s    kubelet            Readiness probe failed: 2022/05/10 14:26:54 failed to connect to service at ":18000": context deadline exceeded

logs from the pod

{"severity":"INFO","timestamp":"2022-05-10T14:25:56.629935147Z","caller":"logging/config.go:116","message":"Successfully created the logger."}
{"severity":"INFO","timestamp":"2022-05-10T14:25:56.629991527Z","caller":"logging/config.go:117","message":"Logging level set to: info"}
{"severity":"INFO","timestamp":"2022-05-10T14:25:56.630065312Z","caller":"logging/config.go:79","message":"Fetch GitHub commit ID from kodata failed","error":"open /var/run/ko/HEAD: no such file or directory"}
{"severity":"INFO","timestamp":"2022-05-10T14:25:56.630091394Z","logger":"net-kourier-controller","caller":"profiling/server.go:64","message":"Profiling enabled: false"}
{"severity":"INFO","timestamp":"2022-05-10T14:25:56.634867894Z","logger":"net-kourier-controller","caller":"leaderelection/context.go:46","message":"Running with Standard leader election"}
{"severity":"INFO","timestamp":"2022-05-10T14:25:57.7937142Z","logger":"net-kourier-controller","caller":"ingress/controller.go:198","message":"Priming the config with 1402 ingresses"}
@nak3
Copy link
Contributor

nak3 commented May 11, 2022

Could you please try to get the goroutine dump?

1. Enable debug log and profile options.

$ kubectl patch cm config-logging  -n knative-serving -p '{"data":{"'loglevel.net-kourier-controller'":"'debug'"}}'
$ kubectl patch cm config-observability  -n knative-serving -p '{"data":{"'profiling.enable'":"'true'"}}'

You should see the following logs after eanbling.

$ kubectl   -n knative-serving  logs net-kourier-controller-bc494674f-sn5rt | grep -E "level|Profiling" | tail -2
{"severity":"INFO","timestamp":"2022-05-11T13:18:09.589770154Z","logger":"net-kourier-controller","caller":"logging/config.go:226","message":"Updating logging level for net-kourier-controller from info to debug."}
{"severity":"INFO","timestamp":"2022-05-11T13:18:42.820313549Z","logger":"net-kourier-controller","caller":"profiling/server.go:102","message":"Profiling enabled: true"}

2. (After the controller hung up) port forward & dump 3 times.

$ kubectl port-forward -n knative-serving deploy/net-kourier-controller 8008

$ curl http://127.0.0.1:8008/debug/pprof/goroutine?debug=1

(wait for 5 sec)

$ curl http://127.0.0.1:8008/debug/pprof/goroutine?debug=1

(wait for 5 sec)

$ curl http://127.0.0.1:8008/debug/pprof/goroutine?debug=1

After that, please share the output of curl commands above and pod logs of net-kourier-controller.
Also, if you can login to the Kubernetes Node where net-kourier-controller is running, please check the CPU usage on the host.

@junydania
Copy link
Author

@nak3 see data below generated

goroutine profile: total 891
360 @ 0x4384b6 0x430bd3 0x463369 0x4cc1f2 0x4cd55a 0x4cd548 0x52c329 0x53c6c5 0x7a1bae 0x5b4103 0x5b425d 0x7a296c 0x468c81
#   0x463368    internal/poll.runtime_pollWait+0x88     runtime/netpoll.go:234
#   0x4cc1f1    internal/poll.(*pollDesc).wait+0x31     internal/poll/fd_poll_runtime.go:84
#   0x4cd559    internal/poll.(*pollDesc).waitRead+0x259    internal/poll/fd_poll_runtime.go:89
#   0x4cd547    internal/poll.(*FD).Read+0x247          internal/poll/fd_unix.go:167
#   0x52c328    net.(*netFD).Read+0x28              net/fd_posix.go:56
#   0x53c6c4    net.(*conn).Read+0x44               net/net.go:183
#   0x7a1bad    net/http.(*persistConn).Read+0x4d       net/http/transport.go:1926
#   0x5b4102    bufio.(*Reader).fill+0x102          bufio/bufio.go:101
#   0x5b425c    bufio.(*Reader).Peek+0x5c           bufio/bufio.go:139
#   0x7a296b    net/http.(*persistConn).readLoop+0x1ab      net/http/transport.go:2087
360 @ 0x4384b6 0x448252 0x7a463b 0x468c81
#   0x7a463a    net/http.(*persistConn).writeLoop+0xfa  net/http/transport.go:2386
30 @ 0x4384b6 0x40666c 0x406098 0x17c2aa5 0x468c81
#   0x17c2aa4   knative.dev/networking/pkg/status.(*Prober).IsReady.func4+0x44  knative.dev/networking@v0.0.0-20220120043934-ec785540a732/pkg/status/status.go:258
30 @ 0x4384b6 0x448252 0x17c2b76 0x468c81
#   0x17c2b75   knative.dev/networking/pkg/status.(*Prober).IsReady.func3+0x95  knative.dev/networking@v0.0.0-20220120043934-ec785540a732/pkg/status/status.go:245
15 @ 0x4384b6 0x464d3d 0x464d1d 0x470d2c 0xedb75e 0x17c39d5 0x17c3408 0x468c81
#   0x464d1c    sync.runtime_notifyListWait+0x11c                   runtime/sema.go:513
#   0x470d2b    sync.(*Cond).Wait+0x8b                          sync/cond.go:56
#   0xedb75d    k8s.io/client-go/util/workqueue.(*Type).Get+0x9d            k8s.io/client-go@v0.22.5/util/workqueue/queue.go:151
#   0x17c39d4   knative.dev/networking/pkg/status.(*Prober).processWorkItem+0x54    knative.dev/networking@v0.0.0-20220120043934-ec785540a732/pkg/status/status.go:347
#   0x17c3407   knative.dev/networking/pkg/status.(*Prober).Start.func1+0x67        knative.dev/networking@v0.0.0-20220120043934-ec785540a732/pkg/status/status.go:288
7 @ 0x4384b6 0x40666c 0x4060d8 0xf3d9a9 0x8bdc07 0x8bda96 0x8bd989 0xf3d92b 0xf3d8d5 0x8bd7fa 0x468c81
#   0xf3d9a8    k8s.io/client-go/tools/cache.(*processorListener).run.func1+0x48    k8s.io/client-go@v0.22.5/tools/cache/shared_informer.go:772
#   0x8bdc06    k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66       k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:155
#   0x8bda95    k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5         k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:156
#   0x8bd988    k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88          k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:133
#   0xf3d92a    k8s.io/apimachinery/pkg/util/wait.Until+0x6a                k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:90
#   0xf3d8d4    k8s.io/client-go/tools/cache.(*processorListener).run+0x14      k8s.io/client-go@v0.22.5/tools/cache/shared_informer.go:771
#   0x8bd7f9    k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59     k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:73
7 @ 0x4384b6 0x448252 0xf3d676 0x8bd7fa 0x468c81
#   0xf3d675    k8s.io/client-go/tools/cache.(*processorListener).pop+0x155 k8s.io/client-go@v0.22.5/tools/cache/shared_informer.go:742
#   0x8bd7f9    k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59 k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:73
6 @ 0x4384b6 0x40666c 0x406098 0xf2f1e8 0x468c81
#   0xf2f1e7    k8s.io/client-go/tools/cache.(*controller).Run.func1+0x27   k8s.io/client-go@v0.22.5/tools/cache/controller.go:130
6 @ 0x4384b6 0x40666c 0x406098 0xf3cac5 0x8bd582 0x8bd7fa 0x468c81
#   0xf3cac4    k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x44        k8s.io/client-go@v0.22.5/tools/cache/shared_informer.go:628
#   0x8bd581    k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21  k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:56
#   0x8bd7f9    k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59     k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:73
6 @ 0x4384b6 0x448252 0x7cf6c8 0x7cebde 0x468c81
#   0x7cf6c7    golang.org/x/net/http2.(*clientStream).writeRequest+0xaa7   golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/transport.go:1323
#   0x7cebdd    golang.org/x/net/http2.(*clientStream).doRequest+0x1d       golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/transport.go:1185
6 @ 0x4384b6 0x448252 0xf36e6d 0x468c81
#   0xf36e6c    k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2+0x12c  k8s.io/client-go@v0.22.5/tools/cache/reflector.go:373
6 @ 0x4384b6 0x448252 0xf38336 0xf36776 0xf35fc6 0x8bdc07 0x8bda96 0xf35df8 0x8bd582 0x8bd7fa 0x468c81
#   0xf38335    k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x1b5        k8s.io/client-go@v0.22.5/tools/cache/reflector.go:468
#   0xf36775    k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0x695        k8s.io/client-go@v0.22.5/tools/cache/reflector.go:428
#   0xf35fc5    k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x25        k8s.io/client-go@v0.22.5/tools/cache/reflector.go:221
#   0x8bdc06    k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66       k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:155
#   0x8bda95    k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5         k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:156
#   0xf35df7    k8s.io/client-go/tools/cache.(*Reflector).Run+0x1f7         k8s.io/client-go@v0.22.5/tools/cache/reflector.go:220
#   0x8bd581    k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21  k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:56
#   0x8bd7f9    k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59     k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:73
6 @ 0x4384b6 0x464d3d 0x464d1d 0x470d2c 0x7c866b 0x7d5de5 0x512e3f 0x512a5b 0x5126b8 0xef8fbc 0xee8107 0xee84ef 0xa222dc 0x468c81
#   0x464d1c    sync.runtime_notifyListWait+0x11c                       runtime/sema.go:513
#   0x470d2b    sync.(*Cond).Wait+0x8b                              sync/cond.go:56
#   0x7c866a    golang.org/x/net/http2.(*pipe).Read+0xea                    golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/pipe.go:76
#   0x7d5de4    golang.org/x/net/http2.transportResponseBody.Read+0x84              golang.org/x/net@v0.0.0-20211209124913-491a49abca63/http2/transport.go:2384
#   0x512e3e    encoding/json.(*Decoder).refill+0x17e                       encoding/json/stream.go:165
#   0x512a5a    encoding/json.(*Decoder).readValue+0xba                     encoding/json/stream.go:140
#   0x5126b7    encoding/json.(*Decoder).Decode+0x77                        encoding/json/stream.go:63
#   0xef8fbb    k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x19b       k8s.io/apimachinery@v0.22.5/pkg/util/framer/framer.go:152
#   0xee8106    k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0xa6 k8s.io/apimachinery@v0.22.5/pkg/runtime/serializer/streaming/streaming.go:77
#   0xee84ee    k8s.io/client-go/rest/watch.(*Decoder).Decode+0x4e              k8s.io/client-go@v0.22.5/rest/watch/decoder.go:49
#   0xa222db    k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11b            k8s.io/apimachinery@v0.22.5/pkg/watch/streamwatcher.go:105
6 @ 0x4384b6 0x464d3d 0x464d1d 0x470d2c 0xf31dd3 0xf2f476 0x8bdc07 0x8bda96 0x8bd989 0xf2f15b 0xf2f11b 0xf3a929 0x468c81
#   0x464d1c    sync.runtime_notifyListWait+0x11c               runtime/sema.go:513
#   0x470d2b    sync.(*Cond).Wait+0x8b                      sync/cond.go:56
#   0xf31dd2    k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0x1f2     k8s.io/client-go@v0.22.5/tools/cache/delta_fifo.go:525
#   0xf2f475    k8s.io/client-go/tools/cache.(*controller).processLoop+0x35 k8s.io/client-go@v0.22.5/tools/cache/controller.go:183
#   0x8bdc06    k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66   k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:155
#   0x8bda95    k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5     k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:156
#   0x8bd988    k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88      k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:133
#   0xf2f15a    k8s.io/apimachinery/pkg/util/wait.Until+0x2fa           k8s.io/apimachinery@v0.22.5/pkg/util/wait/wait.go:90
#   0xf2f11a    k8s.io/client-go/tools/cache.(*controller).Run+0x2ba        k8s.io/client-go@v0.22.5/tools/cache/controller.go:154
#   0xf3a928    k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x448   k8s.io/client-go@v0.22.5/tools/cache/shared_informer.go:410
4 @ 0x4384b6 0x40666c 0x4060d8 0xedbf27 0x468c81
#   0xedbf26    k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop+0xa6   k8s.io/client-go@v0.22.5/util/workqueue/queue.go:204
3 @ 0x4384b6 0x448252 0xed9dce 0x468c81
#   0xed9dcd    k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x34d   k8s.io/client-go@v0.22.5/util/workqueue/delaying_queue.go:231
2 @ 0x4384b6 0x40666c 0x4060d8 0x168d0f8 0x468c81
#   0x168d0f7   k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0x97  k8s.io/client-go@v0.22.5/tools/record/event.go:301
2 @ 0x4384b6 0x448252 0x493425 0x468c81
#   0x493424    context.propagateCancel.func1+0xa4  context/context.go:279
2 @ 0x4384b6 0x464d3d 0x464d1d 0x470d2c 0xedb75e 0x17234df 0x17234cb 0x1723268 0x468c81
#   0x464d1c    sync.runtime_notifyListWait+0x11c               runtime/sema.go:513
#   0x470d2b    sync.(*Cond).Wait+0x8b                      sync/cond.go:56
#   0xedb75d    k8s.io/client-go/util/workqueue.(*Type).Get+0x9d        k8s.io/client-go@v0.22.5/util/workqueue/queue.go:151
#   0x17234de   knative.dev/pkg/controller.(*twoLaneQueue).Get+0x5e     knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/controller/two_lane_queue.go:144
#   0x17234ca   knative.dev/pkg/controller.(*Impl).processNextWorkItem+0x4a knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/controller/controller.go:506
#   0x1723267   knative.dev/pkg/controller.(*Impl).RunContext.func3+0x67    knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/controller/controller.go:478
2 @ 0x4384b6 0x464d3d 0x464d1d 0x470d2c 0xedb75e 0x1726ede 0x468c81
#   0x464d1c    sync.runtime_notifyListWait+0x11c           runtime/sema.go:513
#   0x470d2b    sync.(*Cond).Wait+0x8b                  sync/cond.go:56
#   0xedb75d    k8s.io/client-go/util/workqueue.(*Type).Get+0x9d    k8s.io/client-go@v0.22.5/util/workqueue/queue.go:151
#   0x1726edd   knative.dev/pkg/controller.process+0xbd         knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/controller/two_lane_queue.go:67
1 @ 0x40bb54 0x465218 0x17288f9 0x468c81
#   0x465217    os/signal.signal_recv+0x97  runtime/sigqueue.go:169
#   0x17288f8   os/signal.loop+0x18     os/signal/signal_unix.go:24
1 @ 0x4384b6 0x40666c 0x406098 0x171a6af 0x468c81
#   0x171a6ae   knative.dev/pkg/metrics.(*metricsWorker).start+0x2e knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/metrics/metrics_worker.go:99
1 @ 0x4384b6 0x40666c 0x406098 0x1723105 0x1724f25 0x468c81
#   0x1723104   knative.dev/pkg/controller.(*Impl).RunContext+0x3e4 knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/controller/controller.go:484
#   0x1724f24   knative.dev/pkg/controller.StartAll.func1+0x64      knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/controller/controller.go:789
1 @ 0x4384b6 0x40666c 0x406098 0x1728acd 0x468c81
#   0x1728acc   knative.dev/pkg/signals.SetupSignalHandler.func1+0x2c   knative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/signals/signal.go:38
1 @ 0x4384b6 0x40666c 0x406098 0x17bd93f 0x468c81
#   0x17bd93e   knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/ingress.createRecorder.func1+0x3e    knative.dev/networking@v0.0.0-20220120043934-ec785540a732/pkg/client/injection/reconciler/networking/v1alpha1/ingress/controller.go:154
1 @ 0x4384b6 0x40666c 0x406098 0x17c3368 0x468c81
#   0x17c3367   knative.dev/networking/pkg/status.(*Prober).Start.func2+0x27    knative.dev/networking@v0.0.0-20220120043934-ec785540a732/pkg/status/status.go:295
1 @ 0x4384b6 0x40666c 0x406098 0x18c612f 0x18c54a7 0x18c536f 0x18c820f 0x4380e7 0x468c81
#   0x18c612e   knative.dev/pkg/injection/sharedmain.MainWithConfig+0xc0e   knative.dev/pkg@v0.0....

@nak3
Copy link
Contributor

nak3 commented Jun 8, 2022

Thank you for the dump.

But did you collect the data as instructed above? I think we need to see the dump 3 times with 5 sec internal to see which threads hung.

Also pod logs + CPU usage on the host is useful.

After that, please share the output of curl commands above and pod logs of net-kourier-controller.
Also, if you can login to the Kubernetes Node where net-kourier-controller is running, please check the CPU usage on the host.

@mbaynton
Copy link

Also experiencing this. We have nearly 9k knative services / ingresses defined. Kubernetes and knative as a whole are basically fine with this, but one little snag is that it takes ~4 hours for kourier to finish its Priming the config with 8901 ingresses before it starts listening on its service port. In a disaster recovery scenario in which we hypothetically lose envoy and kourier, this would introduce a huge time to recovery since the new Envoy pods are useless until they can speak with kourier.

Looks like more info is requested, I'll see if I can help to provide it.

@mbaynton
Copy link

With debug output on, we get more information after the Priming the config line. Looks like two lines per Ingress, note the timestamps between events:

{"severity":"DEBUG","timestamp":"2022-06-28T15:06:15.861408758Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6571 (depth: 64)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6571"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:17.461400595Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6572 (depth: 65)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6572"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:17.461436851Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6572 (depth: 65)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6572"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:19.061818171Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6573 (depth: 66)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6573"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:19.061857471Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6573 (depth: 66)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6573"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:20.661765171Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6574 (depth: 67)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6574"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:20.661801592Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6574 (depth: 67)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6574"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:22.262053674Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6575 (depth: 68)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6575"}
{"severity":"DEBUG","timestamp":"2022-06-28T15:06:22.262090767Z","logger":"net-kourier-controller","caller":"controller/controller.go:420","message":"Adding to queue cloud-outputs/mega6575 (depth: 68)","knative.dev/controller":"knative.dev.net-kourier.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"cloud-outputs/mega6575"}

CPU usage is measurable/more than zero but less than 1% of a core.

Three full pprof dumps about 5 seconds apart:

goroutine profile: total 115
15 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x1863175 0x1862ba8 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c					runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b							sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d			k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x1863174	knative.dev/networking/pkg/status.(*Prober).processWorkItem+0x54	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:348
#	0x1862ba7	knative.dev/networking/pkg/status.(*Prober).Start.func1+0x67		knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:289

7 @ 0x438556 0x40666c 0x4060d8 0xf6afe9 0x8c7507 0x8c7376 0x8c7269 0xf6af6b 0xf6af15 0x8c70da 0x468dc1
#	0xf6afe8	k8s.io/client-go/tools/cache.(*processorListener).run.func1+0x48	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:782
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0xf6af6a	k8s.io/apimachinery/pkg/util/wait.Until+0x6a				k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0xf6af14	k8s.io/client-go/tools/cache.(*processorListener).run+0x14		k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:781
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

7 @ 0x438556 0x4482f2 0xf6acb6 0x8c70da 0x468dc1
#	0xf6acb5	k8s.io/client-go/tools/cache.(*processorListener).pop+0x155	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:752
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x40666c 0x406098 0xf5c1e8 0x468dc1
#	0xf5c1e7	k8s.io/client-go/tools/cache.(*controller).Run.func1+0x27	k8s.io/client-go@v0.23.5/tools/cache/controller.go:130

6 @ 0x438556 0x40666c 0x406098 0xf6a105 0x8c6e62 0x8c70da 0x468dc1
#	0xf6a104	k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x44		k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:638
#	0x8c6e61	k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:56
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x4482f2 0x7d4988 0x7d3e9e 0x468dc1
#	0x7d4987	golang.org/x/net/http2.(*clientStream).writeRequest+0xaa7	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:1340
#	0x7d3e9d	golang.org/x/net/http2.(*clientStream).doRequest+0x1d		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:1202

6 @ 0x438556 0x4482f2 0xf640d7 0x468dc1
#	0xf640d6	k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2+0x136	k8s.io/client-go@v0.23.5/tools/cache/reflector.go:374

6 @ 0x438556 0x4482f2 0xf65776 0xf639f6 0xf63266 0x8c7507 0x8c7376 0xf630a5 0x8c6e62 0x8c70da 0x468dc1
#	0xf65775	k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x1b5		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:469
#	0xf639f5	k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0x675		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:429
#	0xf63265	k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x25		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:221
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0xf630a4	k8s.io/client-go/tools/cache.(*Reflector).Run+0x204			k8s.io/client-go@v0.23.5/tools/cache/reflector.go:220
#	0x8c6e61	k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:56
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0x7cd98b 0x7dafa5 0x515f1f 0x515b3b 0x515798 0xf273dc 0xf160a7 0xf1648f 0xa4367c 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c						runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b								sync/cond.go:56
#	0x7cd98a	golang.org/x/net/http2.(*pipe).Read+0xea					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/pipe.go:76
#	0x7dafa4	golang.org/x/net/http2.transportResponseBody.Read+0x84				golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2401
#	0x515f1e	encoding/json.(*Decoder).refill+0x17e						encoding/json/stream.go:165
#	0x515b3a	encoding/json.(*Decoder).readValue+0xba						encoding/json/stream.go:140
#	0x515797	encoding/json.(*Decoder).Decode+0x77						encoding/json/stream.go:63
#	0xf273db	k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x19b		k8s.io/apimachinery@v0.23.5/pkg/util/framer/framer.go:152
#	0xf160a6	k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0xa6	k8s.io/apimachinery@v0.23.5/pkg/runtime/serializer/streaming/streaming.go:77
#	0xf1648e	k8s.io/client-go/rest/watch.(*Decoder).Decode+0x4e				k8s.io/client-go@v0.23.5/rest/watch/decoder.go:49
#	0xa4367b	k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11b			k8s.io/apimachinery@v0.23.5/pkg/watch/streamwatcher.go:105

6 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf5eddc 0xf5c476 0x8c7507 0x8c7376 0x8c7269 0xf5c15b 0xf5c11b 0xf67d9f 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c				runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b						sync/cond.go:56
#	0xf5eddb	k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0x23b		k8s.io/client-go@v0.23.5/tools/cache/delta_fifo.go:527
#	0xf5c475	k8s.io/client-go/tools/cache.(*controller).processLoop+0x35	k8s.io/client-go@v0.23.5/tools/cache/controller.go:183
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0xf5c15a	k8s.io/apimachinery/pkg/util/wait.Until+0x2fa			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0xf5c11a	k8s.io/client-go/tools/cache.(*controller).Run+0x2ba		k8s.io/client-go@v0.23.5/tools/cache/controller.go:154
#	0xf67d9e	k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x49e	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:414

4 @ 0x438556 0x40666c 0x4060d8 0xf07ea7 0x468dc1
#	0xf07ea6	k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop+0xa6	k8s.io/client-go@v0.23.5/util/workqueue/queue.go:271

3 @ 0x438556 0x4482f2 0xf056ae 0x468dc1
#	0xf056ad	k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x34d	k8s.io/client-go@v0.23.5/util/workqueue/delaying_queue.go:231

2 @ 0x438556 0x40666c 0x4060d8 0x1727278 0x468dc1
#	0x1727277	k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0x97	k8s.io/client-go@v0.23.5/tools/record/event.go:304

2 @ 0x438556 0x4482f2 0x493565 0x468dc1
#	0x493564	context.propagateCancel.func1+0xa4	context/context.go:279

2 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x17c003f 0x17c002b 0x17bfdc8 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c				runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b						sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d		k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x17c003e	knative.dev/pkg/controller.(*twoLaneQueue).Get+0x5e		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:144
#	0x17c002a	knative.dev/pkg/controller.(*Impl).processNextWorkItem+0x4a	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:506
#	0x17bfdc7	knative.dev/pkg/controller.(*Impl).RunContext.func3+0x67	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:491

2 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x17c38be 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c			runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b					sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d	k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x17c38bd	knative.dev/pkg/controller.process+0xbd			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:67

1 @ 0x40bb54 0x465358 0x17c52d9 0x468dc1
#	0x465357	os/signal.signal_recv+0x97	runtime/sigqueue.go:169
#	0x17c52d8	os/signal.loop+0x18		os/signal/signal_unix.go:24

1 @ 0x438556 0x40666c 0x406098 0x17b718f 0x468dc1
#	0x17b718e	knative.dev/pkg/metrics.(*metricsWorker).start+0x2e	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/metrics_worker.go:99

1 @ 0x438556 0x40666c 0x406098 0x17bfc05 0x17c19a9 0x17c19aa 0x1719827 0x468dc1
#	0x17bfc04	knative.dev/pkg/controller.(*Impl).RunContext+0x424	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:497
#	0x17c19a8	knative.dev/pkg/controller.(*Impl).Run+0x28		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:446
#	0x17c19a9	knative.dev/pkg/controller.StartAll.func1+0x29		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:788
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66	golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x40666c 0x406098 0x17c54ad 0x468dc1
#	0x17c54ac	knative.dev/pkg/signals.SetupSignalHandler.func1+0x2c	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/signals/signal.go:38

1 @ 0x438556 0x40666c 0x406098 0x185d0ff 0x468dc1
#	0x185d0fe	knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/ingress.createRecorder.func1+0x3e	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/client/injection/reconciler/networking/v1alpha1/ingress/controller.go:154

1 @ 0x438556 0x40666c 0x406098 0x1862b08 0x468dc1
#	0x1862b07	knative.dev/networking/pkg/status.(*Prober).Start.func2+0x27	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:296

1 @ 0x438556 0x40666c 0x406098 0x196b6a9 0x196aa47 0x196a90f 0x196d72f 0x438187 0x468dc1
#	0x196b6a8	knative.dev/pkg/injection/sharedmain.MainWithConfig+0xbe8	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:288
#	0x196aa46	knative.dev/pkg/injection/sharedmain.MainWithContext+0xe6	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:182
#	0x196a90e	knative.dev/pkg/injection/sharedmain.Main+0x8e			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:120
#	0x196d72e	main.main+0xae							knative.dev/net-kourier/cmd/kourier/main.go:42
#	0x438186	runtime.main+0x226						runtime/proc.go:255

1 @ 0x438556 0x40666c 0x4060d8 0xa42d29 0x468dc1
#	0xa42d28	k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x48	k8s.io/apimachinery@v0.23.5/pkg/watch/mux.go:247

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x5b83f4 0x4c8c5a 0x7c670e 0x7c66c8 0x7c6f55 0x806812 0x87a782 0x87a066 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84						runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31						internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259					internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247							internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28								net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44								net/net.go:183
#	0x5b83f3	bufio.(*Reader).Read+0x1b3							bufio/bufio.go:227
#	0x4c8c59	io.ReadAtLeast+0x99								io/io.go:328
#	0x7c670d	io.ReadFull+0x6d								io/io.go:347
#	0x7c66c7	golang.org/x/net/http2.readFrameHeader+0x27					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:237
#	0x7c6f54	golang.org/x/net/http2.(*Framer).ReadFrame+0x94					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:498
#	0x806811	google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams+0xb1	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:599
#	0x87a781	google.golang.org/grpc.(*Server).serveStreams+0x141				google.golang.org/grpc@v1.42.0/server.go:905
#	0x87a065	google.golang.org/grpc.(*Server).handleRawConn.func1+0x45			google.golang.org/grpc@v1.42.0/server.go:847

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x6e227d 0x4fe958 0x6e2465 0x6dfad2 0x6e536f 0x6e5370 0x5b83f4 0x4c8c5a 0x7c670e 0x7c66c8 0x7c6f55 0x7d9370 0x7d88af 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84			runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259		internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247				internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28					net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44					net/net.go:183
#	0x6e227c	crypto/tls.(*atLeastReader).Read+0x3c			crypto/tls/conn.go:777
#	0x4fe957	bytes.(*Buffer).ReadFrom+0x97				bytes/buffer.go:204
#	0x6e2464	crypto/tls.(*Conn).readFromUntil+0xe4			crypto/tls/conn.go:799
#	0x6dfad1	crypto/tls.(*Conn).readRecordOrCCS+0x111		crypto/tls/conn.go:606
#	0x6e536e	crypto/tls.(*Conn).readRecord+0x16e			crypto/tls/conn.go:574
#	0x6e536f	crypto/tls.(*Conn).Read+0x16f				crypto/tls/conn.go:1277
#	0x5b83f3	bufio.(*Reader).Read+0x1b3				bufio/bufio.go:227
#	0x4c8c59	io.ReadAtLeast+0x99					io/io.go:328
#	0x7c670d	io.ReadFull+0x6d					io/io.go:347
#	0x7c66c7	golang.org/x/net/http2.readFrameHeader+0x27		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:237
#	0x7c6f54	golang.org/x/net/http2.(*Framer).ReadFrame+0x94		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:498
#	0x7d936f	golang.org/x/net/http2.(*clientConnReadLoop).run+0x12f	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2118
#	0x7d88ae	golang.org/x/net/http2.(*ClientConn).readLoop+0x6e	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2014

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x787aff 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84		runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28				net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44				net/net.go:183
#	0x787afe	net/http.(*connReader).backgroundRead+0x3e	net/http/server.go:672

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x7923b4 0x791fdd 0x1719827 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84			runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b		internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218			internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34				net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27				net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c				net/tcpsock.go:262
#	0x7923b3	net/http.(*Server).Serve+0x393				net/http/server.go:3002
#	0x791fdc	net/http.(*Server).ListenAndServe+0x7c			net/http/server.go:2931
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66	golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x7923b4 0x791fdd 0x17b8572 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84				runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31				internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b			internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218				internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34					net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27					net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c					net/tcpsock.go:262
#	0x7923b3	net/http.(*Server).Serve+0x393					net/http/server.go:3002
#	0x791fdc	net/http.(*Server).ListenAndServe+0x7c				net/http/server.go:2931
#	0x17b8571	knative.dev/pkg/metrics.newPrometheusExporter.func1+0x31	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/prometheus_exporter.go:55

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x879642 0xf71ab7 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84							runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31							internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b						internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218							internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34								net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27								net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c								net/tcpsock.go:262
#	0x879641	google.golang.org/grpc.(*Server).Serve+0x361						google.golang.org/grpc@v1.42.0/server.go:779
#	0xf71ab6	knative.dev/net-kourier/pkg/envoy/server.(*XdsServer).RunManagementServer.func1+0x36	knative.dev/net-kourier/pkg/envoy/server/xds_server.go:87

1 @ 0x438556 0x4482f2 0x16b8ba9 0x468dc1
#	0x16b8ba8	github.com/patrickmn/go-cache.(*janitor).Run+0x88	github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079

1 @ 0x438556 0x4482f2 0x174fb79 0x468dc1
#	0x174fb78	go.opencensus.io/stats/view.(*worker).start+0xb8	go.opencensus.io@v0.23.0/stats/view/worker.go:276

1 @ 0x438556 0x4482f2 0x17b384a 0x468dc1
#	0x17b3849	knative.dev/pkg/metrics.(*MemStatsProvider).Start.func1+0xc9	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/memstats.go:375

1 @ 0x438556 0x4482f2 0x17b92ea 0x468dc1
#	0x17b92e9	knative.dev/pkg/metrics.startCleanup.func1+0xe9	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/resource_view.go:115

1 @ 0x438556 0x4482f2 0x17c3b05 0x468dc1
#	0x17c3b04	knative.dev/pkg/controller.(*twoLaneQueue).runConsumer+0x184	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:106

1 @ 0x438556 0x4482f2 0x7ed93b 0x7ee025 0x803f66 0x468dc1
#	0x7ed93a	google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x11a	google.golang.org/grpc@v1.42.0/internal/transport/controlbuf.go:407
#	0x7ee024	google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x84	google.golang.org/grpc@v1.42.0/internal/transport/controlbuf.go:534
#	0x803f65	google.golang.org/grpc/internal/transport.NewServerTransport.func2+0xc5	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:322

1 @ 0x438556 0x4482f2 0x80a585 0x468dc1
#	0x80a584	google.golang.org/grpc/internal/transport.(*http2Server).keepalive+0x244	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:1119

1 @ 0x438556 0x4482f2 0x80ee16 0x80ecef 0x80fb92 0x4c8c5a 0x80fb05 0x80fadb 0x8761a7 0x876a06 0x87710e 0x88bae5 0xe937cc 0xeeeea5 0x468dc1
#	0x80ee15	google.golang.org/grpc/internal/transport.(*recvBufferReader).read+0x95										google.golang.org/grpc@v1.42.0/internal/transport/transport.go:178
#	0x80ecee	google.golang.org/grpc/internal/transport.(*recvBufferReader).Read+0x16e									google.golang.org/grpc@v1.42.0/internal/transport/transport.go:172
#	0x80fb91	google.golang.org/grpc/internal/transport.(*transportReader).Read+0x31										google.golang.org/grpc@v1.42.0/internal/transport/transport.go:484
#	0x4c8c59	io.ReadAtLeast+0x99																io/io.go:328
#	0x80fb04	io.ReadFull+0xa4																io/io.go:347
#	0x80fada	google.golang.org/grpc/internal/transport.(*Stream).Read+0x7a											google.golang.org/grpc@v1.42.0/internal/transport/transport.go:468
#	0x8761a6	google.golang.org/grpc.(*parser).recvMsg+0x46													google.golang.org/grpc@v1.42.0/rpc_util.go:559
#	0x876a05	google.golang.org/grpc.recvAndDecompress+0x65													google.golang.org/grpc@v1.42.0/rpc_util.go:690
#	0x87710d	google.golang.org/grpc.recv+0x6d														google.golang.org/grpc@v1.42.0/rpc_util.go:758
#	0x88bae4	google.golang.org/grpc.(*serverStream).RecvMsg+0x184												google.golang.org/grpc@v1.42.0/stream.go:1563
#	0xe937cb	github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3.(*aggregatedDiscoveryServiceStreamAggregatedResourcesServer).Recv+0x4b	github.com/envoyproxy/go-control-plane@v0.10.1/envoy/service/discovery/v3/ads.pb.go:317
#	0xeeeea4	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler.func1+0xa4							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:449

1 @ 0x438556 0x4482f2 0x8c73fb 0x8c7269 0x172fc59 0x172fbd2 0x172f36b 0x1738b84 0x1738a31 0x468dc1
#	0x8c73fa	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0x13a			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:167
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0x172fc58	k8s.io/apimachinery/pkg/util/wait.Until+0xd8				k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0x172fbd1	k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x51	k8s.io/client-go@v0.23.5/tools/leaderelection/leaderelection.go:268
#	0x172f36a	k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x16a	k8s.io/client-go@v0.23.5/tools/leaderelection/leaderelection.go:212
#	0x1738b83	knative.dev/pkg/leaderelection.(*runUntilCancelled).Run+0x43		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:309
#	0x1738a30	knative.dev/pkg/leaderelection.(*runAll).Run.func1+0x70			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:293

1 @ 0x438556 0x4482f2 0xeed394 0xeeedb4 0xef6512 0xef64e4 0xe936df 0x87f8da 0x880dc5 0x87ab58 0x468dc1
#	0xeed393	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process+0x573							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:224
#	0xeeedb3	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler+0xf3							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:463
#	0xef6511	github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamHandler+0x51							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/v3/server.go:182
#	0xef64e3	github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamAggregatedResources+0x23						github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/v3/server.go:186
#	0xe936de	github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3._AggregatedDiscoveryService_StreamAggregatedResources_Handler+0x9e	github.com/envoyproxy/go-control-plane@v0.10.1/envoy/service/discovery/v3/ads.pb.go:298
#	0x87f8d9	google.golang.org/grpc.(*Server).processStreamingRPC+0xeb9										google.golang.org/grpc@v1.42.0/server.go:1540
#	0x880dc4	google.golang.org/grpc.(*Server).handleStream+0x9e4											google.golang.org/grpc@v1.42.0/server.go:1620
#	0x87ab57	google.golang.org/grpc.(*Server).serveStreams.func1.2+0x97										google.golang.org/grpc@v1.42.0/server.go:921

1 @ 0x438556 0x4482f2 0xf719d3 0x18e2632 0x468dc1
#	0xf719d2	knative.dev/net-kourier/pkg/envoy/server.(*XdsServer).RunManagementServer+0x432	knative.dev/net-kourier/pkg/envoy/server/xds_server.go:92
#	0x18e2631	knative.dev/net-kourier/pkg/reconciler/ingress.NewController.func12+0xb1	knative.dev/net-kourier/pkg/reconciler/ingress/controller.go:233

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x17196a7 0x17c191f 0x196b92d 0x1719827 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x17196a6	golang.org/x/sync/errgroup.(*Group).Wait+0x26			golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:40
#	0x17c191e	knative.dev/pkg/controller.StartAll+0x11e			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:791
#	0x196b92c	knative.dev/pkg/injection/sharedmain.MainWithConfig.func2+0x2c	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:283
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66		golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x17387e9 0x17bfec9 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x17387e8	knative.dev/pkg/leaderelection.(*runAll).Run+0xa8		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:296
#	0x17bfec8	knative.dev/pkg/controller.(*Impl).RunContext.func2+0x68	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:481

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x1862aa6 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x1862aa5	knative.dev/networking/pkg/status.(*Prober).Start.func3+0x25	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:303

1 @ 0x462f25 0x173e915 0x173e72d 0x173b8ab 0x19053da 0x1905f8e 0x78e8cf 0x7901c9 0x1907396 0x791e3b 0x78d9a8 0x468dc1
#	0x462f24	runtime/pprof.runtime_goroutineProfileWithLabels+0x24	runtime/mprof.go:746
#	0x173e914	runtime/pprof.writeRuntimeProfile+0xb4			runtime/pprof/pprof.go:724
#	0x173e72c	runtime/pprof.writeGoroutine+0x4c			runtime/pprof/pprof.go:684
#	0x173b8aa	runtime/pprof.(*Profile).WriteTo+0x14a			runtime/pprof/pprof.go:331
#	0x19053d9	net/http/pprof.handler.ServeHTTP+0x499			net/http/pprof/pprof.go:253
#	0x1905f8d	net/http/pprof.Index+0x12d				net/http/pprof/pprof.go:371
#	0x78e8ce	net/http.HandlerFunc.ServeHTTP+0x2e			net/http/server.go:2047
#	0x7901c8	net/http.(*ServeMux).ServeHTTP+0x148			net/http/server.go:2425
#	0x1907395	knative.dev/pkg/profiling.(*Handler).ServeHTTP+0x35	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/profiling/server.go:74
#	0x791e3a	net/http.serverHandler.ServeHTTP+0x43a			net/http/server.go:2879
#	0x78d9a7	net/http.(*conn).serve+0xb07				net/http/server.go:1930

mbaynton@mike-laptop:~$ curl http://localhost:8008/debug/pprof/goroutine?debug=1
goroutine profile: total 115
15 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x1863175 0x1862ba8 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c					runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b							sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d			k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x1863174	knative.dev/networking/pkg/status.(*Prober).processWorkItem+0x54	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:348
#	0x1862ba7	knative.dev/networking/pkg/status.(*Prober).Start.func1+0x67		knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:289

7 @ 0x438556 0x40666c 0x4060d8 0xf6afe9 0x8c7507 0x8c7376 0x8c7269 0xf6af6b 0xf6af15 0x8c70da 0x468dc1
#	0xf6afe8	k8s.io/client-go/tools/cache.(*processorListener).run.func1+0x48	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:782
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0xf6af6a	k8s.io/apimachinery/pkg/util/wait.Until+0x6a				k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0xf6af14	k8s.io/client-go/tools/cache.(*processorListener).run+0x14		k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:781
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

7 @ 0x438556 0x4482f2 0xf6acb6 0x8c70da 0x468dc1
#	0xf6acb5	k8s.io/client-go/tools/cache.(*processorListener).pop+0x155	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:752
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x40666c 0x406098 0xf5c1e8 0x468dc1
#	0xf5c1e7	k8s.io/client-go/tools/cache.(*controller).Run.func1+0x27	k8s.io/client-go@v0.23.5/tools/cache/controller.go:130

6 @ 0x438556 0x40666c 0x406098 0xf6a105 0x8c6e62 0x8c70da 0x468dc1
#	0xf6a104	k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x44		k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:638
#	0x8c6e61	k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:56
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x4482f2 0x7d4988 0x7d3e9e 0x468dc1
#	0x7d4987	golang.org/x/net/http2.(*clientStream).writeRequest+0xaa7	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:1340
#	0x7d3e9d	golang.org/x/net/http2.(*clientStream).doRequest+0x1d		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:1202

6 @ 0x438556 0x4482f2 0xf640d7 0x468dc1
#	0xf640d6	k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2+0x136	k8s.io/client-go@v0.23.5/tools/cache/reflector.go:374

6 @ 0x438556 0x4482f2 0xf65776 0xf639f6 0xf63266 0x8c7507 0x8c7376 0xf630a5 0x8c6e62 0x8c70da 0x468dc1
#	0xf65775	k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x1b5		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:469
#	0xf639f5	k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0x675		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:429
#	0xf63265	k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x25		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:221
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0xf630a4	k8s.io/client-go/tools/cache.(*Reflector).Run+0x204			k8s.io/client-go@v0.23.5/tools/cache/reflector.go:220
#	0x8c6e61	k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:56
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0x7cd98b 0x7dafa5 0x515f1f 0x515b3b 0x515798 0xf273dc 0xf160a7 0xf1648f 0xa4367c 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c						runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b								sync/cond.go:56
#	0x7cd98a	golang.org/x/net/http2.(*pipe).Read+0xea					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/pipe.go:76
#	0x7dafa4	golang.org/x/net/http2.transportResponseBody.Read+0x84				golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2401
#	0x515f1e	encoding/json.(*Decoder).refill+0x17e						encoding/json/stream.go:165
#	0x515b3a	encoding/json.(*Decoder).readValue+0xba						encoding/json/stream.go:140
#	0x515797	encoding/json.(*Decoder).Decode+0x77						encoding/json/stream.go:63
#	0xf273db	k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x19b		k8s.io/apimachinery@v0.23.5/pkg/util/framer/framer.go:152
#	0xf160a6	k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0xa6	k8s.io/apimachinery@v0.23.5/pkg/runtime/serializer/streaming/streaming.go:77
#	0xf1648e	k8s.io/client-go/rest/watch.(*Decoder).Decode+0x4e				k8s.io/client-go@v0.23.5/rest/watch/decoder.go:49
#	0xa4367b	k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11b			k8s.io/apimachinery@v0.23.5/pkg/watch/streamwatcher.go:105

6 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf5eddc 0xf5c476 0x8c7507 0x8c7376 0x8c7269 0xf5c15b 0xf5c11b 0xf67d9f 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c				runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b						sync/cond.go:56
#	0xf5eddb	k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0x23b		k8s.io/client-go@v0.23.5/tools/cache/delta_fifo.go:527
#	0xf5c475	k8s.io/client-go/tools/cache.(*controller).processLoop+0x35	k8s.io/client-go@v0.23.5/tools/cache/controller.go:183
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0xf5c15a	k8s.io/apimachinery/pkg/util/wait.Until+0x2fa			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0xf5c11a	k8s.io/client-go/tools/cache.(*controller).Run+0x2ba		k8s.io/client-go@v0.23.5/tools/cache/controller.go:154
#	0xf67d9e	k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x49e	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:414

4 @ 0x438556 0x40666c 0x4060d8 0xf07ea7 0x468dc1
#	0xf07ea6	k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop+0xa6	k8s.io/client-go@v0.23.5/util/workqueue/queue.go:271

3 @ 0x438556 0x4482f2 0xf056ae 0x468dc1
#	0xf056ad	k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x34d	k8s.io/client-go@v0.23.5/util/workqueue/delaying_queue.go:231

2 @ 0x438556 0x40666c 0x4060d8 0x1727278 0x468dc1
#	0x1727277	k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0x97	k8s.io/client-go@v0.23.5/tools/record/event.go:304

2 @ 0x438556 0x4482f2 0x493565 0x468dc1
#	0x493564	context.propagateCancel.func1+0xa4	context/context.go:279

2 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x17c003f 0x17c002b 0x17bfdc8 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c				runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b						sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d		k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x17c003e	knative.dev/pkg/controller.(*twoLaneQueue).Get+0x5e		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:144
#	0x17c002a	knative.dev/pkg/controller.(*Impl).processNextWorkItem+0x4a	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:506
#	0x17bfdc7	knative.dev/pkg/controller.(*Impl).RunContext.func3+0x67	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:491

2 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x17c38be 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c			runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b					sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d	k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x17c38bd	knative.dev/pkg/controller.process+0xbd			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:67

1 @ 0x40bb54 0x465358 0x17c52d9 0x468dc1
#	0x465357	os/signal.signal_recv+0x97	runtime/sigqueue.go:169
#	0x17c52d8	os/signal.loop+0x18		os/signal/signal_unix.go:24

1 @ 0x438556 0x40666c 0x406098 0x17b718f 0x468dc1
#	0x17b718e	knative.dev/pkg/metrics.(*metricsWorker).start+0x2e	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/metrics_worker.go:99

1 @ 0x438556 0x40666c 0x406098 0x17bfc05 0x17c19a9 0x17c19aa 0x1719827 0x468dc1
#	0x17bfc04	knative.dev/pkg/controller.(*Impl).RunContext+0x424	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:497
#	0x17c19a8	knative.dev/pkg/controller.(*Impl).Run+0x28		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:446
#	0x17c19a9	knative.dev/pkg/controller.StartAll.func1+0x29		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:788
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66	golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x40666c 0x406098 0x17c54ad 0x468dc1
#	0x17c54ac	knative.dev/pkg/signals.SetupSignalHandler.func1+0x2c	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/signals/signal.go:38

1 @ 0x438556 0x40666c 0x406098 0x185d0ff 0x468dc1
#	0x185d0fe	knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/ingress.createRecorder.func1+0x3e	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/client/injection/reconciler/networking/v1alpha1/ingress/controller.go:154

1 @ 0x438556 0x40666c 0x406098 0x1862b08 0x468dc1
#	0x1862b07	knative.dev/networking/pkg/status.(*Prober).Start.func2+0x27	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:296

1 @ 0x438556 0x40666c 0x406098 0x196b6a9 0x196aa47 0x196a90f 0x196d72f 0x438187 0x468dc1
#	0x196b6a8	knative.dev/pkg/injection/sharedmain.MainWithConfig+0xbe8	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:288
#	0x196aa46	knative.dev/pkg/injection/sharedmain.MainWithContext+0xe6	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:182
#	0x196a90e	knative.dev/pkg/injection/sharedmain.Main+0x8e			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:120
#	0x196d72e	main.main+0xae							knative.dev/net-kourier/cmd/kourier/main.go:42
#	0x438186	runtime.main+0x226						runtime/proc.go:255

1 @ 0x438556 0x40666c 0x4060d8 0xa42d29 0x468dc1
#	0xa42d28	k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x48	k8s.io/apimachinery@v0.23.5/pkg/watch/mux.go:247

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x5b83f4 0x4c8c5a 0x7c670e 0x7c66c8 0x7c6f55 0x806812 0x87a782 0x87a066 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84						runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31						internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259					internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247							internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28								net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44								net/net.go:183
#	0x5b83f3	bufio.(*Reader).Read+0x1b3							bufio/bufio.go:227
#	0x4c8c59	io.ReadAtLeast+0x99								io/io.go:328
#	0x7c670d	io.ReadFull+0x6d								io/io.go:347
#	0x7c66c7	golang.org/x/net/http2.readFrameHeader+0x27					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:237
#	0x7c6f54	golang.org/x/net/http2.(*Framer).ReadFrame+0x94					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:498
#	0x806811	google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams+0xb1	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:599
#	0x87a781	google.golang.org/grpc.(*Server).serveStreams+0x141				google.golang.org/grpc@v1.42.0/server.go:905
#	0x87a065	google.golang.org/grpc.(*Server).handleRawConn.func1+0x45			google.golang.org/grpc@v1.42.0/server.go:847

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x6e227d 0x4fe958 0x6e2465 0x6dfad2 0x6e536f 0x6e5370 0x5b83f4 0x4c8c5a 0x7c670e 0x7c66c8 0x7c6f55 0x7d9370 0x7d88af 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84			runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259		internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247				internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28					net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44					net/net.go:183
#	0x6e227c	crypto/tls.(*atLeastReader).Read+0x3c			crypto/tls/conn.go:777
#	0x4fe957	bytes.(*Buffer).ReadFrom+0x97				bytes/buffer.go:204
#	0x6e2464	crypto/tls.(*Conn).readFromUntil+0xe4			crypto/tls/conn.go:799
#	0x6dfad1	crypto/tls.(*Conn).readRecordOrCCS+0x111		crypto/tls/conn.go:606
#	0x6e536e	crypto/tls.(*Conn).readRecord+0x16e			crypto/tls/conn.go:574
#	0x6e536f	crypto/tls.(*Conn).Read+0x16f				crypto/tls/conn.go:1277
#	0x5b83f3	bufio.(*Reader).Read+0x1b3				bufio/bufio.go:227
#	0x4c8c59	io.ReadAtLeast+0x99					io/io.go:328
#	0x7c670d	io.ReadFull+0x6d					io/io.go:347
#	0x7c66c7	golang.org/x/net/http2.readFrameHeader+0x27		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:237
#	0x7c6f54	golang.org/x/net/http2.(*Framer).ReadFrame+0x94		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:498
#	0x7d936f	golang.org/x/net/http2.(*clientConnReadLoop).run+0x12f	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2118
#	0x7d88ae	golang.org/x/net/http2.(*ClientConn).readLoop+0x6e	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2014

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x787aff 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84		runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28				net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44				net/net.go:183
#	0x787afe	net/http.(*connReader).backgroundRead+0x3e	net/http/server.go:672

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x7923b4 0x791fdd 0x1719827 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84			runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b		internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218			internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34				net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27				net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c				net/tcpsock.go:262
#	0x7923b3	net/http.(*Server).Serve+0x393				net/http/server.go:3002
#	0x791fdc	net/http.(*Server).ListenAndServe+0x7c			net/http/server.go:2931
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66	golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x7923b4 0x791fdd 0x17b8572 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84				runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31				internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b			internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218				internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34					net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27					net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c					net/tcpsock.go:262
#	0x7923b3	net/http.(*Server).Serve+0x393					net/http/server.go:3002
#	0x791fdc	net/http.(*Server).ListenAndServe+0x7c				net/http/server.go:2931
#	0x17b8571	knative.dev/pkg/metrics.newPrometheusExporter.func1+0x31	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/prometheus_exporter.go:55

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x879642 0xf71ab7 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84							runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31							internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b						internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218							internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34								net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27								net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c								net/tcpsock.go:262
#	0x879641	google.golang.org/grpc.(*Server).Serve+0x361						google.golang.org/grpc@v1.42.0/server.go:779
#	0xf71ab6	knative.dev/net-kourier/pkg/envoy/server.(*XdsServer).RunManagementServer.func1+0x36	knative.dev/net-kourier/pkg/envoy/server/xds_server.go:87

1 @ 0x438556 0x4482f2 0x16b8ba9 0x468dc1
#	0x16b8ba8	github.com/patrickmn/go-cache.(*janitor).Run+0x88	github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079

1 @ 0x438556 0x4482f2 0x174fb79 0x468dc1
#	0x174fb78	go.opencensus.io/stats/view.(*worker).start+0xb8	go.opencensus.io@v0.23.0/stats/view/worker.go:276

1 @ 0x438556 0x4482f2 0x17b384a 0x468dc1
#	0x17b3849	knative.dev/pkg/metrics.(*MemStatsProvider).Start.func1+0xc9	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/memstats.go:375

1 @ 0x438556 0x4482f2 0x17b92ea 0x468dc1
#	0x17b92e9	knative.dev/pkg/metrics.startCleanup.func1+0xe9	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/resource_view.go:115

1 @ 0x438556 0x4482f2 0x17c3b05 0x468dc1
#	0x17c3b04	knative.dev/pkg/controller.(*twoLaneQueue).runConsumer+0x184	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:106

1 @ 0x438556 0x4482f2 0x7ed93b 0x7ee025 0x803f66 0x468dc1
#	0x7ed93a	google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x11a	google.golang.org/grpc@v1.42.0/internal/transport/controlbuf.go:407
#	0x7ee024	google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x84	google.golang.org/grpc@v1.42.0/internal/transport/controlbuf.go:534
#	0x803f65	google.golang.org/grpc/internal/transport.NewServerTransport.func2+0xc5	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:322

1 @ 0x438556 0x4482f2 0x80a585 0x468dc1
#	0x80a584	google.golang.org/grpc/internal/transport.(*http2Server).keepalive+0x244	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:1119

1 @ 0x438556 0x4482f2 0x80ee16 0x80ecef 0x80fb92 0x4c8c5a 0x80fb05 0x80fadb 0x8761a7 0x876a06 0x87710e 0x88bae5 0xe937cc 0xeeeea5 0x468dc1
#	0x80ee15	google.golang.org/grpc/internal/transport.(*recvBufferReader).read+0x95										google.golang.org/grpc@v1.42.0/internal/transport/transport.go:178
#	0x80ecee	google.golang.org/grpc/internal/transport.(*recvBufferReader).Read+0x16e									google.golang.org/grpc@v1.42.0/internal/transport/transport.go:172
#	0x80fb91	google.golang.org/grpc/internal/transport.(*transportReader).Read+0x31										google.golang.org/grpc@v1.42.0/internal/transport/transport.go:484
#	0x4c8c59	io.ReadAtLeast+0x99																io/io.go:328
#	0x80fb04	io.ReadFull+0xa4																io/io.go:347
#	0x80fada	google.golang.org/grpc/internal/transport.(*Stream).Read+0x7a											google.golang.org/grpc@v1.42.0/internal/transport/transport.go:468
#	0x8761a6	google.golang.org/grpc.(*parser).recvMsg+0x46													google.golang.org/grpc@v1.42.0/rpc_util.go:559
#	0x876a05	google.golang.org/grpc.recvAndDecompress+0x65													google.golang.org/grpc@v1.42.0/rpc_util.go:690
#	0x87710d	google.golang.org/grpc.recv+0x6d														google.golang.org/grpc@v1.42.0/rpc_util.go:758
#	0x88bae4	google.golang.org/grpc.(*serverStream).RecvMsg+0x184												google.golang.org/grpc@v1.42.0/stream.go:1563
#	0xe937cb	github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3.(*aggregatedDiscoveryServiceStreamAggregatedResourcesServer).Recv+0x4b	github.com/envoyproxy/go-control-plane@v0.10.1/envoy/service/discovery/v3/ads.pb.go:317
#	0xeeeea4	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler.func1+0xa4							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:449

1 @ 0x438556 0x4482f2 0x8c73fb 0x8c7269 0x172fc59 0x172fbd2 0x172f36b 0x1738b84 0x1738a31 0x468dc1
#	0x8c73fa	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0x13a			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:167
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0x172fc58	k8s.io/apimachinery/pkg/util/wait.Until+0xd8				k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0x172fbd1	k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x51	k8s.io/client-go@v0.23.5/tools/leaderelection/leaderelection.go:268
#	0x172f36a	k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x16a	k8s.io/client-go@v0.23.5/tools/leaderelection/leaderelection.go:212
#	0x1738b83	knative.dev/pkg/leaderelection.(*runUntilCancelled).Run+0x43		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:309
#	0x1738a30	knative.dev/pkg/leaderelection.(*runAll).Run.func1+0x70			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:293

1 @ 0x438556 0x4482f2 0xeed394 0xeeedb4 0xef6512 0xef64e4 0xe936df 0x87f8da 0x880dc5 0x87ab58 0x468dc1
#	0xeed393	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process+0x573							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:224
#	0xeeedb3	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler+0xf3							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:463
#	0xef6511	github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamHandler+0x51							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/v3/server.go:182
#	0xef64e3	github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamAggregatedResources+0x23						github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/v3/server.go:186
#	0xe936de	github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3._AggregatedDiscoveryService_StreamAggregatedResources_Handler+0x9e	github.com/envoyproxy/go-control-plane@v0.10.1/envoy/service/discovery/v3/ads.pb.go:298
#	0x87f8d9	google.golang.org/grpc.(*Server).processStreamingRPC+0xeb9										google.golang.org/grpc@v1.42.0/server.go:1540
#	0x880dc4	google.golang.org/grpc.(*Server).handleStream+0x9e4											google.golang.org/grpc@v1.42.0/server.go:1620
#	0x87ab57	google.golang.org/grpc.(*Server).serveStreams.func1.2+0x97										google.golang.org/grpc@v1.42.0/server.go:921

1 @ 0x438556 0x4482f2 0xf719d3 0x18e2632 0x468dc1
#	0xf719d2	knative.dev/net-kourier/pkg/envoy/server.(*XdsServer).RunManagementServer+0x432	knative.dev/net-kourier/pkg/envoy/server/xds_server.go:92
#	0x18e2631	knative.dev/net-kourier/pkg/reconciler/ingress.NewController.func12+0xb1	knative.dev/net-kourier/pkg/reconciler/ingress/controller.go:233

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x17196a7 0x17c191f 0x196b92d 0x1719827 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x17196a6	golang.org/x/sync/errgroup.(*Group).Wait+0x26			golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:40
#	0x17c191e	knative.dev/pkg/controller.StartAll+0x11e			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:791
#	0x196b92c	knative.dev/pkg/injection/sharedmain.MainWithConfig.func2+0x2c	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:283
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66		golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x17387e9 0x17bfec9 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x17387e8	knative.dev/pkg/leaderelection.(*runAll).Run+0xa8		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:296
#	0x17bfec8	knative.dev/pkg/controller.(*Impl).RunContext.func2+0x68	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:481

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x1862aa6 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x1862aa5	knative.dev/networking/pkg/status.(*Prober).Start.func3+0x25	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:303

1 @ 0x462f25 0x173e915 0x173e72d 0x173b8ab 0x19053da 0x1905f8e 0x78e8cf 0x7901c9 0x1907396 0x791e3b 0x78d9a8 0x468dc1
#	0x462f24	runtime/pprof.runtime_goroutineProfileWithLabels+0x24	runtime/mprof.go:746
#	0x173e914	runtime/pprof.writeRuntimeProfile+0xb4			runtime/pprof/pprof.go:724
#	0x173e72c	runtime/pprof.writeGoroutine+0x4c			runtime/pprof/pprof.go:684
#	0x173b8aa	runtime/pprof.(*Profile).WriteTo+0x14a			runtime/pprof/pprof.go:331
#	0x19053d9	net/http/pprof.handler.ServeHTTP+0x499			net/http/pprof/pprof.go:253
#	0x1905f8d	net/http/pprof.Index+0x12d				net/http/pprof/pprof.go:371
#	0x78e8ce	net/http.HandlerFunc.ServeHTTP+0x2e			net/http/server.go:2047
#	0x7901c8	net/http.(*ServeMux).ServeHTTP+0x148			net/http/server.go:2425
#	0x1907395	knative.dev/pkg/profiling.(*Handler).ServeHTTP+0x35	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/profiling/server.go:74
#	0x791e3a	net/http.serverHandler.ServeHTTP+0x43a			net/http/server.go:2879
#	0x78d9a7	net/http.(*conn).serve+0xb07				net/http/server.go:1930

mbaynton@mike-laptop:~$ curl http://localhost:8008/debug/pprof/goroutine?debug=1
goroutine profile: total 115
15 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x1863175 0x1862ba8 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c					runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b							sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d			k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x1863174	knative.dev/networking/pkg/status.(*Prober).processWorkItem+0x54	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:348
#	0x1862ba7	knative.dev/networking/pkg/status.(*Prober).Start.func1+0x67		knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:289

7 @ 0x438556 0x40666c 0x4060d8 0xf6afe9 0x8c7507 0x8c7376 0x8c7269 0xf6af6b 0xf6af15 0x8c70da 0x468dc1
#	0xf6afe8	k8s.io/client-go/tools/cache.(*processorListener).run.func1+0x48	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:782
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0xf6af6a	k8s.io/apimachinery/pkg/util/wait.Until+0x6a				k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0xf6af14	k8s.io/client-go/tools/cache.(*processorListener).run+0x14		k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:781
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

7 @ 0x438556 0x4482f2 0xf6acb6 0x8c70da 0x468dc1
#	0xf6acb5	k8s.io/client-go/tools/cache.(*processorListener).pop+0x155	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:752
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x40666c 0x406098 0xf5c1e8 0x468dc1
#	0xf5c1e7	k8s.io/client-go/tools/cache.(*controller).Run.func1+0x27	k8s.io/client-go@v0.23.5/tools/cache/controller.go:130

6 @ 0x438556 0x40666c 0x406098 0xf6a105 0x8c6e62 0x8c70da 0x468dc1
#	0xf6a104	k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x44		k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:638
#	0x8c6e61	k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:56
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x4482f2 0x7d4988 0x7d3e9e 0x468dc1
#	0x7d4987	golang.org/x/net/http2.(*clientStream).writeRequest+0xaa7	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:1340
#	0x7d3e9d	golang.org/x/net/http2.(*clientStream).doRequest+0x1d		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:1202

6 @ 0x438556 0x4482f2 0xf640d7 0x468dc1
#	0xf640d6	k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2+0x136	k8s.io/client-go@v0.23.5/tools/cache/reflector.go:374

6 @ 0x438556 0x4482f2 0xf65776 0xf639f6 0xf63266 0x8c7507 0x8c7376 0xf630a5 0x8c6e62 0x8c70da 0x468dc1
#	0xf65775	k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x1b5		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:469
#	0xf639f5	k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0x675		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:429
#	0xf63265	k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x25		k8s.io/client-go@v0.23.5/tools/cache/reflector.go:221
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0xf630a4	k8s.io/client-go/tools/cache.(*Reflector).Run+0x204			k8s.io/client-go@v0.23.5/tools/cache/reflector.go:220
#	0x8c6e61	k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x21	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:56
#	0x8c70d9	k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x59		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:73

6 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0x7cd98b 0x7dafa5 0x515f1f 0x515b3b 0x515798 0xf273dc 0xf160a7 0xf1648f 0xa4367c 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c						runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b								sync/cond.go:56
#	0x7cd98a	golang.org/x/net/http2.(*pipe).Read+0xea					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/pipe.go:76
#	0x7dafa4	golang.org/x/net/http2.transportResponseBody.Read+0x84				golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2401
#	0x515f1e	encoding/json.(*Decoder).refill+0x17e						encoding/json/stream.go:165
#	0x515b3a	encoding/json.(*Decoder).readValue+0xba						encoding/json/stream.go:140
#	0x515797	encoding/json.(*Decoder).Decode+0x77						encoding/json/stream.go:63
#	0xf273db	k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x19b		k8s.io/apimachinery@v0.23.5/pkg/util/framer/framer.go:152
#	0xf160a6	k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0xa6	k8s.io/apimachinery@v0.23.5/pkg/runtime/serializer/streaming/streaming.go:77
#	0xf1648e	k8s.io/client-go/rest/watch.(*Decoder).Decode+0x4e				k8s.io/client-go@v0.23.5/rest/watch/decoder.go:49
#	0xa4367b	k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11b			k8s.io/apimachinery@v0.23.5/pkg/watch/streamwatcher.go:105

6 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf5eddc 0xf5c476 0x8c7507 0x8c7376 0x8c7269 0xf5c15b 0xf5c11b 0xf67d9f 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c				runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b						sync/cond.go:56
#	0xf5eddb	k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0x23b		k8s.io/client-go@v0.23.5/tools/cache/delta_fifo.go:527
#	0xf5c475	k8s.io/client-go/tools/cache.(*controller).processLoop+0x35	k8s.io/client-go@v0.23.5/tools/cache/controller.go:183
#	0x8c7506	k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1+0x66	k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:155
#	0x8c7375	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0xb5		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:156
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88		k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0xf5c15a	k8s.io/apimachinery/pkg/util/wait.Until+0x2fa			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0xf5c11a	k8s.io/client-go/tools/cache.(*controller).Run+0x2ba		k8s.io/client-go@v0.23.5/tools/cache/controller.go:154
#	0xf67d9e	k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x49e	k8s.io/client-go@v0.23.5/tools/cache/shared_informer.go:414

4 @ 0x438556 0x40666c 0x4060d8 0xf07ea7 0x468dc1
#	0xf07ea6	k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop+0xa6	k8s.io/client-go@v0.23.5/util/workqueue/queue.go:271

3 @ 0x438556 0x4482f2 0xf056ae 0x468dc1
#	0xf056ad	k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x34d	k8s.io/client-go@v0.23.5/util/workqueue/delaying_queue.go:231

2 @ 0x438556 0x40666c 0x4060d8 0x1727278 0x468dc1
#	0x1727277	k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0x97	k8s.io/client-go@v0.23.5/tools/record/event.go:304

2 @ 0x438556 0x4482f2 0x493565 0x468dc1
#	0x493564	context.propagateCancel.func1+0xa4	context/context.go:279

2 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x17c003f 0x17c002b 0x17bfdc8 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c				runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b						sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d		k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x17c003e	knative.dev/pkg/controller.(*twoLaneQueue).Get+0x5e		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:144
#	0x17c002a	knative.dev/pkg/controller.(*Impl).processNextWorkItem+0x4a	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:506
#	0x17bfdc7	knative.dev/pkg/controller.(*Impl).RunContext.func3+0x67	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:491

2 @ 0x438556 0x464e7d 0x464e5d 0x470e6c 0xf0703e 0x17c38be 0x468dc1
#	0x464e5c	sync.runtime_notifyListWait+0x11c			runtime/sema.go:513
#	0x470e6b	sync.(*Cond).Wait+0x8b					sync/cond.go:56
#	0xf0703d	k8s.io/client-go/util/workqueue.(*Type).Get+0x9d	k8s.io/client-go@v0.23.5/util/workqueue/queue.go:157
#	0x17c38bd	knative.dev/pkg/controller.process+0xbd			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:67

1 @ 0x40bb54 0x465358 0x17c52d9 0x468dc1
#	0x465357	os/signal.signal_recv+0x97	runtime/sigqueue.go:169
#	0x17c52d8	os/signal.loop+0x18		os/signal/signal_unix.go:24

1 @ 0x438556 0x40666c 0x406098 0x17b718f 0x468dc1
#	0x17b718e	knative.dev/pkg/metrics.(*metricsWorker).start+0x2e	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/metrics_worker.go:99

1 @ 0x438556 0x40666c 0x406098 0x17bfc05 0x17c19a9 0x17c19aa 0x1719827 0x468dc1
#	0x17bfc04	knative.dev/pkg/controller.(*Impl).RunContext+0x424	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:497
#	0x17c19a8	knative.dev/pkg/controller.(*Impl).Run+0x28		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:446
#	0x17c19a9	knative.dev/pkg/controller.StartAll.func1+0x29		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:788
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66	golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x40666c 0x406098 0x17c54ad 0x468dc1
#	0x17c54ac	knative.dev/pkg/signals.SetupSignalHandler.func1+0x2c	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/signals/signal.go:38

1 @ 0x438556 0x40666c 0x406098 0x185d0ff 0x468dc1
#	0x185d0fe	knative.dev/networking/pkg/client/injection/reconciler/networking/v1alpha1/ingress.createRecorder.func1+0x3e	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/client/injection/reconciler/networking/v1alpha1/ingress/controller.go:154

1 @ 0x438556 0x40666c 0x406098 0x1862b08 0x468dc1
#	0x1862b07	knative.dev/networking/pkg/status.(*Prober).Start.func2+0x27	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:296

1 @ 0x438556 0x40666c 0x406098 0x196b6a9 0x196aa47 0x196a90f 0x196d72f 0x438187 0x468dc1
#	0x196b6a8	knative.dev/pkg/injection/sharedmain.MainWithConfig+0xbe8	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:288
#	0x196aa46	knative.dev/pkg/injection/sharedmain.MainWithContext+0xe6	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:182
#	0x196a90e	knative.dev/pkg/injection/sharedmain.Main+0x8e			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:120
#	0x196d72e	main.main+0xae							knative.dev/net-kourier/cmd/kourier/main.go:42
#	0x438186	runtime.main+0x226						runtime/proc.go:255

1 @ 0x438556 0x40666c 0x4060d8 0xa42d29 0x468dc1
#	0xa42d28	k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x48	k8s.io/apimachinery@v0.23.5/pkg/watch/mux.go:247

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x5b83f4 0x4c8c5a 0x7c670e 0x7c66c8 0x7c6f55 0x806812 0x87a782 0x87a066 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84						runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31						internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259					internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247							internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28								net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44								net/net.go:183
#	0x5b83f3	bufio.(*Reader).Read+0x1b3							bufio/bufio.go:227
#	0x4c8c59	io.ReadAtLeast+0x99								io/io.go:328
#	0x7c670d	io.ReadFull+0x6d								io/io.go:347
#	0x7c66c7	golang.org/x/net/http2.readFrameHeader+0x27					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:237
#	0x7c6f54	golang.org/x/net/http2.(*Framer).ReadFrame+0x94					golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:498
#	0x806811	google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams+0xb1	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:599
#	0x87a781	google.golang.org/grpc.(*Server).serveStreams+0x141				google.golang.org/grpc@v1.42.0/server.go:905
#	0x87a065	google.golang.org/grpc.(*Server).handleRawConn.func1+0x45			google.golang.org/grpc@v1.42.0/server.go:847

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x6e227d 0x4fe958 0x6e2465 0x6dfad2 0x6e536f 0x6e5370 0x5b83f4 0x4c8c5a 0x7c670e 0x7c66c8 0x7c6f55 0x7d9370 0x7d88af 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84			runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259		internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247				internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28					net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44					net/net.go:183
#	0x6e227c	crypto/tls.(*atLeastReader).Read+0x3c			crypto/tls/conn.go:777
#	0x4fe957	bytes.(*Buffer).ReadFrom+0x97				bytes/buffer.go:204
#	0x6e2464	crypto/tls.(*Conn).readFromUntil+0xe4			crypto/tls/conn.go:799
#	0x6dfad1	crypto/tls.(*Conn).readRecordOrCCS+0x111		crypto/tls/conn.go:606
#	0x6e536e	crypto/tls.(*Conn).readRecord+0x16e			crypto/tls/conn.go:574
#	0x6e536f	crypto/tls.(*Conn).Read+0x16f				crypto/tls/conn.go:1277
#	0x5b83f3	bufio.(*Reader).Read+0x1b3				bufio/bufio.go:227
#	0x4c8c59	io.ReadAtLeast+0x99					io/io.go:328
#	0x7c670d	io.ReadFull+0x6d					io/io.go:347
#	0x7c66c7	golang.org/x/net/http2.readFrameHeader+0x27		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:237
#	0x7c6f54	golang.org/x/net/http2.(*Framer).ReadFrame+0x94		golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/frame.go:498
#	0x7d936f	golang.org/x/net/http2.(*clientConnReadLoop).run+0x12f	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2118
#	0x7d88ae	golang.org/x/net/http2.(*ClientConn).readLoop+0x6e	golang.org/x/net@v0.0.0-20220225172249-27dd8689420f/http2/transport.go:2014

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d063a 0x4d0628 0x52f409 0x53f965 0x787aff 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84		runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31		internal/poll/fd_poll_runtime.go:84
#	0x4d0639	internal/poll.(*pollDesc).waitRead+0x259	internal/poll/fd_poll_runtime.go:89
#	0x4d0627	internal/poll.(*FD).Read+0x247			internal/poll/fd_unix.go:167
#	0x52f408	net.(*netFD).Read+0x28				net/fd_posix.go:56
#	0x53f964	net.(*conn).Read+0x44				net/net.go:183
#	0x787afe	net/http.(*connReader).backgroundRead+0x3e	net/http/server.go:672

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x7923b4 0x791fdd 0x1719827 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84			runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31			internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b		internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218			internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34				net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27				net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c				net/tcpsock.go:262
#	0x7923b3	net/http.(*Server).Serve+0x393				net/http/server.go:3002
#	0x791fdc	net/http.(*Server).ListenAndServe+0x7c			net/http/server.go:2931
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66	golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x7923b4 0x791fdd 0x17b8572 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84				runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31				internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b			internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218				internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34					net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27					net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c					net/tcpsock.go:262
#	0x7923b3	net/http.(*Server).Serve+0x393					net/http/server.go:3002
#	0x791fdc	net/http.(*Server).ListenAndServe+0x7c				net/http/server.go:2931
#	0x17b8571	knative.dev/pkg/metrics.newPrometheusExporter.func1+0x31	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/prometheus_exporter.go:55

1 @ 0x438556 0x430bb7 0x463445 0x4cf2d2 0x4d262c 0x4d2619 0x530bb5 0x5489e8 0x547bbd 0x879642 0xf71ab7 0x468dc1
#	0x463444	internal/poll.runtime_pollWait+0x84							runtime/netpoll.go:303
#	0x4cf2d1	internal/poll.(*pollDesc).wait+0x31							internal/poll/fd_poll_runtime.go:84
#	0x4d262b	internal/poll.(*pollDesc).waitRead+0x22b						internal/poll/fd_poll_runtime.go:89
#	0x4d2618	internal/poll.(*FD).Accept+0x218							internal/poll/fd_unix.go:402
#	0x530bb4	net.(*netFD).accept+0x34								net/fd_unix.go:173
#	0x5489e7	net.(*TCPListener).accept+0x27								net/tcpsock_posix.go:140
#	0x547bbc	net.(*TCPListener).Accept+0x3c								net/tcpsock.go:262
#	0x879641	google.golang.org/grpc.(*Server).Serve+0x361						google.golang.org/grpc@v1.42.0/server.go:779
#	0xf71ab6	knative.dev/net-kourier/pkg/envoy/server.(*XdsServer).RunManagementServer.func1+0x36	knative.dev/net-kourier/pkg/envoy/server/xds_server.go:87

1 @ 0x438556 0x4482f2 0x16b8ba9 0x468dc1
#	0x16b8ba8	github.com/patrickmn/go-cache.(*janitor).Run+0x88	github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079

1 @ 0x438556 0x4482f2 0x174fb79 0x468dc1
#	0x174fb78	go.opencensus.io/stats/view.(*worker).start+0xb8	go.opencensus.io@v0.23.0/stats/view/worker.go:276

1 @ 0x438556 0x4482f2 0x17b384a 0x468dc1
#	0x17b3849	knative.dev/pkg/metrics.(*MemStatsProvider).Start.func1+0xc9	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/memstats.go:375

1 @ 0x438556 0x4482f2 0x17b92ea 0x468dc1
#	0x17b92e9	knative.dev/pkg/metrics.startCleanup.func1+0xe9	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/metrics/resource_view.go:115

1 @ 0x438556 0x4482f2 0x17c3b05 0x468dc1
#	0x17c3b04	knative.dev/pkg/controller.(*twoLaneQueue).runConsumer+0x184	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/two_lane_queue.go:106

1 @ 0x438556 0x4482f2 0x7ed93b 0x7ee025 0x803f66 0x468dc1
#	0x7ed93a	google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x11a	google.golang.org/grpc@v1.42.0/internal/transport/controlbuf.go:407
#	0x7ee024	google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x84	google.golang.org/grpc@v1.42.0/internal/transport/controlbuf.go:534
#	0x803f65	google.golang.org/grpc/internal/transport.NewServerTransport.func2+0xc5	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:322

1 @ 0x438556 0x4482f2 0x80a585 0x468dc1
#	0x80a584	google.golang.org/grpc/internal/transport.(*http2Server).keepalive+0x244	google.golang.org/grpc@v1.42.0/internal/transport/http2_server.go:1119

1 @ 0x438556 0x4482f2 0x80ee16 0x80ecef 0x80fb92 0x4c8c5a 0x80fb05 0x80fadb 0x8761a7 0x876a06 0x87710e 0x88bae5 0xe937cc 0xeeeea5 0x468dc1
#	0x80ee15	google.golang.org/grpc/internal/transport.(*recvBufferReader).read+0x95										google.golang.org/grpc@v1.42.0/internal/transport/transport.go:178
#	0x80ecee	google.golang.org/grpc/internal/transport.(*recvBufferReader).Read+0x16e									google.golang.org/grpc@v1.42.0/internal/transport/transport.go:172
#	0x80fb91	google.golang.org/grpc/internal/transport.(*transportReader).Read+0x31										google.golang.org/grpc@v1.42.0/internal/transport/transport.go:484
#	0x4c8c59	io.ReadAtLeast+0x99																io/io.go:328
#	0x80fb04	io.ReadFull+0xa4																io/io.go:347
#	0x80fada	google.golang.org/grpc/internal/transport.(*Stream).Read+0x7a											google.golang.org/grpc@v1.42.0/internal/transport/transport.go:468
#	0x8761a6	google.golang.org/grpc.(*parser).recvMsg+0x46													google.golang.org/grpc@v1.42.0/rpc_util.go:559
#	0x876a05	google.golang.org/grpc.recvAndDecompress+0x65													google.golang.org/grpc@v1.42.0/rpc_util.go:690
#	0x87710d	google.golang.org/grpc.recv+0x6d														google.golang.org/grpc@v1.42.0/rpc_util.go:758
#	0x88bae4	google.golang.org/grpc.(*serverStream).RecvMsg+0x184												google.golang.org/grpc@v1.42.0/stream.go:1563
#	0xe937cb	github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3.(*aggregatedDiscoveryServiceStreamAggregatedResourcesServer).Recv+0x4b	github.com/envoyproxy/go-control-plane@v0.10.1/envoy/service/discovery/v3/ads.pb.go:317
#	0xeeeea4	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler.func1+0xa4							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:449

1 @ 0x438556 0x4482f2 0x8c73fb 0x8c7269 0x172fc59 0x172fbd2 0x172f36b 0x1738b84 0x1738a31 0x468dc1
#	0x8c73fa	k8s.io/apimachinery/pkg/util/wait.BackoffUntil+0x13a			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:167
#	0x8c7268	k8s.io/apimachinery/pkg/util/wait.JitterUntil+0x88			k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:133
#	0x172fc58	k8s.io/apimachinery/pkg/util/wait.Until+0xd8				k8s.io/apimachinery@v0.23.5/pkg/util/wait/wait.go:90
#	0x172fbd1	k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x51	k8s.io/client-go@v0.23.5/tools/leaderelection/leaderelection.go:268
#	0x172f36a	k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x16a	k8s.io/client-go@v0.23.5/tools/leaderelection/leaderelection.go:212
#	0x1738b83	knative.dev/pkg/leaderelection.(*runUntilCancelled).Run+0x43		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:309
#	0x1738a30	knative.dev/pkg/leaderelection.(*runAll).Run.func1+0x70			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:293

1 @ 0x438556 0x4482f2 0xeed394 0xeeedb4 0xef6512 0xef64e4 0xe936df 0x87f8da 0x880dc5 0x87ab58 0x468dc1
#	0xeed393	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process+0x573							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:224
#	0xeeedb3	github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler+0xf3							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/sotw/v3/server.go:463
#	0xef6511	github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamHandler+0x51							github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/v3/server.go:182
#	0xef64e3	github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamAggregatedResources+0x23						github.com/envoyproxy/go-control-plane@v0.10.1/pkg/server/v3/server.go:186
#	0xe936de	github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3._AggregatedDiscoveryService_StreamAggregatedResources_Handler+0x9e	github.com/envoyproxy/go-control-plane@v0.10.1/envoy/service/discovery/v3/ads.pb.go:298
#	0x87f8d9	google.golang.org/grpc.(*Server).processStreamingRPC+0xeb9										google.golang.org/grpc@v1.42.0/server.go:1540
#	0x880dc4	google.golang.org/grpc.(*Server).handleStream+0x9e4											google.golang.org/grpc@v1.42.0/server.go:1620
#	0x87ab57	google.golang.org/grpc.(*Server).serveStreams.func1.2+0x97										google.golang.org/grpc@v1.42.0/server.go:921

1 @ 0x438556 0x4482f2 0xf719d3 0x18e2632 0x468dc1
#	0xf719d2	knative.dev/net-kourier/pkg/envoy/server.(*XdsServer).RunManagementServer+0x432	knative.dev/net-kourier/pkg/envoy/server/xds_server.go:92
#	0x18e2631	knative.dev/net-kourier/pkg/reconciler/ingress.NewController.func12+0xb1	knative.dev/net-kourier/pkg/reconciler/ingress/controller.go:233

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x17196a7 0x17c191f 0x196b92d 0x1719827 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x17196a6	golang.org/x/sync/errgroup.(*Group).Wait+0x26			golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:40
#	0x17c191e	knative.dev/pkg/controller.StartAll+0x11e			knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:791
#	0x196b92c	knative.dev/pkg/injection/sharedmain.MainWithConfig.func2+0x2c	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/injection/sharedmain/main.go:283
#	0x1719826	golang.org/x/sync/errgroup.(*Group).Go.func1+0x66		golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x17387e9 0x17bfec9 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x17387e8	knative.dev/pkg/leaderelection.(*runAll).Run+0xa8		knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/leaderelection/context.go:296
#	0x17bfec8	knative.dev/pkg/controller.(*Impl).RunContext.func2+0x68	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/controller/controller.go:481

1 @ 0x438556 0x4493ec 0x4493c6 0x464b85 0x4744b1 0x1862aa6 0x468dc1
#	0x464b84	sync.runtime_Semacquire+0x24					runtime/sema.go:56
#	0x4744b0	sync.(*WaitGroup).Wait+0x70					sync/waitgroup.go:130
#	0x1862aa5	knative.dev/networking/pkg/status.(*Prober).Start.func3+0x25	knative.dev/networking@v0.0.0-20220524205304-22d1b933cf73/pkg/status/status.go:303

1 @ 0x462f25 0x173e915 0x173e72d 0x173b8ab 0x19053da 0x1905f8e 0x78e8cf 0x7901c9 0x1907396 0x791e3b 0x78d9a8 0x468dc1
#	0x462f24	runtime/pprof.runtime_goroutineProfileWithLabels+0x24	runtime/mprof.go:746
#	0x173e914	runtime/pprof.writeRuntimeProfile+0xb4			runtime/pprof/pprof.go:724
#	0x173e72c	runtime/pprof.writeGoroutine+0x4c			runtime/pprof/pprof.go:684
#	0x173b8aa	runtime/pprof.(*Profile).WriteTo+0x14a			runtime/pprof/pprof.go:331
#	0x19053d9	net/http/pprof.handler.ServeHTTP+0x499			net/http/pprof/pprof.go:253
#	0x1905f8d	net/http/pprof.Index+0x12d				net/http/pprof/pprof.go:371
#	0x78e8ce	net/http.HandlerFunc.ServeHTTP+0x2e			net/http/server.go:2047
#	0x7901c8	net/http.(*ServeMux).ServeHTTP+0x148			net/http/server.go:2425
#	0x1907395	knative.dev/pkg/profiling.(*Handler).ServeHTTP+0x35	knative.dev/pkg@v0.0.0-20220524202603-19adf798efb8/profiling/server.go:74
#	0x791e3a	net/http.serverHandler.ServeHTTP+0x43a			net/http/server.go:2879
#	0x78d9a7	net/http.(*conn).serve+0xb07				net/http/server.go:1930

@junydania
Copy link
Author

@nak3 any thoughts on this: #879

@nak3 nak3 mentioned this issue Jul 19, 2022
Closed
@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 17, 2022
@knative-prow-robot
Copy link

This issue or pull request is stale because it has been open for 90 days with no activity.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close

/lifecycle stale

@github-actions github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 17, 2022
@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 16, 2023
@nak3 nak3 mentioned this issue Jul 4, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nak3 @mbaynton @junydania @knative-prow-robot