AMQP traffic not reliably showing in UI

[engnatha]

Describe the bug
With v51.0.0 kubeshark installed, I'm seeing next to no traffic reported over AMQP. The reports are very intermittent and usually attributed to the wrong source or destination. As a quick demo, I have two services publishing to rabbitmq and one service subscribed. The service subscribed is showing that there are around 10 messages per second being received. The UI rarely shows anything. When it does show things, it's typically a basic deliver method. These typically show up as going between "kubernetes" and "kubernetes". Even more rarely I see a basic publish method that is attributed to the proper source and destination.

Provide more information
minikube

To Reproduce
Set up helm chart values

serviceAccount:
  create: true
tap:
  docker:
    tag: v51.0.0
  ingress:
    enabled: true
  tls: true
  servicemesh: false
  nokernelmodule: true

Install helm chart

helm install --version 51.0.0 -f ~/k8s_configs/kubeshark_helm_values.yaml kubeshark gsw-helm/kubeshark

Add minikube ingress controller

minikube addons enable ingress

Navigate to front end in web browser at http://ks.svc.cluster.local/
Observe little to no AMQP traffic

Expected behavior
One basic publish method in the UI per message produced by publishers. One basic deliver method in UI per message received by subscriber.

Logs
Worker logs typically show this

2023-10-20T18:13:06Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="30.349µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:09Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="34.72µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:12Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="69.287µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:15Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="41.119µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:18Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="70.716µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:24Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="47.007µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:31Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="19.597µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:38Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="22.415µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:40Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="32.295µs" method=GET path=/pcaps/total-size status_code=200
2023-10-20T18:13:52Z INF server/controllers/socket.go:40 > WebSocket read: error="websocket: close 1006 (abnormal closure): unexpected EOF"
2023-10-20T18:13:52Z INF server/controllers/socket.go:60 > WebSocket is closed!
2023-10-20T18:13:52Z INF server/middlewares/logger.go:69 > body_size=0 client_id=172.17.0.16 latency=1m0s method=GET path=/ws?node=minikube&q=amqp&worker=192.168.49.2%3A8897 status_code=200
2023-10-20T18:14:08Z INF server/middlewares/logger.go:69 > body_size=19 client_id=172.17.0.16 latency="20.401µs" method=GET path=/pcaps/total-size status_code=200

Hub logs

2023-10-20T18:15:11Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:16Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:19Z ERR main.go:67 > License update: error="Status code: 500"
2023-10-20T18:15:21Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:26Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:31Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:36Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:41Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:46Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:48Z ERR pkg/kubernetes/start.go:45 > K8s watch failure: error="error in kubectl pod watch"
2023-10-20T18:15:51Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:15:56Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:16:01Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:16:01Z INF server/controllers/socket.go:56 > WebSocket read: error="websocket: close 1006 (abnormal closure): unexpected EOF"
2023-10-20T18:16:01Z INF server/controllers/socket.go:154 > WebSocket client is closed!
2023-10-20T18:16:01Z ERR server/controllers/socket.go:116 > WebSocket client read: error="read tcp 172.17.0.16:60640->192.168.49.2:8897: use of closed network connection"
2023-10-20T18:16:01Z INF server/middlewares/logger.go:70 > body_size=0 client_id=172.17.0.21 latency=1m0s method=GET path=/ws?session-token=&refresh-token= status_code=200
2023-10-20T18:16:01Z INF server/controllers/socket.go:110 > Closed connection to the worker at: url=ws://192.168.49.2:8897/ws?node=minikube&q=amqp&worker=192.168.49.2%3A8897
2023-10-20T18:16:01Z ERR server/controllers/license.go:27 > error=EOF
2023-10-20T18:16:01Z INF server/middlewares/logger.go:70 > body_size=154 client_id=172.17.0.21 latency="113.876µs" method=GET path=/license status_code=200
2023-10-20T18:16:06Z ERR server/controllers/license.go:27 > error=EOF
2023-10-20T18:16:06Z INF server/middlewares/logger.go:70 > body_size=154 client_id=172.17.0.21 latency="144.658µs" method=GET path=/license status_code=200
2023-10-20T18:16:06Z INF server/controllers/socket.go:66 > Number of currently registered workers: value=1
2023-10-20T18:16:06Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:16:06Z INF server/controllers/socket.go:97 > Connecting to the worker at: url=ws://192.168.49.2:8897/ws?node=minikube&q=amqp&worker=192.168.49.2%3A8897
2023-10-20T18:16:06Z INF server/controllers/socket.go:105 > Connected to the worker at: url=ws://192.168.49.2:8897/ws?node=minikube&q=amqp&worker=192.168.49.2%3A8897
2023-10-20T18:16:08Z INF server/middlewares/logger.go:70 > body_size=157 client_id=172.17.0.21 latency="30.969µs" method=GET path=/pods/targeted?refresh-token= status_code=200
2023-10-20T18:16:08Z INF server/middlewares/logger.go:70 > body_size=19 client_id=172.17.0.21 latency="429.607µs" method=GET path=/pcaps/total-size?refresh-token= status_code=200
2023-10-20T18:16:11Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube
2023-10-20T18:16:16Z INF server/controllers/socket.go:74 > Iterating a worker host: host=192.168.49.2:8897 name=minikube

Screenshots
N/A

Desktop (please complete the following information):

• OS: Ubuntu 20.04 kernel 5.15.0-86-generic
• Web Browser: Chrome

Additional context
Resource utilization of the worker looked stable

I have tried turning off other applications at the same time to free up system resources. I have also tried adjusting the tap.regex setting in the chart to reduce the number of pods we're capturing traffic for to just those of interest.

[engnatha]

In case it's useful, the worker always starts up with a failure in tracer and then runs succesfully after a restart. This is the error it crashes with.

2023-10-20T18:20:30Z INF tracer/misc/data.go:20 > Set the data directory to: data-dir=data
2023-10-20T18:20:30Z INF tracer/main.go:41 > Starting tracer...
2023-10-20T18:20:30Z INF tracer/tracer.go:39 > Initializing tracer (chunksSize: 409600) (logSize: 4096)
2023-10-20T18:20:30Z INF tracer/tracer.go:53 > Detected Linux kernel version: 5.15.0-86-generic
2023-10-20T18:20:30Z INF tracer/pkg/pipe/impl.go:50 > Created a named pipe: name=data/pipe.log
2023-10-20T18:20:30Z INF tracer/pkg/pipe/impl.go:57 > Opened the named pipe: name=data/pipe.log
2023-10-20T18:20:30Z ERR tracer/main.go:75 > error="failed to create perf ring for CPU 0: can't mmap: operation not permitted"
panic: failed to create perf ring for CPU 0: can't mmap: operation not permitted

goroutine 1 [running]:
main.run()
	/app/tracer/main.go:49 +0x1b2
main.main()
	/app/tracer/main.go:37 +0x4bf

[engnatha]

Here's an example of it not attributing source and destination correctly. This is supposed to be a publish from one of my services to the message broker.

[engnatha]

It's still not what I'm expecting, but things do seem to show up more reliably if I put kubeshark in its own namespace on install.