Mizu 0.22.0 + Istio 1.10 + mTLS not showing GRPC traffic

[derrickburns]

Any ideas?

[nimrod-up9]

Hi @derrickburns, thanks for submitting an issue.
I'm checking it out. Will update as soon as possible.

[nimrod-up9]

I can confirm that there is a reproducible issue with Mizu 0.22.16 and Istio 1.11.
Working to fix it.

@derrickburns, if you can reproduce the issue and upload the logs, that would be very helpful. It is done by running mizu tap with --set dump-logs=true. The logs are saved to a zip in ~/.mizu dir.

[derrickburns]

So, strangely, this test started to report GRPC requests then stopped.

Thanks for investigating. This is a nice tool!

$ ~/bin/mizu tap -A --service-mesh  qgs-minerva\.\* --set dump-logs=true
Mizu will store up to 200MB of traffic, old traffic will be cleared once the limit is reached.
Tapping pods in all namespaces
+qgs-minerva-85dc9d7589-4td9x
+qgs-minerva-85dc9d7589-d82x2
+qgs-minerva-85dc9d7589-z4jtw
Waiting for Mizu Agent to start...
Mizu is available at http://localhost:8899
t^CYou can find the zip file with all logs in /Users/derrick.burns/.mizu/mizu_logs_2022_01_20__21_08_07.zip

mizu_cli.log
mizu_logs_2022_01_20__21_08_07.zip

[peter-dolkens]

So, strangely, this test started to report GRPC requests then stopped.

I had similar while testing this tool but for regular http traffic. Had a simple service with a basic http healthcheck hit every 30 seconds. It reported for 4 minutes, then just stopped. Pausing and resuming the stream resumed streaming again.

[nimrod-up9]

There is an issue which causes Mizu to loose track of long-living HTTP and gRPC connections after about 10 seconds, not just in Istio. A fix will be released at the start of next week.

Stopping after 4 minutes is probably a different bug.
@peter-dolkens, could you share logs?

[peter-dolkens]

I'll keep an eye out for it. Unfortunately I wasn't logging at the time. Will raise a new ticket if it happens again

[nimrod-up9]

There was an issue which caused Mizu to stop tapping gRPC connections after 10 seconds. The issue was fixed in 0.23.5 c2739a6. Will make its way into stable version 0.24.0 scheduled to be released in the next few hours.

Please let us know if you find the gRPC tapping works smoother or if the issue persists.

Also note that Mizu can only tap gRPC if it is up before the connection is formed (Due to HTTP/2 header compression). The most convenient way to get Mizu to tap gRPC is to tap only the gRPC server and restart the gRPC client after running the Mizu.

[nimrod-up9]

Version 0.24.0 released. gRPC tapping should work smoother now.
Closing this issue.

Thank you for reporting. Please feel free to comment here / open new issues if there is a need.
@derrickburns @peter-dolkens