Kafka L7 visibility policy causes connection errors due to parser errors

[chancez]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

When using Kafka L7 visibility policies, I'm seeing my Kafka consumers get disconnected roughly every 30-60 seconds. I enabled debug logging and verbose envoy logs and was able to find the error related to the disconnect, it seems the Kafka parser is encountering an error and closing the connection as a result.

Cilium Version

I'm using a recent master build, but this is present in all versions I've tried.

Kernel Version

Linux lima-docker 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 12:56:31 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Kubernetes Version

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"clean", BuildDate:"2022-09-01T23:50:12Z", GoVersion:"go1.18.5", Compiler:"gc", Platform:"linux/arm64"}

Sysdump

cilium-sysdump-20221019-110146.zip

Relevant log output

level=debug msg="time=\"2022-10-19T17:59:56Z\" level=debug msg=\"Ignoring Kafka message apiKey=8,apiVersion=5,len=156: null due to parse error\" error=\"invalid array length\"" subsys=envoy-filter threadID=638
level=debug msg="time=\"2022-10-19T17:59:56Z\" level=warning msg=\"Unable to parse Kafka request; closing Kafka connection\" error=\"invalid array length\"" subsys=envoy-filter threadID=638

Anything else?

I have a demo app to reproduce but it's private. PM me if you need a way to reproduce the error.

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[daviddyball]

Can confirm this behaviour on my own Cilium 1.12.7 deployment.

[caiobegotti]

Got here through #25489 as I'm seeing a similar issue with Kafka L7 policies with Cilium 1.13.1.

[daviddyball]

Is this an issue with envoy or with Cilium itself? Do we have to wait for upstream envoy to resolve an issue there, or is it Cilium's use of envoy?

[chancez]

@daviddyball The Kafka protocol handling is custom implemented envoy filter, so it's something we need to resolve. I believe this is the current implementation of the filter: https://github.com/cilium/proxy/tree/main/proxylib/kafka

[alexpirtea-flowx]

Got the same issue on version 1.14.4
Turns out the culprit is the kafka version
Example:
docker.io/wurstmeister/kafka:1.1.0 - works
docker.io/bitnami/kafka:3.4.1-debian-11-r0 - reproduces issue