KubeMQ binds only on IPv6 interfaces

[rweland]

My Setup:
1 Node k3s deployment on IPv4 ONLY lab network
Other workloads up and working as expected
KubeMQ Community latest release (as of 2/25/22) deployed using default config

What happens:
When the community edition is deployed it starts, however all listeners only bind to the containers IPv6 address. This makes it impossible to connect to the container with the CLI from an IPv4 network.

What's expected:
Listeners should bind to both IPv4 and IPv6 interfaces.

Digging in the code, the raw TCP socket listeners appear to be correctly setting the bind port according to the net.Listen documentation. As well the echo servers appear to be setup correctly as well.
Ref: grpc/server.go:111
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
Ref: rest/server.go:172
_ = s.echoWebServer.Start(fmt.Sprintf(":%d", conf.Port))

However as seen from the startup log output, it appears to ignore this.

2022-02-26T02:07:21.322Z	INFO	starting kubemq community edition version: , cores: 16	{"host": "f1a8683371dc", "module": "server"}
2022-02-26T02:07:21.323Z	INFO	⇨ http server started on [::]:8080
	{"host": "f1a8683371dc", "module": "server-api"}
2022-02-26T02:07:22.324Z	INFO	Starting broker-server version 2.1.9	{"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z	INFO	Git commit [not set]	{"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z	INFO	Listening for client connections on 0.0.0.0:39983	{"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z	INFO	Server id is NAT3D4HT53N6MRBB4T35ZVRXEPEC3CSRAWKTVZ5JXGCI3TWT2772BX2E	{"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.324Z	INFO	Server is ready	{"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.325Z	INFO	In-Memory client server connection pipe: ya3PsP8PSdGJarJJX9lIIC started	{"host": "f1a8683371dc", "module": "broker"}
2022-02-26T02:07:22.350Z	INFO	STREAM: Starting broker-streaming-server[kubemq] version 0.19.0	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.350Z	INFO	STREAM: ServerID: 08rN2ijHYGp37AYGf7AYqI	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.350Z	INFO	STREAM: Go version: go1.17.7	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.350Z	INFO	STREAM: Git commit: [not set]	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.352Z	INFO	STREAM: Recovering the state...	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.352Z	INFO	STREAM: No recovered state	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM: Message store is FILE	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM: Store location: store/f1a8683371dc	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM: ---------- Store Limits ----------	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM: Channels:            unlimited	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM: --------- Channels Limits --------	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM:   Subscriptions:     unlimited	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM:   Messages     :     unlimited	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM:   Bytes        :     unlimited	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM:   Age          :       24h0m0s	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM:   Inactivity   :       24h0m0s	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM: ----------------------------------	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:22.603Z	INFO	STREAM: Streaming Server is ready	{"host": "f1a8683371dc", "module": "broker-store-server"}
2022-02-26T02:07:23.603Z	INFO	routing service loaded	{"host": "f1a8683371dc", "module": "services"}
2022-02-26T02:07:23.606Z	WARN	started insecure grpc server at port 50000	{"host": "f1a8683371dc", "module": "grpc"}
2022-02-26T02:07:23.606Z	WARN	grpc interface is accepting traffic	{"host": "f1a8683371dc", "module": "grpc"}
2022-02-26T02:07:23.606Z	WARN	rest interface is accepting traffic	{"host": "f1a8683371dc", "module": "rest-interface"}
2022-02-26T02:07:23.606Z	INFO	started insecure rest server at port 9090	{"host": "f1a8683371dc", "module": "rest-interface"}
2022-02-26T02:07:23.606Z	INFO	⇨ http server started on [::]:9090
	{"host": "f1a8683371dc", "module": "rest-api"}
2022-02-26T02:07:23.606Z	INFO	STREAM: Channel "_QUEUE_DELAY_" has been created	{"host": "f1a8683371dc", "module": "broker-store-server"}

This is confirmed by remoting into the container and listing the bound ipv4 and ipv6 ports
Open IPv4 ports

I have no name!@f1a8683371dc:/kubemq$ declare -a array=($(tail /proc/net/tcp | cut -d":" -f"3"|cut -d" " -f"1")) && for port in ${array[@]}; do echo $((0x$port)); done | sort | uniq     
57560

Open IPv6 ports

I have no name!@f1a8683371dc:/kubemq$ declare -a array=($(tail /proc/net/tcp6 | cut -d":" -f"3"|cut -d" " -f"1")) && for port in ${array[@]}; do echo $((0x$port)); done | sort | uniq    
36742
39983
42349
50000
8080
9090

Things I've tried:

• Rebuild container from source
• Rebuild with different base image (bullseye-slim)
• Hardcoded address/ports (ex 0.0.0.0:50000 for grpc)
• Hardcoded ListenerNetwork for echo servers to tcp

The only thing that has successfully allowed the project to bind on ipv4 was to set the first param of Listen and the ListenerNetwork property of the echo servers to tcp4, which of course forced IPv4 only.

2022-02-26T03:00:41.577Z INFO ⇨ http server started on 0.0.0.0:9090

I've run out of things to try to allow both ipv4 and ipv6, any ideas?

[rweland]

I found this post in the golang issues golang/go#9334 which seems to be a long standing issue with the net package of golang. The issue doesn't appear to be going anywhere at the moment.

I found a similar issue in echo servers history labstack/echo#1658. Their solution was to add the aforementioned configuration setting I referenced in the issue. I think that may be the solution to go with.

I have put in a PR #21 to add transport as a config setting for api, grpc, and rest if that is the direction we want to take.

[kubemq]

PR Merged