Gateway not able to parse ipv6 addresses

[shishirkh]

Describe the bug
I am trying to get a 1 gateway & 3 broker setup done on an env with ipv6.

I am getting this in each broker, so I know they are starting correctly.

2020-12-02 07:38:26.429 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-2 succeeded. Started 14 steps in 16049 ms.

however when I try to exec inside the broker and try "zbctl status" or "zbctl status --insecure" I keep getting this error

kubectl exec -it z2943-broker1-01 /bin/bash
root@z2943-broker1-01:/usr/local/zeebe# zbctl status
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
root@z2943-broker1-01:/usr/local/zeebe# zbctl status --insecure
Error: rpc error: code = Unknown desc = 

If I try to specify the pod address...

root@z2943-broker1-01:/usr/local/zeebe# zbctl status --address 240b:c0e0:202:5e2b:b424:2:0:51f
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: address tcp/c0e0: unknown port"
root@z2943-broker1-01:/usr/local/zeebe# zbctl status --address [240b:c0e0:202:5e2b:b424:2:0:51f]
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: address [240b:c0e0: missing ']' in address"

My guess is that the brokers are not able to differentiate between a ipv4 and ipv6 address.

To Reproduce
1 gateway with 3 clusters with the below settings -

for broker - > image version is 0.25.1
env vars are ->
ZEEBE_BROKER_CLUSTER_NODEID: '1' #different 0,1,2 for each of the three brokers.
ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT: '3'
ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR: '3'
ZEEBE_BROKER_CLUSTER_CLUSTERSIZE: '3'
ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS: '{{APP_NAME}}-broker1-cl-0:26502,{{APP_NAME}}-broker2-cl-0:26502,{{APP_NAME}}-broker3-cl-0:26502'
ZEEBE_BROKER_CLUSTER_CLUSTERNAME: 'zeebe-cluster'
ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT: '26501'
ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT: '26502'
ZEEBE_BROKER_NETWORK_MONITOIRNGAPI_PORT: '9600'

for gateway -> image version is 0.25.1
env vars are ->
ZEEBE_STANDALONE_GATEWAY: 'true'
ZEEBE_GATEWAY_NETWORK_HOST: '0.0.0.0'
ZEEBE_GATEWAY_NETWORK_PORT: '26500'
ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT: '{{APP_NAME}}-broker1-cl-0:26502'
ZEEBE_GATEWAY_CLUSTER_CLUSTERNAME: 'zeebe-cluster'
ZEEBE_GATEWAY_CLUSTER_MEMBERID: '3'
ZEEBE_GATEWAY_CLUSTER_HOST: '0.0.0.0'
ZEEBE_GATEWAY_CLUSTER_PORT: '26502'

Expected behavior
Gateway should be able to connect to the brokers and I should be able to use zbctl status to see the brokers,replications,partitions.

Log/Stacktrace
This error comes up inside the gateway

2020-12-02 07:45:11.612 [] [grpc-default-executor-0] ERROR io.grpc.internal.SerializingExecutor - Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@501e95a8
java.lang.NumberFormatException: For input string: "c0e0"
	at java.lang.NumberFormatException.forInputString(Unknown Source) ~[?:?]
	at java.lang.Integer.parseInt(Unknown Source) ~[?:?]
	at java.lang.Integer.parseInt(Unknown Source) ~[?:?]
	at io.zeebe.gateway.EndpointManager.addBrokerInfo(EndpointManager.java:78) ~[zeebe-gateway-0.25.1.jar:0.25.1]
	at io.zeebe.gateway.EndpointManager.lambda$topology$1(EndpointManager.java:253) ~[zeebe-gateway-0.25.1.jar:0.25.1]
	at org.agrona.collections.IntArrayList.forEach(IntArrayList.java:588) ~[agrona-1.8.0.jar:1.8.0]
	at io.zeebe.gateway.EndpointManager.topology(EndpointManager.java:250) ~[zeebe-gateway-0.25.1.jar:0.25.1]
	at io.zeebe.gateway.GatewayGrpcService.topology(GatewayGrpcService.java:135) ~[zeebe-gateway-0.25.1.jar:0.25.1]
	at io.zeebe.gateway.protocol.GatewayGrpc$MethodHandlers.invoke(GatewayGrpc.java:1532) ~[zeebe-gateway-protocol-impl-0.25.1.jar:0.25.1]
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) ~[grpc-stub-1.33.0.jar:1.33.0]
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) ~[grpc-core-1.33.0.jar:1.33.0]
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:814) ~[grpc-core-1.33.0.jar:1.33.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.33.0.jar:1.33.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.33.0.jar:1.33.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

My pods are

kubectl get pods -o wide | grep z2943
z2943-broker1-01                        1/1     Running            0          54m     240b:c0e0:202:5e2b:b424:2:0:51f
z2943-broker2-01                        1/1     Running            0          54m     240b:c0e0:202:5e2b:b424:2:0:37b
z2943-broker3-01                        1/1     Running            0          54m     240b:c0e0:202:5e2b:b424:2:0:c56
z2943-gateway-01                        1/1     Running            0          53m     240b:c0e0:202:5e2b:b424:2:0:af3

Environment:

• OS: kubernetes - ipv6
• Zeebe Version: 0.25.1
• Configuration: 1 gateway with 3 brokers

[npepinpe]

I can confirm this - we assume in some parts that we will use ipv4 addresses. This is an implicit assumption, where we expect addresses to be defined as host:port, but an ipv6 address will have many : 🙂

[shishirkh]

Thanks for the prompt replies here and on the slack channel. @npepinpe
Can you give an ETA on when the zeebe team will be able to work on it and make zeebe IPV6 compatible?

[npepinpe]

Prioritized as low as our current focus is increasing stability for Camunda Cloud, which doesn't use ipv6 - however we'd be happy to accept a PR here, and I hope we eventually support ipv6.

[aivinog1]

@npepinpe Hi! I would like to take this one :)

[npepinpe]

Go for it! I only had a brief look, and the problem seems mostly that we sometimes parse addresses by doing hostString.split(":"), which is obviously wrong for IPv6. Ideally we should also convert to InetAddress or InetSocketAddress and then get the host/port from there. AFAIK all our services should work with InetSocketAddress (some Atomix stuff wants Address, but afaik this already works with IPv6, though we should double check that).

[aivinog1]

Hey @npepinpe! I have a question about this: since this is a bug should I backport this to previous versions? I will create a backport PRs to be sure.