Online Boutique does not work on IPv6 single stack kubernetes cluster

[naoki9911]

Describe the bug

The Online Boutique page shows Internal Server Error.
Online Boutique is deployed to IPv6 single stack cluster.

To Reproduce

1. Create IPv6 single stack cluster with Calico.
2. Deploy Online Boutique and enable port-forwaring.
3. Access to the deployed service and the page shows Internal Server Error.

Logs

Logs for frontend

{"message":"Tracing disabled.","severity":"info","timestamp":"2022-11-01T10:11:21.144638627Z"}
{"message":"Profiling disabled.","severity":"info","timestamp":"2022-11-01T10:11:21.144921177Z"}
{"message":"starting server on :8080","severity":"info","timestamp":"2022-11-01T10:11:21.146948308Z"}
{"message":"Failed to fetch the name of the cluster in which the pod is runningGet \"http://169.254.169.254/computeMetadata/v1/instance/attributes/cluster-name\": dial tcp 169.254.169.254:80: connect: network is unreachable","severity":"error","timestamp":"2022-11-01T10:11:21.150043979Z"}
{"message":"Failed to fetch the Zone of the node where the pod is scheduledGet \"http://169.254.169.254/computeMetadata/v1/instance/zone\": dial tcp 169.254.169.254:80: connect: network is unreachable","severity":"error","timestamp":"2022-11-01T10:11:21.150230692Z"}
{"cluster":"","hostname":"frontend-7966dc59fb-n5zkh","message":"Loaded deployment details","severity":"debug","timestamp":"2022-11-01T10:11:21.150254973Z","zone":""}
{"http.req.id":"7b74fd6d-b0b7-4dd9-94a4-88de48f1122c","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.410934568Z"}
{"http.req.id":"a23de3f6-3b0c-4236-8268-c8afd36b4088","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.41088243Z"}
{"http.req.id":"7b74fd6d-b0b7-4dd9-94a4-88de48f1122c","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.412115234Z"}
{"http.req.id":"a23de3f6-3b0c-4236-8268-c8afd36b4088","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.412203078Z"}
{"http.req.id":"6c9b5433-fd25-4c00-9825-0d834122d8d0","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.409260277Z"}
{"http.req.id":"e19656c0-1cd5-4585-bf22-e1f95aa80bdc","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.409328551Z"}
{"http.req.id":"e19656c0-1cd5-4585-bf22-e1f95aa80bdc","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.410027627Z"}
{"http.req.id":"6c9b5433-fd25-4c00-9825-0d834122d8d0","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.410502244Z"}
{"http.req.id":"6022bf9e-79f9-4cfd-9c24-f74c6840a9e5","http.req.method":"GET","http.req.path":"/","message":"request started","session":"dbd99d36-9590-4583-a910-5bb01b7e7db1","severity":"debug","timestamp":"2022-11-01T10:11:58.038227138Z"}
{"currency":"USD","http.req.id":"6022bf9e-79f9-4cfd-9c24-f74c6840a9e5","http.req.method":"GET","http.req.path":"/","message":"home","session":"dbd99d36-9590-4583-a910-5bb01b7e7db1","severity":"info","timestamp":"2022-11-01T10:11:58.038483621Z"}
{"http.req.id":"d91d0fa8-23d8-4df2-8e02-447c683841ef","http.req.method":"GET","http.req.path":"/","message":"request started","session":"77e22e41-1f84-437e-ae7e-1ce0407fc9c9","severity":"debug","timestamp":"2022-11-01T10:11:59.026430074Z"}
{"currency":"USD","http.req.id":"d91d0fa8-23d8-4df2-8e02-447c683841ef","http.req.method":"GET","http.req.path":"/","message":"home","session":"77e22e41-1f84-437e-ae7e-1ce0407fc9c9","severity":"info","timestamp":"2022-11-01T10:11:59.026580378Z"}
{"error":"could not retrieve currencies: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [(here is ipv6 address)]:7000: connect: connection refused\"","http.req.id":"6022bf9e-79f9-4cfd-9c24-f74c6840a9e5","http.req.method":"GET","http.req.path":"/","message":"request error","session":"dbd99d36-9590-4583-a910-5bb01b7e7db1","severity":"error","timestamp":"2022-11-01T10:11:59.045619196Z"}

Screenshots

Environment

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.25.3

Additional context

The cause of this problem seems to be that some services' gRPC servers are only listening on IPv4.

microservices-demo/src/currencyservice/server.js

Line 173 in 573f4db

`0.0.0.0:${PORT}`,

microservices-demo/src/emailservice/email_client.py

Line 26 in 573f4db

channel = grpc.insecure_channel('0.0.0.0:8080')

microservices-demo/src/paymentservice/server.js

Line 67 in 573f4db

`0.0.0.0:${port}`,

Exposure

This would affect anyone using IPv6 single stack cluster.

[NimJay]

@naoki9911 first off, wow, thank you for this super clear description of your issue!

Question(s)
I haven't assigned a high priority to this issue because I'm guessing IPv6 single-stack isn't super common yet (among Kubernetes users).
Do you know if this a false assumption?
For instance, do you have data that suggests that a lot of networks (used in production by Kubernetes users) disable IPv4?

[mathieu-benoit]

Out of curiosity @naoki9911, what could be the recommendation for the 3 statements you highlighted with 0.0.0.0?

• Do you think we could replace this value by [::] to be generic and support both ipv4 and ipv6?
• Or we should update the code with kind of a if ipv4 do 0.0.0.0 else [::]?

Would love to hear your insights regarding this, thanks in advance!

[naoki9911]

Thanks for your replys!

I haven't assigned a high priority to this issue because I'm guessing IPv6 single-stack isn't super common yet (among Kubernetes users).
Do you know if this a false assumption?

I completely agree with you. I think IPv6 single-stack cluster is very unusual, too.
I tried IPv6 single-stack for my experiments and I'm not in a hurry at all about this issue.

This issue just intends to inform you that some services are not supporting IPv6 :)
If you don't mind, I hope this issue is opened until services completely support IPv6.

what could be the recommendation for the 3 statements you highlighted with 0.0.0.0?

I prefer replacing 0.0.0.0 with [::] to support both IPv4 and IPv6.
But I'm not an expert on IPv6 and Kubernetes, so I don't know if it will not cause any problems.
It may allow unintended IPv6 communication in IPv4 dedicated cluster.

[mathieu-benoit]

Hey @naoki9911, if you have time and still have access to your single-stack IPv6 cluster, could you please give #1340 a try and report back what you will find there? Thanks in advance!

[naoki9911]

@mathieu-benoit
Thank you for your great work!
I will try the patch.