ipv6 validation

[xaleeks]

Have been getting lots of pings from Telco&Cloud customers who would like to run Contour on ipv6 environments, this means both Single Stack and Dual Stack k8s clusters. We need to validate Contour deployments on both Tanzu k8s products as well on DIY cloud platforms. Before identifying the exact set of stipulations for 'Contour support ipv6', the very basic scenario we have in mind is:

We assume the entire k8s cluster is running on ipv6 (assuming worker nodes running ipv6, pods have ipv6 connectivity with each other & to the internet, an ipv6 capable CNI, an ipv6 capable external Loadbalancer), Contour should run as usual and brings traffic into pods running on ipv6 addresses. And the same functionality for pods with dual ipv4 / ipv6 addresses.

timeframe: v1.13 or v1.14 before May

[youngnick]

I think that all the conditions I can think of for this to work are:

• Contour must be reachable from Envoy, currently we supply the DNS name contour for Envoy to connect back to Contour. In an IPv6 world, this will be a AAAA lookup, so the DNS must support AAAA for service records (since the contour name is the name of the Service).
• Contour must be able to publish the endpoints of the pods to route to to Envoy.
• Envoy must be reachable on its serving address (aka the "outside" address). So you'll either need a Service of type: LoadBalancer (what we do now), or some equivalent functionality that works with IPv6.
• Envoy must be able to route the traffic to the running pods. So the dynamic IPv6 routing must be working in the cluster.

I think that any cluster that's actually working in IPv6 only mode should probably work. I don't know about dual-stack, it's possible we may need to do some work with Endpoints, I haven't looked at dual-stack enough to be sure.

In any case, the steps from here seem to be the same:

• Test Contour on an IPv6-only cluster.
• Test Contour on a dual-stack cluster.

Ideally we figure out a way to run the current set of integration tests in both those cases.

[sunjayBhatia]

What do we think about running some validation (the integration test suite probably) against an ipv6 cluster in CI and writing a guide on what all the flags to flip are to support a cluster with ipv6 only?

Initially seems like some bits to do around Cluster discovery (see #3564) and how we parse ipv6 addresses consistently, there are some rough edges where some flags need to take :: but others need [::] etc.

[xaleeks]

I’m going to tag this v1.15 to account for the ongoing investigation work, not to mean to be delivered in 1.15.

[sunjayBhatia]

See envoyproxy/ratelimit#245

Ratelimit server doesnt work with ipv6 so that integration test is blocked

[sunjayBhatia]

The last bits of this are for running a job in CI to run integration tests against an ipv6 only cluster, what do we think about moving this to the 1.16 milestone?

[sunjayBhatia]

ratelimit PR to allow server to listen on any address properly: envoyproxy/ratelimit#252

[sunjayBhatia]

to save this for later, this is the diff needed to fully configure kind etc. with ipv6 to run the integration tests

diff --git a/_integration/testsuite/kind-expose-port.yaml b/_integration/testsuite/kind-expose-port.yaml
index 17be4d1a..456ff6cd 100644
--- a/_integration/testsuite/kind-expose-port.yaml
+++ b/_integration/testsuite/kind-expose-port.yaml
@@ -1,12 +1,14 @@
 kind: Cluster
 apiVersion: kind.x-k8s.io/v1alpha4
+networking:
+  ipFamily: ipv6
 nodes:
 - role: control-plane
 - role: worker
   extraPortMappings:
   - containerPort: 80
     hostPort: 9080
-    listenAddress: "0.0.0.0"
+    listenAddress: "::"
   - containerPort: 443
     hostPort: 9443
-    listenAddress: "0.0.0.0"
+    listenAddress: "::"
diff --git a/examples/contour/03-contour.yaml b/examples/contour/03-contour.yaml
index ef55bc0d..702484ff 100644
--- a/examples/contour/03-contour.yaml
+++ b/examples/contour/03-contour.yaml
@@ -38,8 +38,14 @@ spec:
       - args:
         - serve
         - --incluster
-        - --xds-address=0.0.0.0
+        - '--xds-address=::'
         - --xds-port=8001
+        - '--stats-address=::'
+        - '--debug-http-address=::1'
+        - '--http-address=::'
+        - '--envoy-service-http-address=::'
+        - '--envoy-service-https-address=::'
+        - '--health-address=::'
         - --contour-cafile=/certs/ca.crt
         - --contour-cert-file=/certs/tls.crt
         - --contour-key-file=/certs/tls.key
diff --git a/examples/contour/03-envoy.yaml b/examples/contour/03-envoy.yaml
index 36ca2256..ba76af5f 100644
--- a/examples/contour/03-envoy.yaml
+++ b/examples/contour/03-envoy.yaml
@@ -99,6 +99,7 @@ spec:
       - args:
         - bootstrap
         - /config/envoy.json
+        - --admin-address=::1
         - --xds-address=contour
         - --xds-port=8001
         - --xds-resource-version=v3

Also when running the integration-tester test suite, the ADDRESS env var has to be set to [::1], the go version will be better, the base url will be a proper ipv6 address

[sunjayBhatia]

Also httpbin only listens on ipv4 which is somewhat problematic: https://github.com/postmanlabs/httpbin/blob/f8ec666b4d1b654e4ff6aedd356f510dcac09f83/Dockerfile#L22

this diff was needed to get the httpbin fixture to be usable:

diff --git a/_integration/testsuite/fixtures/httpbin.yaml b/_integration/testsuite/fixtures/httpbin.yaml
index 24d031a1..4b979aa1 100644
--- a/_integration/testsuite/fixtures/httpbin.yaml
+++ b/_integration/testsuite/fixtures/httpbin.yaml
@@ -29,6 +29,13 @@ spec:
       containers:
       - name: httpbin
         image: docker.io/kennethreitz/httpbin
+        args:
+        - -b
+        - '[::]:80'
+        - httpbin:app
+        - -k
+        - gevent
+        command: ["gunicorn"]
         ports:
         - name: http
           containerPort: 80

[sunjayBhatia]

still waiting for some upstream things (ratelimit CI to push an image) and running e2e tests against an ipv6 cluster might be something we consider doing as part of our expanded testing efforts, removing from 1.16.0 for now