Add options for configuring IPv4 and IPv6 support with Calico #11688

hakman · 2021-06-04T14:01:33Z

Sample config to get started with AWS, Calico and IPv6, using a kOps cluster with public topology:

  kubeAPIServer:
    bindAddress: "::"
  nonMasqueradeCIDR: fd00:10:96::/64
  serviceClusterIPRange: fd00:10:96::/108
  kubeControllerManager:
    clusterCIDR: fd00:10:96::/80
    allocateNodeCIDRs: false
  kubeDNS:
    provider: CoreDNS
    upstreamNameservers:
    - 2620:119:35::35
    - 2620:119:53::53
  networking:
    calico:
      ipv4Support: false
      ipv6Support: true
      awsSrcDstCheck: Disable
  subnets:
   - type: Public
     ipv6CIDR: 2a05:d014:...::/64

Ref: #8432

/cc @justinsb @johngmyers @aojea

aojea · 2021-06-04T14:36:25Z

is there ipv6 CI job?

hakman · 2021-06-04T14:45:34Z

is there ipv6 CI job?

Not yet. There are a few things left on another PR and after that will try to get a CI job to run.

hakman · 2021-06-04T15:20:51Z

/retest

johngmyers

Not blocking, but it would be good if we could automatically set the IPv*Support fields based on a cluster-level setting as to whether the cluster is v4-only, dual-stack, or v6-only.

johngmyers · 2021-06-05T21:12:13Z

upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.16.yaml.template

+            {{- if (WithDefaultBool .Networking.Calico.IPv6Support false) }}
+            - name: CALICO_IPV6POOL_CIDR
+              value: "{{ .KubeControllerManager.ClusterCIDR }}"
+            - name: CALICO_IPV6POOL_NAT_OUTGOING


Why do we need to NAT outgoing IPv6? Could we not assign a routable CIDR to the pool? Or does that require mucking about with ENI mode?

There are a few considerations here:

default behavior for Calico is to do outgoing NAT

this is a setting that will be applied for the default IPv6 Pool created on first start.

I don't see how assigning a routable CIDR would be possible until AWS adds a routable IPv6 CIDR blocks to each node.
With these in mind, seems a safe default for now. It can easily be changed after initial start and, once there is some way to use routable CIRDS, it can be split into a separate option.

johngmyers · 2021-06-05T21:24:16Z

upup/models/cloudup/resources/addons/coredns.addons.k8s.io/k8s-1.12.yaml.template

@@ -84,7 +84,7 @@ data:
          ttl 30
        }
        prometheus :9153
-        forward . /etc/resolv.conf {
+        forward . {{ or (join KubeDNS.UpstreamNameservers " ") "/etc/resolv.conf" }} {


Is this an unrelated change?

Okay, missing feature from Coredns that you need in order to not muck with the node-level DNS? This should perhaps be a separate PR.

I thought about making this a separate PR, but it was a one liner, related to current work.
As AWS DNS servers don't support IPv6, this was the easiest way to get CoreDNS to work.

johngmyers · 2021-06-05T21:34:06Z

upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.16.yaml.template

@@ -3857,8 +3861,17 @@ spec:
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
+            {{- if (WithDefaultBool .Networking.Calico.IPv6Support false) }}
+            - name: CALICO_IPV6POOL_CIDR
+              value: "{{ .KubeControllerManager.ClusterCIDR }}"


According to https://kubernetes.io/docs/concepts/services-networking/dual-stack/ the KCM --cluster-cidr is, in dual-stack, an IPv4 CIDR and an IPv6 CIDR, separated by a comma. It doesn't look like calico/node accepts that syntax in either CALICO_IPV6POOL_CIDR or CALICO_IPV4POOL_CIDR.

I think we'd need a template function to parse ClusterCIDR into its two components.

Actually, I'm okay with not supporting dual-stack in the ClusterCIDR. We should probably have API validation around that, though.

For now, single stack ClusterCIDR is ok from my point of view too.
Dual-stack is also blocked by kubernetes/cloud-provider-aws#230, as the kubelet expects the cloud provider to return both IPv4 and IPv6 for the node.

upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.16.yaml.template

johngmyers · 2021-06-05T22:14:13Z

I think the only thing blocking is API validation to prohibit the combination of Calico and dual-stack ClusterCIDR.

johngmyers · 2021-06-05T22:16:53Z

Okay, existing ClusterCIDR validation prohibits multiple CIDRs.

johngmyers

All of these comments can be followup work.

k8s-ci-robot · 2021-06-05T22:25:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johngmyers

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [johngmyers]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

johngmyers · 2021-06-05T23:43:50Z

Suggest API validation to prohibit enabling both IPv4 and IPv6 for Calico.

johngmyers · 2021-06-05T23:45:59Z

Hrm, perhaps a single, tri-state setting? With values 'ipv4', 'ipv6', 'dual', default 'ipv4'.

johngmyers · 2021-06-05T23:47:13Z

Or key off of how ClusterCIDR parses?

hakman · 2021-06-06T04:55:02Z

Or key off of how ClusterCIDR parses?

I think this may be a good way to pull the defaults, but would sill allow overriding for experiments.

hakman · 2021-06-06T05:20:12Z

In other news, I ran the Conformance tests manually on a fresh cluster and got the following failures:

[Fail] [sig-network] Services [It] should have session affinity work for NodePort service [LinuxOnly] [Conformance]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:203

[Fail] [sig-network] Services [It] should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:203

[Fail] [sig-network] EndpointSlice [It] should create Endpoints and EndpointSlices for Pods matching a Service [Conformance]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/endpointslice.go:320

[Fail] [sig-network] HostPort [It] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:113

[Fail] [sig-network] Services [It] should have session affinity timeout work for NodePort service [LinuxOnly] [Conformance]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:203

These are happening because the tests try to connect to the IPv4 address of the service exposed via NodePort. If the tests would connect to the IPv6 address instead, they would pass (tested by setting --node-ip=<IPv6-Address> and using a custom cloud provider binary). For some reason, using --node-ip=:: did not help.

The cloud provider also only provides the IPv4 address for the node, so this is an expected failure.

johngmyers · 2021-06-06T05:28:01Z

In retrospect, there shouldn't be IPv4Support and IPv6Support fields in the CalicoNetworkingSpec. The templates should key off of something that derives from whether the cluster has an IPv4 or IPv6 (or both) ClusterCIDR.

hakman · 2021-06-06T05:37:13Z

I agree, but this may allow some use cases like some fake pod dual-stack that works at node level.
Also, those are the default IP pools and what kOps does is just give a simple way to set the defaults correctly. Advanced users will do a lot more magic there.
So, the fields should be ok to exist, as overrides for advanced users, but the settings should default to the IP family of what's set in ClusterCIDR.

johngmyers · 2021-06-06T05:49:05Z

The Calico templates don't support dual-stack. For example, CALICO_ROUTER_ID should not be set to "hash" if it's dual-stack.

Advanced configurations will undoubtedly require code to configure all the things needed for the particular use case.

The fields don't make sense to expose. They just clutter up the API.

johngmyers · 2021-06-06T05:52:00Z

I think we might need code in pkg/model/kubelet.go to set --node-ip correctly in the IPv6 case. There's that b.UsesSecondaryIP() case there that currently assumes IPv4.

hakman · 2021-06-06T15:13:35Z

The Calico templates don't support dual-stack. For example, CALICO_ROUTER_ID should not be set to "hash" if it's dual-stack.

This is more of a quote from the docs which are lacking a lot of details when it comes to IPv6. "hash" means a CRC32 of the hostname, which contains the IP. It has nothing to do with how dual-stack is configured in Calico.

Calico allows lots of config using calicoctl, outside of the daemonset template, including creating new IP pools. kOps only configures the default IP pool, which is based on ClusterCIDR.

I think we might need code in pkg/model/kubelet.go to set --node-ip correctly in the IPv6 case. There's that b.UsesSecondaryIP() case there that currently assumes IPv4.

I am aware of that option and will take it into consideration, weather to support it or consider it mutually exclusive with IPv6.

k8s-ci-robot requested review from aojea, johngmyers and justinsb June 4, 2021 14:01

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. area/addons area/api labels Jun 4, 2021

Add options for configuring IPv4 and IPv6 support with Calico

70f77a3

hakman force-pushed the ipv6-calico branch from 88d0d26 to 70f77a3 Compare June 4, 2021 14:05

johngmyers requested changes Jun 5, 2021

View reviewed changes

k8s-ci-robot assigned johngmyers Jun 5, 2021

johngmyers self-requested a review June 5, 2021 22:24

johngmyers approved these changes Jun 5, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 5, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 5, 2021

k8s-ci-robot merged commit 7001de3 into kubernetes:master Jun 5, 2021

k8s-ci-robot added this to the v1.22 milestone Jun 5, 2021

hakman deleted the ipv6-calico branch June 6, 2021 02:53

hakman mentioned this pull request Jun 8, 2021

panic: runtime error: invalid memory address or nil pointer dereference #11715

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add options for configuring IPv4 and IPv6 support with Calico #11688

Add options for configuring IPv4 and IPv6 support with Calico #11688

hakman commented Jun 4, 2021 •

edited

Loading

aojea commented Jun 4, 2021

hakman commented Jun 4, 2021

hakman commented Jun 4, 2021

johngmyers left a comment

johngmyers Jun 5, 2021

hakman Jun 6, 2021

johngmyers Jun 5, 2021

johngmyers Jun 5, 2021

hakman Jun 6, 2021 •

edited

Loading

johngmyers Jun 5, 2021

johngmyers Jun 5, 2021 •

edited

Loading

hakman Jun 6, 2021

johngmyers commented Jun 5, 2021

johngmyers commented Jun 5, 2021

johngmyers left a comment

k8s-ci-robot commented Jun 5, 2021

johngmyers commented Jun 5, 2021

johngmyers commented Jun 5, 2021

johngmyers commented Jun 5, 2021

hakman commented Jun 6, 2021

hakman commented Jun 6, 2021

johngmyers commented Jun 6, 2021

hakman commented Jun 6, 2021

johngmyers commented Jun 6, 2021

johngmyers commented Jun 6, 2021

hakman commented Jun 6, 2021

Add options for configuring IPv4 and IPv6 support with Calico #11688

Add options for configuring IPv4 and IPv6 support with Calico #11688

Conversation

hakman commented Jun 4, 2021 • edited Loading

aojea commented Jun 4, 2021

hakman commented Jun 4, 2021

hakman commented Jun 4, 2021

johngmyers left a comment

Choose a reason for hiding this comment

johngmyers Jun 5, 2021

Choose a reason for hiding this comment

hakman Jun 6, 2021

Choose a reason for hiding this comment

johngmyers Jun 5, 2021

Choose a reason for hiding this comment

johngmyers Jun 5, 2021

Choose a reason for hiding this comment

hakman Jun 6, 2021 • edited Loading

Choose a reason for hiding this comment

johngmyers Jun 5, 2021

Choose a reason for hiding this comment

johngmyers Jun 5, 2021 • edited Loading

Choose a reason for hiding this comment

hakman Jun 6, 2021

Choose a reason for hiding this comment

johngmyers commented Jun 5, 2021

johngmyers commented Jun 5, 2021

johngmyers left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 5, 2021

johngmyers commented Jun 5, 2021

johngmyers commented Jun 5, 2021

johngmyers commented Jun 5, 2021

hakman commented Jun 6, 2021

hakman commented Jun 6, 2021

johngmyers commented Jun 6, 2021

hakman commented Jun 6, 2021

johngmyers commented Jun 6, 2021

johngmyers commented Jun 6, 2021

hakman commented Jun 6, 2021

hakman commented Jun 4, 2021 •

edited

Loading

hakman Jun 6, 2021 •

edited

Loading

johngmyers Jun 5, 2021 •

edited

Loading