Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IPv4/IPv6 dual-stack support #563

Open
leblancd opened this issue Apr 18, 2018 · 142 comments
Open

Add IPv4/IPv6 dual-stack support #563

leblancd opened this issue Apr 18, 2018 · 142 comments
Assignees
Projects
Milestone

Comments

@leblancd
Copy link

@leblancd leblancd commented Apr 18, 2018

Feature Description

  • One-line feature description (can be used as a release note):
    IPv4/IPv6 dual-stack support and awareness for Kubernetes pods, nodes, and services
  • Primary contact (assignee): @leblancd
  • Responsible SIGs: sig-network
  • Design proposal link (community repo): Add IPv4/IPv6 dual stack KEP (old)
  • KEP PR: #808
  • KEP: 20180612-ipv4-ipv6-dual-stack
  • Link to e2e and/or unit tests: TBD
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @thockin @dcbw @luxas
  • Approver (likely from SIG/area to which feature belongs): @thockin
  • Feature target (which target equals to which milestone):
    • Alpha release target 1.11
    • Beta release target 1.20
    • Stable release target (x.y)

Corresponding kubernetes/kubernetes Issue: kubernetes/kubernetes#62822

@leblancd
Copy link
Author

@leblancd leblancd commented Apr 19, 2018

Cross Reference with kubernetes/kubernetes: Issue #62822

@justaugustus
Copy link
Member

@justaugustus justaugustus commented Apr 20, 2018

Thanks for the update!

/assign @leblancd
/kind feature
/sig network
/milestone 1.11

@idvoretskyi
Copy link
Member

@idvoretskyi idvoretskyi commented May 2, 2018

@leblancd
Copy link
Author

@leblancd leblancd commented May 8, 2018

@idvoretskyi - No design doc yet, but we'll start collaborating on one shortly.

@sb1975
Copy link

@sb1975 sb1975 commented May 10, 2018

Does this mean Kubernetes Ingress will support Dual-Stack ?
Does this mean CNI ( Calico) would need to run Dual stack ( both BIRD and BIRD6 daemons for example) ?

@leblancd
Copy link
Author

@leblancd leblancd commented May 11, 2018

@sb1975 - Regarding dual-stack ingress support, that's something we'll need to hash out, but here are my preliminary thoughts:

  • Dual stack ingress support will mostly depend upon which ingress controller you use (whether it's supported and how it's implemented). Existing ingress controllers will probably need some changes to support dual-stack.
  • I expect that the ingress configuration for a typical ingress controller won't change (the config might e.g. still map an L7 address to a service name / service port, with no mention of V4/V6 family)
  • In the case where a service has endpoint pods that are dual-stack, the ingress controller(s) might need changes to map ingress packets based on the packets' family, i.e. map IPv4 ingress packets to an IPv4 endpoint, and map IPv6 ingress packets to an IPv6 endpoint. For the purposes of load-balance weighting, a dual-stack endpoint should count as a single endpoint target.
  • We might want to consider FUTURE support for having an ingress controller map across V4/V6 families (map ingress IPv4 packets to an IPv6 backend, and vice versa), but our initial development will be for strict dual-stack (i.e. separate, independent stacks).
    ========================
    Regarding Calico and other CNI plugins:
  • The CNI plugins won't HAVE TO run in dual-stack mode if a cluster scenario doesn't require dual-stack, they should still be able to run IPv4-only or IPv6-only (if the plugin supports it).
  • Dual-stack support will probably require changes in the various CNI plugins, but that work is considered outside of the scope of this Kubernetes issue (we're focusing on making Kubernetes work for any arbitrary dual-stack plugin, probably using the bridge plugin as a reference), and the CNI work will be done separately on a case-by-case basis.
  • For Calico specifically, I'm not an expert but I believe that a single BIRD daemon can be configured to handle both IPv4 and IPv6 routes (search for "template bgp" here: http://bird.network.cz/?get_doc&v=20&f=bird-3.html#ss3.1). That said, although Calico already supports dual-stack addresses at the pod, there might be changes required to get the BGP routing working for both families.
@sb1975
Copy link

@sb1975 sb1975 commented May 15, 2018

@leblancd : So here is the scenario :

  1. Lets say we will use NGINX ingress controller
  2. I am exposing my services via Ingress.
  3. I am running my pods configured on dual-stack
  4. I am trying to reach the service remotely using A and AAAA dns-records.
    Hope all of these
  5. In summary : I want to connect to pod interfaces using either IPv4 or IPv6 addresses, as resolved by my own queries for A and/or AAAA records for the pod service name.
    Can I get involved in this initiative to test,documentation,architecture: but need some guidance.
    How do I get to know about the progress of this please.
@leblancd
Copy link
Author

@leblancd leblancd commented May 17, 2018

@sb1975 - Good question re. the NGINX ingress controller with dual-stack. I'm not an expert on the NGINX ingress controller (maybe someone more familiar can jump in), but here's how I would see the work flow:

  • When you try to reach the Kube services from outside, your DNS controller should resolve the service with A and AAAA DNS records for the ingress controller. It would be your client/app's choice to use the A vs. the AAAA record to reach the ingress controller. So external access to the ingress controller would be dual-stack.
  • At the NGINX ingress controller, NGINX would then look at the L7 URL (regardless of request being in an IPv4 or IPv6 packet) and load balance that to upstream endpoints. If the ingress controller load balancer is configured with ipv6=on (which is default, see https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#configuring-http-load-balancing-using-dns), and the service endpoint(s) are dual-stack, then the upstream configuration should have both IPv4 and IPv6 entries for each dual-stack endpoint. As designed, the NGINX load balancer treats the IPv4 entry and the IPv6 entry for an endpoint as separate servers. (See the line in the fore-mentioned doc "If a domain name resolves to several IP addresses, the addresses are saved to the upstream configuration and load balanced.") This can be considered good news or bad news. The good news is that load balancing will be done across IPv4 and IPv6 endpoints (giving you some redundancy), where e.g. an incoming IPv4 request could get mapped to either an IPv4 or an IPv6 endpoint. But the potential bad news is with load balancing granularity: a connection to an IPv4 endpoint and a connection to the corresponding IPv6 endpoint will be treated (for load balancing considerations) as 2 loads to separate endpoints, rather than 2 separate loads to the same endpoint. If this load-balancing granularity is a concern, then someone could disable load balancing to IPv6 (or to IPv4, if there's a config knob for that?), so that load balancing would be to IPv4-only endpoints. OR, maybe NGINX load balancer can be modified to treat a connection to an IPv4 address and a connection to its corresponding IPv6 address as 2 loads to the same endpoint.

As for helping and getting involved, this would be greatly appreciated! We're about to start working in earnest on dual-stack (it's been a little delayed by the work in getting CI working for IPv6-only). I'm hoping to come out with an outline for a spec (Google Doc or KEPs WIP doc) soon, and would be looking for help in reviewing, and maybe writing some sections. We'll also DEFINITELY need help with official documentation (beyond the design spec), and with defining and implementing dual-stack E2E tests. Some of the areas which I'm still a bit sketchy on for the design include:

  • How are health/liveness/readiness probes affected or handled with dual-stack
  • Will there be an impact on network policies?
  • Load balancer concerns?
  • Cloud provider plugin concerns?
  • L3/L4 ingress concerns?
    If you've thought about any of these, maybe you could help with those sections?

We're also considering an intermediate "dual-stack at the edge" (with IPv6-only inside the cluster) approach, where access from outside the cluster to K8s services would be dual-stack, but this would be mapped (e.g. via NGINX ingress controller) to IPv6-only endpoints inside the cluster (or use stateless NAT46). Pods and services in the cluster would need to be all IPv6, but the big advantage would be that dual-stack external access would be available much more quickly from a time-to-market perspective.

@caseydavenport
Copy link
Member

@caseydavenport caseydavenport commented May 30, 2018

/milestone 1.12

@justaugustus
Copy link
Member

@justaugustus justaugustus commented May 31, 2018

@leblancd / @caseydavenport - I'm noticing a lot of discussion here and a milestone change.
Should this be pulled from the 1.11 milestone?

@leblancd
Copy link
Author

@leblancd leblancd commented May 31, 2018

@justaugustus - Yes, this should be moved to 1.12. Do I need to delete a row in the release spreadsheet, or is there anything I need to do to get this changed?

@justaugustus
Copy link
Member

@justaugustus justaugustus commented Jun 1, 2018

@leblancd I've got it covered. Thanks for following up! :)

@justaugustus
Copy link
Member

@justaugustus justaugustus commented Jul 18, 2018

@leblancd @kubernetes/sig-network-feature-requests --

This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

@justaugustus
Copy link
Member

@justaugustus justaugustus commented Jul 31, 2018

@leblancd --
Feature Freeze is today. Are you planning on graduating this to Beta in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

@leblancd
Copy link
Author

@leblancd leblancd commented Jul 31, 2018

Hi @justaugustus - Beta status will need to slip into Kubernetes 1.13. We are making (albeit slow) progress on the design KEP (kubernetes/community#2254), and we're getting close to re-engaging with the CI test PR, but the Kubernetes 1.12 target was a bit too optimistic.

I'll update the description/summary above with the information you requested earlier. Thank you for your patience.

@justaugustus justaugustus modified the milestones: v1.12, v1.13 Jul 31, 2018
@bridgetkromhout
Copy link
Contributor

@bridgetkromhout bridgetkromhout commented Jan 21, 2021

KEP: 20180612-ipv4-ipv6-dual-stack

KEP has been updated to the new format: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack

@bridgetkromhout
Copy link
Contributor

@bridgetkromhout bridgetkromhout commented Jan 22, 2021

We're planning to move dual-stack from alpha to beta for 1.21.

Updates to this enhancements issue:

  • Feature target (which target equals to which milestone):
    • Alpha release target 1.11 (revised: 1.20)
    • Beta release target 1.21
    • Stable release target (x.y)
@fmuyassarov
Copy link

@fmuyassarov fmuyassarov commented Jan 22, 2021

Hi @fmuyassarov - We are working with sig-network to determine the plan to move this feature to beta in 1.21. Would you like to see this enhancement go to beta?

Hi @lachie83 . Yes, would be nice to see dual-stack graduating to beta in 1.21.

@JamesLaverack
Copy link
Member

@JamesLaverack JamesLaverack commented Jan 29, 2021

👋 Hey @lachie83, 1.21 release team enhancements shadow here.

We're currently tracking this KEP for graduation to beta Kubernetes 1.21, and the only issue I can see is the production readiness review. I'm aware there's an ongoing pull request to add this though: #2327. We'll continue to track this in the meantime.

As one query: The kep.yaml states that SIG Cluster Lifecycle are participating. Do they need to do any work on this, and if so are they signed on for this release?

@bridgetkromhout
Copy link
Contributor

@bridgetkromhout bridgetkromhout commented Jan 29, 2021

Hi, @JamesLaverack - it looks like SIG Cluster Lifecycle's inclusion may be outdated. Aside from the sig-network involvement, the only other place we might be looking for confirmation of the testing is from sig-testing's kind subproject. I think @aojea can confirm whether we should change https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/563-dual-stack/kep.yaml#L11 to sig-testing or perhaps remove it.

@aojea
Copy link
Member

@aojea aojea commented Jan 29, 2021

  • sig-cluster-lifecycle

they've implemented some bits in kubeadm, but they should not do anything else, @neolit123 can you confirm?

Aside from the sig-network involvement, the only other place we might be looking for confirmation of the testing is from sig-testing's kind subproject

kind has dualstack as priority for next release kubernetes-sigs/kind#2024 and haa been running periodic job with an "unofficial" version for several months cc: @BenTheElder

Technically we are set, and both sigs are aware, I can't say if we need to do anything else or this is enough

@neolit123
Copy link
Member

@neolit123 neolit123 commented Feb 3, 2021

@aojea

they've implemented some bits in kubeadm, but they should not do anything else, @neolit123 can you confirm?

for kubeadm the dual stack support is tracked here:
kubernetes/kubeadm#1612

i see some remaining tasks in the punch card there and the kubeadm feature gate is still alpha

cc @Arvinderpal do you have time to work on these updates for 1.21, or perhaps you can delegate to someone else from SIG Net or SIG CL?

@Arvinderpal
Copy link

@Arvinderpal Arvinderpal commented Feb 3, 2021

@neolit123 I have not been following dual-stack developments recently and I'm not sure if any of the tasks in kubernetes/kubeadm#1612 are still relevant anymore.
It may be best if some like @aojea or others from the SIG Net take a look.

@aojea
Copy link
Member

@aojea aojea commented Feb 3, 2021

replied in the kubeadm issue, I think that only kubeadm docs are missing, rest is good

@neolit123
Copy link
Member

@neolit123 neolit123 commented Feb 3, 2021

thanks. ok, looks like:

Add documentation on enabling dual-stack via kubadm

is still a viable item.

if the core dual stack feature gate as moving to Beta in 1.21, we should move the kubeadm gate too.

@JamesLaverack
Copy link
Member

@JamesLaverack JamesLaverack commented Feb 7, 2021

Hey @lachie83, enhancements 1.21 shadow here again,

Enhancements Freeze is 2 days away, Feb 9th EOD PST

The enhancements team is aware that KEP update is currently in progress (PR #2327). Please make sure to work on PRR questionnaires and requirements and get it merged before the freeze. For PRR related questions or to boost the PR for PRR review, please reach out in Slack on the #prod-readiness channel.

Any enhancements that do not complete the following requirements by the freeze will require an exception.

  • [DONE] The KEP must be merged in an implementable state
  • [DONE] The KEP must have test plans
  • [DONE] The KEP must have graduation criteria
  • [IN PROGRESS] The KEP must have a production readiness review

Thanks all for the clarification on your participating SIGs too. :)

@bridgetkromhout
Copy link
Contributor

@bridgetkromhout bridgetkromhout commented Feb 9, 2021

Hi, @JamesLaverack - as we discussed on Slack we have #2327 (our PRR) merged. I think according to your checklist we should now be listed as Tracked for 1.21, though right now I still see us listed as At Risk. Thanks. Edit: I see we're now tracked! Thanks.

@JamesLaverack
Copy link
Member

@JamesLaverack JamesLaverack commented Feb 9, 2021

HI @bridgetkromhout, Thanks for the notification. I've taken a quick look and with that merged you're correct that you've covered everything. I've updated the 1.21 tracking sheet to be "Tracked" for this enhancement.

@JamesLaverack
Copy link
Member

@JamesLaverack JamesLaverack commented Feb 19, 2021

Hi @lachie83,

Since your Enhancement is scheduled to be in 1.21, please keep in mind the important upcoming dates:

  • Tuesday, March 9th: Week 9 — Code Freeze
  • Tuesday, March 16th: Week 10 — Docs Placeholder PR deadline
    • If this enhancement requires new docs or modification to existing docs, please follow the steps in the Open a placeholder PR doc to open a PR against k/website repo.

As a reminder, please link all of your k/k PR(s) and k/website PR(s) to this issue so we can track them.

Thanks!

@pacoxu
Copy link
Member

@pacoxu pacoxu commented Feb 22, 2021

/assign
for kubernetes/kubeadm#1612 (comment), I will work on the kubeadm part.

@reylejano
Copy link
Member

@reylejano reylejano commented Feb 26, 2021

Doc PR for 1.21 is k/website PR 26675

@JamesLaverack
Copy link
Member

@JamesLaverack JamesLaverack commented Mar 3, 2021

Hi @lachie83

Enhancements team is marking this enhancement as "At Risk" for the upcoming code freeze due to not seeing any linked k/k PR(s) for this enhancement. (Unless I've missed them! Please tell me if I have.)

Please make sure to provide all k/k PR(s) and k/website PR(s) to this issue so it can be tracked by the release team.

P.S. Should I be tagging others for updates about this? @bridgetkromhout? @aojea maybe? I only have Lachie down in our spreadsheet as a contact but I'm happy to ping others if requested too.

@bridgetkromhout
Copy link
Contributor

@bridgetkromhout bridgetkromhout commented Mar 3, 2021

Hi @JamesLaverack this enhancement is intended to graduate from alpha to beta; it's not a new code addition in k/k. We filed the PRR and I'll open the placeholder docs PR. What other updates were you looking for? Thanks.

@bridgetkromhout
Copy link
Contributor

@bridgetkromhout bridgetkromhout commented Mar 3, 2021

The k/k PR for this change (alpha to beta, putting feature gate on by default) is in kubernetes/kubernetes#98969.

@JamesLaverack
Copy link
Member

@JamesLaverack JamesLaverack commented Mar 3, 2021

Hi @bridgetkromhout.

Thank you for the clarification. I wasn't sure what k/k code changes were required and I missed the link for kubernetes/kubernetes#98969. But as that's merged and there are no other changes I've flipped this enhancement back to "Tracked" (from "At Risk") and marked it as done for code freeze.

@bridgetkromhout
Copy link
Contributor

@bridgetkromhout bridgetkromhout commented Mar 4, 2021

Thanks, @JamesLaverack. I also just submitted a 1.21 docs update in kubernetes/website#26826 and didn't put it in draft, as it's ready for review at this time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
SIG-NETWORK
  
In progress
Linked pull requests

Successfully merging a pull request may close this issue.