Detect CIDR IPv4 or IPv6 version to select nexthop #59749

zioproto · 2018-02-12T08:58:49Z

What this PR does / why we need it:

The node InternalIP is used as nexthop by the Kubernetes master to create routes in the Neutron router for Pods reachability.
If a node has more than one InternalIPs, eventually IPv4 and IPv6, a random InternalIP from the list is returned.
This can lead to the bug described in #59421
We need to check when we build a route that the CIDR and the nexthop belong to the same IP Address Family (both IPv4 or both IPv6)

Which issue(s) this PR fixes :
Fixes #59421
It is related to #55202

Special notes for your reviewer:
This is the suggested way to fix the problem after the discussion in #59502

old release note: Issue #59421 was closed: the Openstack deployment with dual stack (IPv4 and IPv6) nodes and network-plugin=kubenet is now working properly.

Release note:

Fixing a bug in OpenStack cloud provider, where dual stack deployments (IPv4 and IPv6) did not work well when using kubenet as the network plugin.

FengyunPan2

Sounds good, can we keep getAddressByName()?
Or use a small method since we are using the same thing 2 times, maybe use it at other function later.

zioproto · 2018-02-12T14:50:54Z

@FengyunPan2 is it okay then if I pass route.DestinationCIDR as an argument to getAddressByName in order to compute in the function if I should get an IPv4 or IPv6 result ?

zioproto · 2018-02-12T16:28:15Z

@FengyunPan2 I pushed a new amended commit. Now the implementation uses the function and there is not code duplication.

leblancd · 2018-02-12T17:25:54Z

pkg/cloudprovider/providers/openstack/openstack.go

@@ -520,12 +521,12 @@ func getAddressByName(client *gophercloud.ServiceClient, name types.NodeName) (s
 	}

 	for _, addr := range addrs {
-		if addr.Type == v1.NodeInternalIP {
+		if (addr.Type == v1.NodeInternalIP) && (govalidator.IsIPv6(addr.Address) == needIPv6) {


Might be better to use the more generic 'net' package (instead of asaskevich/govalidator) to determine whether the address is IPv6, e.g.:

isIPv6 := net.ParseIP(addr.Address).To4() == nil if addr.Type == v1.NodeInternalIP && isIPv6 == needIPv6 { return addr.Address, nil }

hello @leblancd
I checked https://stackoverflow.com/questions/22751035/golang-distinguish-ipv4-ipv6

Please check in particular this answer: https://stackoverflow.com/a/48519490/1506396
It is the exaplanation why to use govalidator instead of the net package.

Hi @zioproto:
I would respectfully disagree with what the poster is claiming in this stackoverflow answer. Specifically, the list of addresses that are labeled "Valid IPv6 notations" should be labeled instead "IPv4-mapped IPv6 Notations" (RFC4291 and RFC4038). These addresses have an IPv6 format, but are intended to be internal representations of IPv4 addresses on the wire. They're essentially IPv4 addresses encapsulated in an IPv6 frame, and this is done so that dual-stack software can conveniently handle both IPv4 and IPv6 addresses as 16-byte entities. The net package's To4() takes care to look for this pattern in a 16-byte slice, and (correctly, I believe) creates an 4-byte (IPv4) slice from the lowest 4 bytes (since Golang can handle slices of arbitrary length, so there's no need to perpetuate the 16-byte internal representation of IPv4-mapped IPv6 addresses). In the examples given in the post, the only addresses that the net's To4() gets wrong, in my opinion, are the examples with :port appended, but To4() assumes that the caller has taken care of eliminating any :port (e.g. with splitHostPort()).

leblancd · 2018-02-12T17:30:53Z

pkg/cloudprovider/providers/openstack/openstack_routes.go

@@ -146,7 +148,10 @@ func (r *Routes) CreateRoute(ctx context.Context, clusterName string, nameHint s

 	onFailure := newCaller()

-	addr, err := getAddressByName(r.compute, route.TargetNode)
+	IP, _, _ := net.ParseCIDR(route.DestinationCIDR)
+	CIDRisV6 := govalidator.IsIPv6(IP.String())


Same comment as before, can use 'net' package to test for IPv6, e.g.:

CIDRisV6 := IP.To4() == nil

dims · 2018-02-12T21:39:10Z

/ok-to-test

leblancd · 2018-02-12T21:42:06Z

Looks good to me. Thanks, @zioproto for taking care of this!

leblancd · 2018-02-12T22:46:52Z

/test pull-kubernetes-bazel-test

FengyunPan2 · 2018-02-13T01:14:27Z

@zioproto Yeah, nice work.
/lgtm

k8s-ci-robot · 2018-02-13T01:14:34Z

@FengyunPan2: changing LGTM is restricted to assignees, and only kubernetes org members may be assigned issues.

In response to this:

@zioproto Yeah, nice work.
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dims · 2018-02-13T02:34:25Z

/approve
/lgtm

anguslees · 2018-02-13T02:34:19Z

pkg/cloudprovider/providers/openstack/openstack.go

 			return addr.Address, nil
 		}
 	}

-	return addrs[0].Address, nil
+	return "", ErrNoAddressFound


With this last change, this function will only return InternalIPs. It should instead try ExternalIPs before giving up.

Aside: An alternate version might be to pass an address family down to getAddressesByName and thence nodeAddresses (with some value to indicate "all" families) and then do the filtering at the lowest level, when we know the address family from the openstack data structure. Happy to go with whatever version you think is best.

anguslees · 2018-02-13T02:35:53Z

pkg/cloudprovider/providers/openstack/openstack_routes.go

@@ -146,7 +147,10 @@ func (r *Routes) CreateRoute(ctx context.Context, clusterName string, nameHint s

 	onFailure := newCaller()

-	addr, err := getAddressByName(r.compute, route.TargetNode)
+	IP, _, _ := net.ParseCIDR(route.DestinationCIDR)
+	CIDRisV6 := IP.To4() == nil


Both IP and CIDRisV6 should have (at least) a lowercased first letter, to conform with golang's public/private styleguide/rules.

anguslees · 2018-02-13T02:36:58Z

pkg/cloudprovider/providers/openstack/openstack_routes.go

@@ -219,7 +223,10 @@ func (r *Routes) DeleteRoute(ctx context.Context, clusterName string, route *clo

 	onFailure := newCaller()

-	addr, err := getAddressByName(r.compute, route.TargetNode)
+	IP, _, _ := net.ParseCIDR(route.DestinationCIDR)
+	CIDRisV6 := IP.To4() == nil


Ditto lowercased first letter of local variable.

dims · 2018-02-13T02:44:05Z

/hold

dims · 2018-02-13T02:48:52Z

/lgtm cancel

zioproto · 2018-02-13T09:57:10Z

/test pull-kubernetes-unit

dims · 2018-02-13T18:46:20Z

/hold cancel

@anguslees please take another look

dims · 2018-02-13T20:15:13Z

/lgtm

dims · 2018-02-13T20:15:39Z

/lgtm cancel

dims · 2018-02-13T20:16:15Z

weird, i wanted to indicate lgtm without approving ... bot applied both! never ming. just waiting for @anguslees :)

anguslees

/lgtm

k8s-ci-robot · 2018-02-14T05:26:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anguslees, FengyunPan2, zioproto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these OWNERS Files:

~~pkg/cloudprovider/providers/openstack/OWNERS~~ [anguslees]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-02-14T06:06:58Z

/test all [submit-queue is verifying that this PR is safe to merge]

fejta-bot · 2018-02-14T09:09:10Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

zioproto · 2018-02-14T09:09:35Z

/test pull-kubernetes-e2e-kops-aws

k8s-github-robot · 2018-02-14T10:08:09Z

/test all [submit-queue is verifying that this PR is safe to merge]

zioproto · 2018-02-14T10:56:48Z

/test pull-kubernetes-unit

k8s-github-robot · 2018-02-14T11:33:37Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-02-14T12:10:10Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

zioproto · 2018-02-14T13:49:00Z

/cherrypick-candidate

…pstream-release-1.9 Automatic merge from submit-queue. Automated cherry pick of #59749: Detect CIDR IPv4 or IPv6 version to select nexthop Cherry pick of #59749 on release-1.9. #59749: Detect CIDR IPv4 or IPv6 version to select nexthop ```release-note Fixing a bug in OpenStack cloud provider, where dual stack deployments (IPv4 and IPv6) did not work well when using kubenet as the network plugin. ```

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 12, 2018

k8s-ci-robot requested review from anguslees and dims February 12, 2018 08:58

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 12, 2018

zioproto mentioned this pull request Feb 12, 2018

Prevent IPv6 addresses to be v1.NodeInternalIP #59502

Closed

zioproto pushed a commit to zioproto/k8s-on-openstack that referenced this pull request Feb 12, 2018

Test kubernetes/kubernetes#59749

90ec8e9

FengyunPan2 reviewed Feb 12, 2018

View reviewed changes

zioproto force-pushed the issues/59421-CheckCIDR branch from 877f937 to a4c4dd2 Compare February 12, 2018 16:15

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 12, 2018

leblancd reviewed Feb 12, 2018

View reviewed changes

zioproto force-pushed the issues/59421-CheckCIDR branch from a4c4dd2 to ad7089a Compare February 12, 2018 21:33

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 12, 2018

k8s-ci-robot assigned dims Feb 13, 2018

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 13, 2018

anguslees suggested changes Feb 13, 2018

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 13, 2018

zioproto force-pushed the issues/59421-CheckCIDR branch from 4c127c4 to 2eff8bf Compare February 13, 2018 08:51

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 13, 2018

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 13, 2018

k8s-ci-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 13, 2018

k8s-ci-robot assigned anguslees Feb 14, 2018

anguslees approved these changes Feb 14, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 14, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 14, 2018

k8s-github-robot merged commit 0dda5c8 into kubernetes:master Feb 14, 2018

dims mentioned this pull request Feb 14, 2018

Automated cherry pick of #59749: Detect CIDR IPv4 or IPv6 version to select nexthop #59868

Merged

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Feb 14, 2018

wojtek-t mentioned this pull request Apr 5, 2018

'PATCH node-status' latency slo violations #62064

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect CIDR IPv4 or IPv6 version to select nexthop #59749

Detect CIDR IPv4 or IPv6 version to select nexthop #59749

zioproto commented Feb 12, 2018 •

edited by mbohlool

FengyunPan2 left a comment

zioproto commented Feb 12, 2018 •

edited

zioproto commented Feb 12, 2018

leblancd Feb 12, 2018

zioproto Feb 12, 2018

leblancd Feb 12, 2018

leblancd Feb 12, 2018

dims commented Feb 12, 2018

leblancd commented Feb 12, 2018 •

edited

leblancd commented Feb 12, 2018

FengyunPan2 commented Feb 13, 2018

k8s-ci-robot commented Feb 13, 2018

dims commented Feb 13, 2018

anguslees Feb 13, 2018

anguslees Feb 13, 2018

anguslees Feb 13, 2018

dims commented Feb 13, 2018

dims commented Feb 13, 2018

zioproto commented Feb 13, 2018 •

edited

dims commented Feb 13, 2018

dims commented Feb 13, 2018

dims commented Feb 13, 2018

dims commented Feb 13, 2018

anguslees left a comment

k8s-ci-robot commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

fejta-bot commented Feb 14, 2018

zioproto commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

zioproto commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

zioproto commented Feb 14, 2018

Detect CIDR IPv4 or IPv6 version to select nexthop #59749

Detect CIDR IPv4 or IPv6 version to select nexthop #59749

Conversation

zioproto commented Feb 12, 2018 • edited by mbohlool

FengyunPan2 left a comment

Choose a reason for hiding this comment

zioproto commented Feb 12, 2018 • edited

zioproto commented Feb 12, 2018

leblancd Feb 12, 2018

Choose a reason for hiding this comment

zioproto Feb 12, 2018

Choose a reason for hiding this comment

leblancd Feb 12, 2018

Choose a reason for hiding this comment

leblancd Feb 12, 2018

Choose a reason for hiding this comment

dims commented Feb 12, 2018

leblancd commented Feb 12, 2018 • edited

leblancd commented Feb 12, 2018

FengyunPan2 commented Feb 13, 2018

k8s-ci-robot commented Feb 13, 2018

dims commented Feb 13, 2018

anguslees Feb 13, 2018

Choose a reason for hiding this comment

anguslees Feb 13, 2018

Choose a reason for hiding this comment

anguslees Feb 13, 2018

Choose a reason for hiding this comment

dims commented Feb 13, 2018

dims commented Feb 13, 2018

zioproto commented Feb 13, 2018 • edited

dims commented Feb 13, 2018

dims commented Feb 13, 2018

dims commented Feb 13, 2018

dims commented Feb 13, 2018

anguslees left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

fejta-bot commented Feb 14, 2018

zioproto commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

zioproto commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

k8s-github-robot commented Feb 14, 2018

zioproto commented Feb 14, 2018

zioproto commented Feb 12, 2018 •

edited by mbohlool

zioproto commented Feb 12, 2018 •

edited

leblancd commented Feb 12, 2018 •

edited

zioproto commented Feb 13, 2018 •

edited