Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1798788: Set Kubelet node IP to non-vip #1444

Merged
merged 2 commits into from Feb 12, 2020
Merged

Bug 1798788: Set Kubelet node IP to non-vip #1444

merged 2 commits into from Feb 12, 2020

Conversation

celebdor
Copy link
Contributor

@celebdor celebdor commented Feb 6, 2020

This change prevents:

  • Kubelet choosing as node IP an address from a non control plane subnet
    (like the provisioning network)
  • Kubelet choosing a deprecated IPv6 address as its node IP that in
    several platform can be a VIP

- How to verify it

Deploy on an IPv6 environment on a platform that uses keepalived managed Virtual IPs for its API, DNS and Ingress and see that the node IP is none of those, as those are created by keepalived as deprecated addresses.

@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 6, 2020
@openshift-ci-robot
Copy link
Contributor

@celebdor: This pull request references Bugzilla bug 1797655, which is invalid:

  • expected the bug to target the "4.4.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1797655: Set Kubelet node IP to non-vip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 6, 2020
@celebdor celebdor changed the title Bug 1797655: Set Kubelet node IP to non-vip Bug 1798788 : Set Kubelet node IP to non-vip Feb 6, 2020
@openshift-ci-robot
Copy link
Contributor

@celebdor: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Bug 1798788 : Set Kubelet node IP to non-vip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 6, 2020
@celebdor
Copy link
Contributor Author

celebdor commented Feb 6, 2020

/bugzilla refresh

@openshift-ci-robot
Copy link
Contributor

@celebdor: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@celebdor celebdor changed the title Bug 1798788 : Set Kubelet node IP to non-vip Bug 1798788: Set Kubelet node IP to non-vip Feb 6, 2020
@openshift-ci-robot
Copy link
Contributor

@celebdor: This pull request references Bugzilla bug 1798788, which is invalid:

  • expected the bug to target the "4.4.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1798788: Set Kubelet node IP to non-vip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 6, 2020
@celebdor
Copy link
Contributor Author

celebdor commented Feb 6, 2020

I propose the new non_virtual_ip for all of the platforms.
/cc @jcpowermac @mandre @rgolangh @hardys @karmab

@celebdor
Copy link
Contributor Author

celebdor commented Feb 6, 2020

Only ran the scripts in the PR in isolation still, I have not deployed with a new MCO yet (it's 2:36am already).

@rgolangh
Copy link
Contributor

rgolangh commented Feb 6, 2020

/test e2e-ovirt

@openshift-ci-robot openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 6, 2020
@mandre
Copy link
Member

mandre commented Feb 6, 2020

/test e2e-openstack

@celebdor
Copy link
Contributor Author

celebdor commented Feb 6, 2020

/test e2e-ovirt
/test e2e-openstack

@russellb russellb mentioned this pull request Feb 6, 2020
Merged
@russellb
Copy link
Member

russellb commented Feb 6, 2020

This didn't work for me in my first run of this with a backport to 4.3. The new kubelet-nodeip-configuration service never ran.

[core@master-1 ~]$ systemctl status kubelet-nodeip-configuration
● kubelet-nodeip-configuration.service - Writes an Environment file for Kubelet to choose a valide node IP address
   Loaded: loaded (/etc/systemd/system/kubelet-nodeip-configuration.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Feb 12, 2020
@openshift-ci-robot
Copy link
Contributor

@russellb: This pull request references Bugzilla bug 1798788, which is valid.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@celebdor
baremetal: Set Kubelet node IP to non-vip
d93d1e9 
This change prevents:
* Kubelet choosing as node IP an address from a non control plane subnet
  (like the provisioning network)
* Kubelet choosing a deprecated IPv6 address as its node IP that in
  several platform can be a VIP

Signed-off-by: Antoni Segura Puimedon <antoni@redhat.com>
@celebdor
Prevent CRI-O picking the wrong node IP to stream
065082e 
CRI-O on startup could get addresses with expiring leases or from
interfaces that are in isolated networks. This patch extends
kubelet-nodeip.service to fix CRI-O as well.

Signed-off-by: Antoni Segura Puimedon <antoni@redhat.com>
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Feb 12, 2020
@russellb
Copy link
Member

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 12, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: celebdor, kikisdeliveryservice, russellb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 12, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 8542734 into openshift:master Feb 12, 2020
@openshift-ci-robot
Copy link
Contributor

@celebdor: All pull requests linked via external trackers have merged. Bugzilla bug 1798788 has been moved to the MODIFIED state.

In response to this:

Bug 1798788: Set Kubelet node IP to non-vip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

@celebdor: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-ovirt a8dc8fa25a364fb26086e25c4631a8a2f50db745 link /test e2e-ovirt
ci/prow/e2e-openstack a8dc8fa25a364fb26086e25c4631a8a2f50db745 link /test e2e-openstack
ci/prow/e2e-aws-scaleup-rhel7 065082e link /test e2e-aws-scaleup-rhel7

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@russellb
Copy link
Member

/cherry-pick release-4.3

@openshift-cherrypick-robot
Copy link

@russellb: #1444 failed to apply on top of branch "release-4.3":

.git/rebase-apply/patch:260: new blank line at EOF.
+
warning: 1 line adds whitespace errors.
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	templates/worker/00-worker/baremetal/files/baremetal-non-virtual-ip.yaml
Falling back to patching base and 3-way merge...
Auto-merging templates/common/baremetal/files/baremetal-non-virtual-ip.yaml
CONFLICT (content): Merge conflict in templates/common/baremetal/files/baremetal-non-virtual-ip.yaml
Patch failed at 0001 baremetal: Set Kubelet node IP to non-vip

In response to this:

/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@russellb
Copy link
Member

It looks like this one depends on #1445. We could either place #1445 with a new PR that includes both changes, or wait for #1445 to go in first

@celebdor

@russellb russellb mentioned this pull request Feb 12, 2020
Merged
@russellb
Copy link
Member

opened #1466

@mandre mandre mentioned this pull request Feb 18, 2020
Merged
mandre added a commit to mandre/machine-config-operator that referenced this pull request Feb 18, 2020
@mandre
OpenStack: Set Kubelet node IP to non-vip
f957927 
This change prevents:
* Kubelet choosing as node IP an address from a non control plane subnet
  (like the provisioning network)
* Kubelet choosing a deprecated IPv6 address as its node IP that in
  several platform can be a VIP

This ports the BM change from
openshift#1444 to
OpenStack platform.
russellb added a commit to openshift-kni/cluster-kube-apiserver-operator that referenced this pull request Feb 18, 2020
@russellb
baremetal: Explicitly advertise the host IP chosen by kubelet
ca8bec2 
We've updated MCO to ensure IPv6 VIPs are not accidentally used by
kubelet.  Make use of that value and set it as the address
kube-apiserver should advertise.  Otherwise, it uses the same broken
default behavior that kubelet does and will incorrectly choose IPv6
VIPs, breaking the cluster.

The change to specify the correct IP to kubelet is:
openshift/machine-config-operator#1444
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/cluster-kube-apiserver-operator that referenced this pull request Feb 20, 2020
@russellb
baremetal: Explicitly advertise the host IP chosen by kubelet
79a61ad 
We've updated MCO to ensure IPv6 VIPs are not accidentally used by
kubelet.  Make use of that value and set it as the address
kube-apiserver should advertise.  Otherwise, it uses the same broken
default behavior that kubelet does and will incorrectly choose IPv6
VIPs, breaking the cluster.

The change to specify the correct IP to kubelet is:
openshift/machine-config-operator#1444
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/cluster-kube-apiserver-operator that referenced this pull request Feb 21, 2020
@russellb
baremetal: Explicitly advertise the host IP chosen by kubelet
19d9088 
We've updated MCO to ensure IPv6 VIPs are not accidentally used by
kubelet.  Make use of that value and set it as the address
kube-apiserver should advertise.  Otherwise, it uses the same broken
default behavior that kubelet does and will incorrectly choose IPv6
VIPs, breaking the cluster.

The change to specify the correct IP to kubelet is:
openshift/machine-config-operator#1444
vrutkovs pushed a commit to vrutkovs/machine-config-operator that referenced this pull request Mar 1, 2020
@mandre @vrutkovs
OpenStack: Set Kubelet node IP to non-vip
aeb2de1 
This change prevents:
* Kubelet choosing as node IP an address from a non control plane subnet
  (like the provisioning network)
* Kubelet choosing a deprecated IPv6 address as its node IP that in
  several platform can be a VIP

This ports the BM change from
openshift#1444 to
OpenStack platform.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jcpowermac jcpowermac jcpowermac left review comments

@kikisdeliveryservice kikisdeliveryservice kikisdeliveryservice approved these changes

@cgwalters cgwalters Awaiting requested review from cgwalters

@russellb russellb Awaiting requested review from russellb

@hardys hardys Awaiting requested review from hardys

@karmab karmab Awaiting requested review from karmab

@mandre mandre Awaiting requested review from mandre

@rgolangh rgolangh Awaiting requested review from rgolangh

Assignees

@russellb russellb

@kikisdeliveryservice kikisdeliveryservice

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet