Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for configuring Cilium enable-host-reachable-services. #11333

Merged
merged 1 commit into from
Apr 27, 2021
Merged

Add support for configuring Cilium enable-host-reachable-services. #11333

merged 1 commit into from
Apr 27, 2021

Conversation

bjhaid
Copy link
Contributor

@bjhaid bjhaid commented Apr 27, 2021

After upgrading Cilium to 1.8 via kops one of our clusters had a total
outage due to cilium reporting errors as below:

level=error msg="endpoint regeneration failed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=592 error="Failed to load tc filter: exit status 1" identity=40147 ipv4= ipv6= k8sPodName=/ subsys=endpoint

upon searching Cilium slack we found the below thread:

https://cilium.slack.com/archives/C1MATJ5U5/p1616400216167600

which recommended setting enable-host-reachable-services to true will
address the problems. We set the field and it fixed our issues too,
however we observed that kops does not have a means to configure this
hence this PR.

We will like to have this backported after it has been merged.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 27, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @bjhaid. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 27, 2021
@joshbranham
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 27, 2021
@olemarkus
Copy link
Member

Thanks for the PR. This change will add the needed configuration flag, but it will not make any actual changes to the cilium manifests.

You can find the template you need to modify here: upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.12-v1.9.yaml.template

@bjhaid
Add support for configuring Cilium enable-host-reachable-services.
88ed9f6 
After upgrading Cilium to 1.8 via kops one of our clusters had a total
outage due to cilium reporting errors as below:

```
level=error msg="endpoint regeneration failed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=592 error="Failed to load tc filter: exit status 1" identity=40147 ipv4= ipv6= k8sPodName=/ subsys=endpoint
```

upon searching Cilium slack we found the below thread:

https://cilium.slack.com/archives/C1MATJ5U5/p1616400216167600

which recommended setting `enable-host-reachable-services` to true will
address the problems. We set the field and it fixed our issues too,
however we observed that kops does not have a means to configure this
hence this PR.

We will like to have this backported after it has been merged.
@bjhaid
Copy link
Contributor Author

bjhaid commented Apr 27, 2021

thanks @olemarkus, addressed

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: olemarkus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Apr 27, 2021
@k8s-ci-robot k8s-ci-robot merged commit e035dec into kubernetes:master Apr 27, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Apr 27, 2021
bjhaid added a commit to bjhaid/kops that referenced this pull request Apr 27, 2021
@bjhaid
Add enable-host-reachable-services to 1.8 and generic cilium.
af1189b 
This completes kubernetes#11333. I missed these updating these other template
files when I submitted kubernetes#11333.
This was referenced Apr 27, 2021
Merged
Closed
bjhaid added a commit to bjhaid/kops that referenced this pull request Apr 28, 2021
@bjhaid
Add enable-host-reachable-services to 1.8 and generic cilium.
0df968e 
This completes kubernetes#11333. I missed these updating these other template
files when I submitted kubernetes#11333.
bjhaid added a commit to bjhaid/kops that referenced this pull request Apr 28, 2021
@bjhaid
Add enable-host-reachable-services to 1.8 and generic cilium.
6d9ba8a 
This completes kubernetes#11333. I missed these updating these other template
files when I submitted kubernetes#11333.
hakman pushed a commit to hakman/kops that referenced this pull request Apr 28, 2021
@bjhaid @hakman
Add enable-host-reachable-services to 1.8 and generic cilium.
ed0a1c5 
This completes kubernetes#11333. I missed these updating these other template
files when I submitted kubernetes#11333.
@hakman hakman mentioned this pull request Apr 28, 2021
Merged
k8s-ci-robot added a commit that referenced this pull request Apr 28, 2021
@k8s-ci-robot
Merge pull request #11348 from hakman/automated-cherry-pick-of-#11333-#…
3c59448 
…11337-upstream-release-1.20

Automated cherry pick of #11333: Add support for configuring Cilium #11337: Add `enable-host-reachable-services` to 1.8 and generic
Deepak1100 pushed a commit to Deepak1100/kops that referenced this pull request May 5, 2021
@bjhaid @Deepak1100
Add enable-host-reachable-services to 1.8 and generic cilium.
1c1f225 
This completes kubernetes#11333. I missed these updating these other template
files when I submitted kubernetes#11333.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@olemarkus olemarkus olemarkus approved these changes

@johngmyers johngmyers Awaiting requested review from johngmyers

@joshbranham joshbranham Awaiting requested review from joshbranham

Assignees

@olemarkus olemarkus

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/addons area/api cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.21
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@bjhaid @k8s-ci-robot @joshbranham @olemarkus