Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Per-node configuration overrides (v2) #22656

Merged
merged 6 commits into from
Dec 15, 2022

Conversation

squeed
Copy link
Contributor

@squeed squeed commented Dec 9, 2022

This un-reverts #22163, which was reverted in #22630. The only difference

The message from 22163 is included below.


This adds the ability to tweak configuration on a per-node basis, as designed in #22036. Specifically, there is an InitContainer that implements the configuration resolution logic. It writes a directory that imitates ConfigMaps, so there are no code changes in the agent itself.

This adds a new type, ClilumNodeConfig, that allows for tweaking configuration flags on a fine-grained basis. For example, you could create

apiVersion: cilium.io/v2alpha1
kind: CiliumNodeConfig
metadata:
  namespace: kube-system
  name: kube-proxy-replacement
spec:
  nodeSelector: 
    matchLabels:
      io.cilium.kube-proxy-replacement: "" 
  defaults:
    kube-proxy-replacement: strict

then add a NodeAntiAffinity to the kube-proxy daemonset. Then, one by one, label nodes with io.cilium.kube-proxy-replacement: "". This will cause that node to begin using this particular configuration override.

Fixes: #22036

Add per-node configuration overrides. There is a new Kubernetes resource type, CiliumNodeConfig, which allows for fine-grained configuration of Nodes based on label selectors.

(Note: This also includes a CI workaround for the extra roles; I'll fix that in cilium-cli and remove it shortly).

@squeed squeed requested review from a team as code owners December 9, 2022 15:21
@squeed squeed requested a review from thorn3r December 9, 2022 15:21
@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Dec 9, 2022
@github-actions github-actions bot added the kind/community-contribution This was a contribution made by a community member. label Dec 9, 2022
@squeed squeed added release-note/minor This PR changes functionality that users may find relevant to operating Cilium. and removed kind/community-contribution This was a contribution made by a community member. labels Dec 9, 2022
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Dec 9, 2022
@squeed
Copy link
Contributor Author

squeed commented Dec 9, 2022

/test

@squeed
Copy link
Contributor Author

squeed commented Dec 9, 2022

This needs cilium/cilium-cli#1286 to merge for CI to be green (I'm not sure why the previous PR worked).

@aanm aanm added release-note/major This PR introduces major new functionality to Cilium. release-blocker/1.13 This issue will prevent the release of the next version of Cilium. needs-backport/1.13 and removed release-note/minor This PR changes functionality that users may find relevant to operating Cilium. labels Dec 9, 2022
@maintainer-s-little-helper maintainer-s-little-helper bot added this to Needs backport from master in 1.13.0-rc4 Dec 9, 2022
@squeed squeed requested a review from a team as a code owner December 12, 2022 15:34
@joestringer joestringer added backport-pending/1.13 The backport for Cilium 1.13.x for this PR is in progress. and removed needs-backport/1.13 labels Dec 21, 2022
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Needs backport from master to Backport pending to v1.13 in 1.13.0-rc4 Dec 21, 2022
@joestringer joestringer added backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. and removed backport-pending/1.13 The backport for Cilium 1.13.x for this PR is in progress. labels Dec 22, 2022
@joestringer joestringer moved this from Backport pending to v1.13 to Backport done to v1.10 in 1.13.0-rc4 Dec 22, 2022
sayboras added a commit to sayboras/cilium that referenced this pull request Dec 22, 2022
This is to make sure that cilium operator is having update permission
for the newly added CiliumNodeConfigs CRD, otherwise, we will have the
below issue:

```
2022-12-21T05:29:34.477230831Z level=fatal msg="Unable to register CRDs" error="Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"ciliumnodeconfigs.cilium.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope" subsys=cilium-operator-generic
```

Relates: cilium#22656
Signed-off-by: Tam Mach <tam.mach@cilium.io>
joamaki pushed a commit that referenced this pull request Dec 23, 2022
This is to make sure that cilium operator is having update permission
for the newly added CiliumNodeConfigs CRD, otherwise, we will have the
below issue:

```
2022-12-21T05:29:34.477230831Z level=fatal msg="Unable to register CRDs" error="Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"ciliumnodeconfigs.cilium.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope" subsys=cilium-operator-generic
```

Relates: #22656
Signed-off-by: Tam Mach <tam.mach@cilium.io>
aditighag pushed a commit to aditighag/cilium that referenced this pull request Jan 9, 2023
[ upstream commit c46eb0b ]

This is to make sure that cilium operator is having update permission
for the newly added CiliumNodeConfigs CRD, otherwise, we will have the
below issue:

```
2022-12-21T05:29:34.477230831Z level=fatal msg="Unable to register CRDs" error="Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"ciliumnodeconfigs.cilium.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope" subsys=cilium-operator-generic
```

Relates: cilium#22656
Signed-off-by: Tam Mach <tam.mach@cilium.io>
Signed-off-by: Aditi Ghag <aditi@cilium.io>
aditighag pushed a commit that referenced this pull request Jan 10, 2023
[ upstream commit c46eb0b ]

This is to make sure that cilium operator is having update permission
for the newly added CiliumNodeConfigs CRD, otherwise, we will have the
below issue:

```
2022-12-21T05:29:34.477230831Z level=fatal msg="Unable to register CRDs" error="Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"ciliumnodeconfigs.cilium.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope" subsys=cilium-operator-generic
```

Relates: #22656
Signed-off-by: Tam Mach <tam.mach@cilium.io>
Signed-off-by: Aditi Ghag <aditi@cilium.io>
squeed added a commit that referenced this pull request Jan 16, 2023
Introduced in #22656, now that cilium-cli install can handle this case,
we can remove this workaround.

Fixes: #22517
Reverts: 43cb8e9

Signed-off-by: Casey Callendrello <cdc@isovalent.com>
joestringer pushed a commit that referenced this pull request Jan 18, 2023
Introduced in #22656, now that cilium-cli install can handle this case,
we can remove this workaround.

Fixes: #22517
Reverts: 43cb8e9

Signed-off-by: Casey Callendrello <cdc@isovalent.com>
qmonnet pushed a commit that referenced this pull request Jan 31, 2023
[ upstream commit d67aed5 ]

Introduced in #22656, now that cilium-cli install can handle this case,
we can remove this workaround.

[ Backport note: Fixed conflict related to most touched filed being
    removed in branch v1.13 with commit e845e1d ("Prepare v1.13
    stable branch"). ]

Fixes: #22517
Reverts: 43cb8e9

Signed-off-by: Casey Callendrello <cdc@isovalent.com>
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
qmonnet pushed a commit that referenced this pull request Feb 1, 2023
[ upstream commit d67aed5 ]

Introduced in #22656, now that cilium-cli install can handle this case,
we can remove this workaround.

[ Backport note: Fixed conflict related to most touched filed being
    removed in branch v1.13 with commit e845e1d ("Prepare v1.13
    stable branch"). ]

Fixes: #22517
Reverts: 43cb8e9

Signed-off-by: Casey Callendrello <cdc@isovalent.com>
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
giorio94 added a commit to giorio94/cilium-cli that referenced this pull request Feb 10, 2023
This commit adds the removal of the cilium-config-agent role and
rolebinding (introduced in cilium/cilium#22656) during the
uninstallation process, to ensure that the status of the cluster is
properly restored.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
giorio94 added a commit to giorio94/cilium-cli that referenced this pull request Feb 10, 2023
This commit adds the removal of the cilium-config-agent role and
rolebinding (introduced in cilium/cilium#22656) during the
uninstallation process, to ensure that the status of the cluster is
properly restored.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
tklauser pushed a commit to cilium/cilium-cli that referenced this pull request Feb 10, 2023
This commit adds the removal of the cilium-config-agent role and
rolebinding (introduced in cilium/cilium#22656) during the
uninstallation process, to ensure that the status of the cluster is
properly restored.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
YutaroHayakawa pushed a commit to YutaroHayakawa/cilium that referenced this pull request Mar 17, 2023
[ upstream commit c46eb0b ]

This is to make sure that cilium operator is having update permission
for the newly added CiliumNodeConfigs CRD, otherwise, we will have the
below issue:

```
2022-12-21T05:29:34.477230831Z level=fatal msg="Unable to register CRDs" error="Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"ciliumnodeconfigs.cilium.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope" subsys=cilium-operator-generic
```

Relates: cilium#22656
Signed-off-by: Tam Mach <tam.mach@cilium.io>
Signed-off-by: Aditi Ghag <aditi@cilium.io>
Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
YutaroHayakawa pushed a commit to YutaroHayakawa/cilium that referenced this pull request Mar 17, 2023
[ upstream commit d67aed5 ]

Introduced in cilium#22656, now that cilium-cli install can handle this case,
we can remove this workaround.

[ Backport note: Fixed conflict related to most touched filed being
    removed in branch v1.13 with commit e845e1d ("Prepare v1.13
    stable branch"). ]

Fixes: cilium#22517
Reverts: 43cb8e9

Signed-off-by: Casey Callendrello <cdc@isovalent.com>
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
michi-covalent pushed a commit to michi-covalent/cilium that referenced this pull request May 30, 2023
This commit adds the removal of the cilium-config-agent role and
rolebinding (introduced in cilium#22656) during the
uninstallation process, to ensure that the status of the cluster is
properly restored.

Signed-off-by: Marco Iorio <marco.iorio@isovalent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-blocker/1.13 This issue will prevent the release of the next version of Cilium. release-note/major This PR introduces major new functionality to Cilium.
Projects
No open projects
1.13.0-rc4
Backport done to v1.13
Development

Successfully merging this pull request may close these issues.

CFP: Per-Node configuration overrides
6 participants