aks: fix AKS cluster creation following new taint limitations #17529
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nbusseneau
added
area/CI
nbusseneau
force-pushed
the
pr/workflows-fix-aks-pools
branch
3 times, most recently
from
e31024d
to
4500382
Compare
|
Link to test run of AKS workflow: https://github.com/cilium/cilium/actions/runs/1333557658 |
joestringer
requested changes
Oct 11, 2021
There was a problem hiding this comment.
Minor nit in the docs but otherwise makes sense.
I debated a little bit about the use of .. code-block:: bash rather than .. code-block:: shell-session but in this case unless we dress up the # lines with some extra #s at the beginning, I think that shell-session would interpret the comments as commands to run as root. So bash seems fine to me (also consistent with other commands in the same guide).
aanm
requested changes
Oct 11, 2021
nbusseneau
force-pushed
the
pr/workflows-fix-aks-pools
branch
from
4500382
to
5c82cbd
Compare
nbusseneau
force-pushed
the
pr/workflows-fix-aks-pools
branch
from
5c82cbd
to
21e4044
Compare
joestringer
reviewed
Oct 12, 2021
aanm
approved these changes
Oct 12, 2021
Context: we recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402. AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578. It is not possible anymore for us to recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To work around this new limitation, we propose the following workflow instead: - Replace the initial node pool with a system node pool tainted with `CriticalAddonsOnly=true:NoSchedule`, preventing application pods from being scheduled on it. - Create a secondary user node pool tainted with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
See 36be0aa for full context. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
nbusseneau
force-pushed
the
pr/workflows-fix-aks-pools
branch
from
21e4044
to
e5a48c7
Compare
joestringer
approved these changes
Oct 13, 2021
tklauser
approved these changes
Oct 13, 2021
nbusseneau
force-pushed
the
pr/workflows-fix-aks-pools
branch
from
e5a48c7
to
fd85f2e
Compare
|
All reviews are in, no need to run any additional CI as this only modifies documentation and the AKS workflow, removing temp test commit and marking as ready to merge. The following AKS workflows can be re-enabled after merging:
I will also apply similar changes to |
nbusseneau
added
the
ready-to-merge
nbusseneau
added a commit
to cilium/cilium-cli
that referenced
this pull request
Oct 15, 2021
Re-impacted from: cilium/cilium#17529 Context: we recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402. AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578. It is not possible anymore for us to recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To work around this new limitation, we propose the following workflow instead: - Replace the initial node pool with a system node pool tainted with `CriticalAddonsOnly=true:NoSchedule`, preventing application pods from being scheduled on it. - Create a secondary user node pool tainted with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
|
@nbusseneau Could you take a look at how this should be backported, if at all? These files do not exist in 1.10 due to 65eb99a. |
|
@joamaki I added the backport label for the documentation changes. The workflows indeed are already "backported" in a sense, since they live in |
nbusseneau
added a commit
to cilium/cilium-cli
that referenced
this pull request
Oct 18, 2021
Re-impacted from: cilium/cilium#17529 Context: we recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402. AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578. It is not possible anymore for us to recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To work around this new limitation, we propose the following workflow instead: - Replace the initial node pool with a system node pool tainted with `CriticalAddonsOnly=true:NoSchedule`, preventing application pods from being scheduled on it. - Create a secondary user node pool tainted with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
nbusseneau
added a commit
to cilium/cilium-cli
that referenced
this pull request
Oct 19, 2021
Re-impacted from: cilium/cilium#17529 Context: we recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402. AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578. It is not possible anymore for us to recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To work around this new limitation, we propose the following workflow instead: - Replace the initial node pool with a system node pool tainted with `CriticalAddonsOnly=true:NoSchedule`, preventing application pods from being scheduled on it. - Create a secondary user node pool tainted with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
aanm
pushed a commit
to cilium/cilium-cli
that referenced
this pull request
Oct 20, 2021
Re-impacted from: cilium/cilium#17529 Context: we recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402. AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578. It is not possible anymore for us to recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To work around this new limitation, we propose the following workflow instead: - Replace the initial node pool with a system node pool tainted with `CriticalAddonsOnly=true:NoSchedule`, preventing application pods from being scheduled on it. - Create a secondary user node pool tainted with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
joestringer
moved this from Needs backport from master
to Backport done to v1.10
in 1.10.6
aditighag
pushed a commit
to aditighag/cilium-cli
that referenced
this pull request
Apr 21, 2023
Re-impacted from: cilium/cilium#17529 Context: we recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402. AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578. It is not possible anymore for us to recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To work around this new limitation, we propose the following workflow instead: - Replace the initial node pool with a system node pool tainted with `CriticalAddonsOnly=true:NoSchedule`, preventing application pods from being scheduled on it. - Create a secondary user node pool tainted with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
michi-covalent
pushed a commit
to michi-covalent/cilium
that referenced
this pull request
May 30, 2023
Re-impacted from: cilium#17529 Context: we recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402. AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578. It is not possible anymore for us to recommend users taint all nodepools with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being managed by the default AKS CNI plugin. To work around this new limitation, we propose the following workflow instead: - Replace the initial node pool with a system node pool tainted with `CriticalAddonsOnly=true:NoSchedule`, preventing application pods from being scheduled on it. - Create a secondary user node pool tainted with `node.cilium.io/agent-not-ready=true:NoSchedule` to prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them. Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context: we recommend users taint all nodepools with
node.cilium.io/agent-not-ready=true:NoScheduleto prevent application pods from being managed by the default AKS CNI plugin.To this end, the proposed workflow users should follow when installing Cilium into AKS was to replace the initial AKS node pool with a new tainted system node pool, as it is not possible to taint the initial AKS node pool, cf. Azure/AKS#1402.
AKS recently pushed a change on the API side that forbids setting up custom taints on system node pools, cf. Azure/AKS#2578.
It is not possible anymore for us to recommend users taint all nodepools with
node.cilium.io/agent-not-ready=true:NoScheduleto prevent application pods from being managed by the default AKS CNI plugin.To work around this new limitation, we propose the following workflow instead:
CriticalAddonsOnly=true:NoSchedule, preventing application pods from being scheduled on it.node.cilium.io/agent-not-ready=true:NoScheduleto prevent application pods from being scheduled on the user node pool until Cilium is ready to manage them.