test: Redeploy DNS after endpointRoutes reconfiguration #16767
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maintainer-s-little-helper
bot
added
the
dont-merge/needs-release-note-label
maintainer-s-little-helper
bot
removed
the
dont-merge/needs-release-note-label
|
test-me-please EDIT: All green. |
nebril
approved these changes
Jul 5, 2021
jibi
approved these changes
Jul 5, 2021
pchaigno
approved these changes
Jul 5, 2021
In general up until now, Cilium has expected endpointRoutes mode to be set to exactly one value upon deployment and for that value to stay the same for the remainder of operation. Toggling it can lead to a mix of endpoints in different datapath modes which is not well covered in CI. In Github issue #16717 we observed that if the testsuite toggles this setting then we can end up with kubedns pods remaining in endpoint routes mode, even though the rest of the daemon (and other pods) are not configured in this mode. This can lead to connectivity issues in DNS, and a range of test failures in subsequent tests because DNS is broken. Longer term to resolve this, we could improve on Cilium to ensure that users can successfully toggle this setting on or off at runtime and properly handle this case, or alternatively shift all logic over to endpoint-routes mode by default and disable the other option. Given that CI for the master branch is in a poor state due to this issue today, and that part of the issue is CI reconfiguring the datapath state of Cilium during the test setup in an unsupported manner, this commit proposes to force DNS pod redeployment as part of setup any time a test reconfigures the endpointRoutes mode. This should mitigate the testing side issue while we mull over the right longer-term solution. Signed-off-by: Joe Stringer <joe@cilium.io>
joestringer
force-pushed
the
pr/joe/fix-ep-routes-ci
branch
from
9c83570
to
0e30022
Compare
12 tasks
|
test-me-please |
|
On the fence about v1.9 backport, from out-of-band discussion with @pchaigno this flake is likely caused by #16227 which was backported to v1.9. I briefly looked through scheduled CI job runs for v1.9 and didn't observe the issue at all. We'll hold off for now. Now that we know how to detect this problem, it's easy to figure out if we see this particular issue. If we see it on v1.9 we can backport it. |
This was referenced Jul 8, 2021
maintainer-s-little-helper
bot
moved this from Needs backport from master
to Backport pending to v1.10
in 1.10.3
maintainer-s-little-helper
bot
moved this from Backport pending to v1.10
to Backport done to v1.10
in 1.10.3
|
Marking for backport to 1.9 following discussion here: #18031 (comment) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In general up until now, Cilium has expected endpointRoutes mode to be
set to exactly one value upon deployment and for that value to stay the
same for the remainder of operation. Toggling it can lead to a mix of
endpoints in different datapath modes which is not well covered in CI.
In Github issue #16717 we observed that if the testsuite toggles this
setting then we can end up with kubedns pods remaining in endpoint
routes mode, even though the rest of the daemon (and other pods) are not
configured in this mode. This can lead to connectivity issues in DNS,
and a range of test failures in subsequent tests because DNS is broken.
Longer term to resolve this, we could improve on Cilium to ensure that
users can successfully toggle this setting on or off at runtime and
properly handle this case, or alternatively shift all logic over to
endpoint-routes mode by default and disable the other option.
Given that CI for the master branch is in a poor state due to this issue
today, and that part of the issue is CI reconfiguring the datapath state
of Cilium during the test setup in an unsupported manner, this commit
proposes to force DNS pod redeployment as part of setup any time a test
reconfigures the endpointRoutes mode. This should mitigate the testing
side issue while we mull over the right longer-term solution.
Fixes: #16717
Fixes: #16227
Related: #14955