v1.11 Backports 2023-04-20 #25011

nbusseneau · 2023-04-20T16:50:52Z

Misc Makefile improvements for quiet mode V=0 #20031 (@joestringer)
pkg/kvstore: Fix for deadlock in etcd status checker #24786 (@hemanthmalla)
daemon: Mark CES feature as beta in agent flag #24850 (@pchaigno)
Fix connectivity issue if nodes share the same name across the clustermesh and wireguard is enabled #24785 (@giorio94)

Once this PR is merged, you can update the PR labels via:

for pr in 20031 24786 24850 24785; do contrib/backporting/set-labels.py $pr done 1.11; done

or with

make add-labels BRANCH=v1.11 ISSUES=20031,24786,24850,24785

[ upstream commit 82d5adc ] When the quiet mode was enabled, the $(CLANG) var would previously have a '@' at the start, which caused errors while attempting to make in this directory because it would be run in the context of a shell rather than directly as a make instruction. Move the $(QUIET) to the start of individual make instructions to resolve this compilation failure. Signed-off-by: Joe Stringer <joe@cilium.io> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

[ upstream commit 29fe753 ] When TESTPKGS only contains unprivileged tests, the PRIV_TEST_PKGS_EVAL evaluation previously filtered down to an empty list of packages that should be tested, and would pass this empty list to dirname, which then reports: dirname: missing operand Try 'dirname --help' for more information. This could happen multiple times during evaluation of the Makefile, and littered the output with no meaning. This could occur even if the privileged tests are not the target being run. Fix this by always adding "." to the list, which evaluates to the root directory of the repository. This causes dirname to succeed. Then, we can filter this root directory back out since there are no privileged tests at this level of the repository. This finally quietens the error. Signed-off-by: Joe Stringer <joe@cilium.io> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

[ upstream commit 9f7e24f ] Pass the verbosity parameters --quiet V=0 to quieten Travis output. Signed-off-by: Joe Stringer <joe@cilium.io> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

[ upstream commit 7f9e0f9 ] The travis logs are frequently polluted with >10K lines of docker pull and build output. While this helps to track the ongoing progress of docker builds that take a long time, it's mostly useless output that developers must scroll past in order to see the useful output. Quieten that output in Travis to just the trigger of building the image plus the final summary that docker outputs. Signed-off-by: Joe Stringer <joe@cilium.io> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

[ upstream commit 9bb669b ] Etcd quorum checks are falsely reported as failing even though connection to etcd is intact. This can cause health checks to fail in both the agent and the operator. This happens due to a deadlock in pkg/kvstore/etcd after a prolonged downtime of etcd. Status check errors are being sent into a channel for the purpose of recreating kvstore connections in clustermesh. However when clustermesh is not used, messages from this channel are never read. The channel uses a buffer of size 128. After etcd has been down long enough to generate 128 errors, we enter a deadlock state. Agent / operator will continue to report etcd quorum failures and inturn health check failures until they're restarted. statusChecker() -> isConnectedAndHasQuorum() -> waitForInitLock() -> goroutine -> for -> ( initLockSucceeded <- err ) -> chan initLockSucceeded returned -> Block on receiving messages from initLockSucceeded channel -> e.statusCheckErrors <- e.latestErrorStatus [Blocked after 128 entries] Blocked goroutines captured from cilium 1.10 operator: goroutine 3309 [chan send, 13456 minutes]: github.com/cilium/cilium/pkg/kvstore.(*etcdClient).statusChecker(0xc00017db30) /go/src/github.com/cilium/cilium/pkg/kvstore/etcd.go:1171 +0x75a created by github.com/cilium/cilium/pkg/kvstore.connectEtcdClient /go/src/github.com/cilium/cilium/pkg/kvstore/etcd.go:801 +0x679 goroutine 7838665 [chan send, 13505 minutes]: g.com/c/cilium/pkg/kvstore.(*etcdClient).waitForInitLock.func1(-,-,-,-) /go/src/github.com/cilium/cilium/pkg/kvstore/etcd.go:433 +0x449 created by github.com/cilium/cilium/pkg/kvstore.(*etcdClient).waitForInitLock /go/src/github.com/cilium/cilium/pkg/kvstore/etcd.go:425 +0x7f Signed-off-by: Hemanth Malla <hemanth.malla@datadoghq.com> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

[ upstream commit a6d0142 ] This commit marks the CiliumEndpointSlice feature as beta (as per the documentation) in the agent flag description. This is necessary because users don't always read the full documentation before turning agent flags on. While at it, change the flag description to match the wording of other flags. Signed-off-by: Paul Chaignon <paul@cilium.io> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

@oulinbao

[ upstream commit 7398de6 ] Currently, the wireguard subsystem in the cilium agent caches information about the known peers by node name only. This can lead to conflicts in case of clustermesh, if nodes in different clusters have the same name, causing in turn connectivity issues. Hence, let's switch to identify peers by full name (i.e., cluster-name/node-name) to ensure uniqueness. This modification does not introduce issues during upgrades, since the node ID is not propagated to the datapath. Fixes: cilium#24227 Reported-by: @oulinbao <oulinbao@163.com> Signed-off-by: Marco Iorio <marco.iorio@isovalent.com> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

nbusseneau · 2023-04-20T16:51:48Z

/test-backport-1.11

Job 'Cilium-PR-K8s-1.19-kernel-4.9' failed:

Click to show.

Test Name

K8sHubbleTest Hubble Observe Test L3/L4 Flow

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.19-kernel-4.9/2822/

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.19-kernel-4.9 so I can create one.

Job 'Cilium-PR-K8s-1.17-kernel-4.9' failed:

Click to show.

Test Name

K8sKafkaPolicyTest Kafka Policy Tests KafkaPolicies

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.17-kernel-4.9/1315/

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.17-kernel-4.9 so I can create one.

Job 'Cilium-PR-K8s-1.21-kernel-4.9' failed:

Click to show.

Test Name

K8sPolicyTest Basic Test checks all kind of Kubernetes policies

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.21-kernel-4.9/2589/

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.21-kernel-4.9 so I can create one.

Job 'Cilium-PR-K8s-1.20-kernel-4.9' failed:

Click to show.

Test Name

K8sPolicyTest Basic Test checks all kind of Kubernetes policies

Failure Output

FAIL: Found 2 k8s-app=cilium logs matching list of errors that must be investigated:

Jenkins URL: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9/1819/

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.20-kernel-4.9 so I can create one.

Job 'Cilium-PR-K8s-1.19-kernel-4.9' hit: #25099 (93.06% similarity)

Job 'Cilium-PR-K8s-1.17-kernel-4.9' hit: #25100 (94.45% similarity)

Job 'Cilium-PR-K8s-1.21-kernel-4.9' has 1 failure but they might be new flake since it also hit 1 known flake: #25101 (93.46% similarity)

pchaigno

My PR looks good. Thanks!

giorio94

Mine too. Thanks!

joestringer

LGTM for #20031

hemanthmalla · 2023-04-20T23:13:04Z

LGTM for my commit. Thanks.

nbusseneau · 2023-04-21T14:53:31Z

Jenkins pipelines failed with CI: [v1.11] Jenkins tests failing on 4.9, agent trying to pass BPF ISA set v3 to init.sh #24394, documenting the failures in the issue.
AKS pipeline failed because K8s 1.23 is not supported on AKS anymore => we are entering one of the scenarios discussed when initially introducing K8s version pinning on stable branches. Since Cilium 1.11 only officially supports up to K8s 1.23, we now have a choice to make:
- Disable AKS testing on 1.11.
- Bump AKS testing to use K8s 1.24 even though it's not officially supported.

I would personally advocate for the first solution.
cc @joestringer @aanm

joestringer · 2023-04-21T19:06:39Z

👍 I think it's fair to say that Cilium branch CI should test the intersection of {k8s releases that were out at the time of cilium release} and {k8s releases supported by the cloud now}. The logical consequence is that if those groups become disjoint then we no longer need to run such tests, as you propose.

We could of course opt to add additional support, but I think it's in everyone's best interest to encourage newer software in cases like this rather than provide more reasons for folks to stick on the older releases.

nbusseneau · 2023-04-24T15:09:56Z

Cool, I will disable AKS testing on 1.11 then!

nbusseneau · 2023-04-24T15:10:03Z

/test-1.16-netnext

sayboras · 2023-04-25T03:39:21Z

/mlh new-flake Cilium-PR-K8s-1.19-kernel-4.9

👍 created #25098

sayboras · 2023-04-25T03:39:42Z

/mlh new-flake Cilium-PR-K8s-1.17-kernel-4.9

👍 created #25099

sayboras · 2023-04-25T03:39:56Z

/mlh new-flake Cilium-PR-K8s-1.21-kernel-4.9

👍 created #25100

sayboras · 2023-04-25T03:40:08Z

/mlh new-flake Cilium-PR-K8s-1.20-kernel-4.9

👍 created #25101

sayboras · 2023-04-25T03:40:24Z

/test-1.17-4.9

sayboras · 2023-04-25T03:40:34Z

/test-1.19-4.9

sayboras · 2023-04-25T03:41:00Z

/test-1.20-4.9

sayboras · 2023-04-25T03:41:07Z

/test-1.21-4.9

sayboras · 2023-04-25T10:22:16Z

Please find below the test failure summary:

test-1.17-4.9

First run failed for Suite-k8s-1.17.K8sPolicyTest Basic Test checks all kind of Kubernetes policies
Second run failed for Suite-k8s-1.17.K8sKafkaPolicyTest Kafka Policy Tests KafkaPolicies

test-1.19-4.9

First run failed for Suite-k8s-1.19.K8sKafkaPolicyTest Kafka Policy Tests KafkaPolicies
Second run failed for Suite-k8s-1.19.K8sHubbleTest Hubble Observe Test L3/L4 Flow

test-1.21-4.9

First run failed for Suite-k8s-1.21.K8sPolicyTest Basic Test checks all kind of Kubernetes policies
Second run failed for
- Suite-k8s-1.21.K8sPolicyTest Basic Test checks all kind of Kubernetes policies
- Suite-k8s-1.21.K8sVerifier Runs the kernel verifier against Cilium's BPF datapath

sayboras · 2023-04-25T10:22:27Z

/test-1.21-4.9

Re-run this test suite to confirm if there is any issue or just flake.

sayboras · 2023-04-25T13:18:22Z

/test-1.21-4.9

Re-run this test suite to confirm if there is any issue or just flake.

Re-run was good ✔️

Required reviews are in, flakes are tracked as part of the below issues. Marking this ready to merge.

#25098
#25099
#25100
#25101

joestringer and others added 7 commits April 20, 2023 16:35

.travis: Make output less verbose

573a5fa

[ upstream commit 9f7e24f ] Pass the verbosity parameters --quiet V=0 to quieten Travis output. Signed-off-by: Joe Stringer <joe@cilium.io> Signed-off-by: Nicolas Busseneau <nicolas@isovalent.com>

nbusseneau requested a review from a team as a code owner April 20, 2023 16:50

nbusseneau added kind/backports This PR provides functionality previously merged into master. backport/1.11 This PR represents a backport for Cilium 1.11.x of a PR that was merged to main. labels Apr 20, 2023

nbusseneau requested review from joestringer, hemanthmalla, pchaigno and giorio94 April 20, 2023 16:50

pchaigno approved these changes Apr 20, 2023

View reviewed changes

giorio94 approved these changes Apr 20, 2023

View reviewed changes

joestringer approved these changes Apr 20, 2023

View reviewed changes

hemanthmalla approved these changes Apr 20, 2023

View reviewed changes

nbusseneau mentioned this pull request Apr 21, 2023

CI: [v1.11] Jenkins tests failing on 4.9, agent trying to pass BPF ISA set v3 to init.sh #24394

Closed

This was referenced Apr 25, 2023

CI: K8sKafkaPolicyTest Kafka Policy Tests KafkaPolicies #25099

Closed

CI: K8sPolicyTest Basic Test checks all kind of Kubernetes policies #25100

Closed

CI: K8sPolicyTest Basic Test checks all kind of Kubernetes policies #25101

Closed

sayboras added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 25, 2023

ti-mo merged commit bac3efe into cilium:v1.11 Apr 26, 2023
50 of 53 checks passed

This was referenced May 12, 2023

Prepare for release v1.11.17 #25428

Closed

Prepare for release v1.11.17 #25445

Closed

Prepare for release v1.11.17 #25446

Closed

Prepare for release v1.11.17 #25447

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.11 Backports 2023-04-20 #25011

v1.11 Backports 2023-04-20 #25011

nbusseneau commented Apr 20, 2023 •

edited by hemanthmalla

nbusseneau commented Apr 20, 2023 •

edited by maintainer-s-little-helper bot

Test Name

Failure Output

Test Name

Failure Output

Test Name

Failure Output

Test Name

Failure Output

pchaigno left a comment

giorio94 left a comment

joestringer left a comment

hemanthmalla commented Apr 20, 2023

nbusseneau commented Apr 21, 2023

joestringer commented Apr 21, 2023

nbusseneau commented Apr 24, 2023

nbusseneau commented Apr 24, 2023

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023 •

edited

sayboras commented Apr 25, 2023

v1.11 Backports 2023-04-20 #25011

v1.11 Backports 2023-04-20 #25011

Conversation

nbusseneau commented Apr 20, 2023 • edited by hemanthmalla

nbusseneau commented Apr 20, 2023 • edited by maintainer-s-little-helper bot

Test Name

Failure Output

Test Name

Failure Output

Test Name

Failure Output

Test Name

Failure Output

pchaigno left a comment

Choose a reason for hiding this comment

giorio94 left a comment

Choose a reason for hiding this comment

joestringer left a comment

Choose a reason for hiding this comment

hemanthmalla commented Apr 20, 2023

nbusseneau commented Apr 21, 2023

joestringer commented Apr 21, 2023

nbusseneau commented Apr 24, 2023

nbusseneau commented Apr 24, 2023

sayboras commented Apr 25, 2023 • edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 • edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 • edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 • edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023

sayboras commented Apr 25, 2023 • edited

sayboras commented Apr 25, 2023

nbusseneau commented Apr 20, 2023 •

edited by hemanthmalla

nbusseneau commented Apr 20, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited by maintainer-s-little-helper bot

sayboras commented Apr 25, 2023 •

edited