Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: connectivity test check-log-errors: Link not found #1894

Open
michi-covalent opened this issue Aug 4, 2023 · 11 comments
Open

CI: connectivity test check-log-errors: Link not found #1894

michi-covalent opened this issue Aug 4, 2023 · 11 comments
Labels
area/CI Continuous Integration testing issue or flake ci/flake Issues tracking failing (integration or unit) tests. needs/triage This issue requires triaging to establish the root cause.

Comments

@michi-covalent
Copy link
Contributor

CI failure

from a cilium-cli scheduled kind workflow run with cilium v1.14.0:

https://github.com/cilium/cilium-cli/actions/runs/5763408772/job/15625139472

  • Copy-paste the output the test failure.
  ❌ Found 1 logs matching list of errors that must be investigated:
2023-08-04T14:55:27.946943329Z level=error msg="Link not found" device=lxc347d09aa7056 subsys=datapath-loader
  🐛 Finalizing Test check-log-errors

cilium-sysdumps.zip
@michi-covalent michi-covalent added area/CI Continuous Integration testing issue or flake needs/triage This issue requires triaging to establish the root cause. ci/flake Issues tracking failing (integration or unit) tests. labels Aug 4, 2023
@michi-covalent
Copy link
Contributor Author

another one: https://github.com/cilium/cilium-cli/actions/runs/5792120557/job/15697942532

@michi-covalent
Copy link
Contributor Author

another occurrence: https://github.com/cilium/cilium-cli/actions/runs/5806751648/job/15740266657

cilium-sysdumps.zip

@michi-covalent
Copy link
Contributor Author

one more https://github.com/cilium/cilium-cli/actions/runs/5841891204/job/15842464838

cilium-sysdumps.zip

@michi-covalent
Copy link
Contributor Author

another one https://github.com/cilium/cilium-cli/actions/runs/6304786593/job/17116889843?pr=1984

cilium-sysdumps.zip

@giorio94
Copy link
Member

Another one: https://github.com/cilium/cilium-cli/actions/runs/6564555844/job/17831153648
Sysdump: cilium-sysdumps.zip

@giorio94 giorio94 mentioned this issue Oct 19, 2023
Merged
@michi-covalent
Copy link
Contributor Author

one more 😿 https://github.com/cilium/cilium-cli/actions/runs/6857820846/job/18647581602?pr=2108

  Found "2023-11-14T01:26:06.147950722Z level=error msg=\"Link not found\" device=lxc7f44cc4ef69e subsys=datapath-loader" in logs 1 times

  Found "2023-11-14T01:26:06.150879461Z level=error msg=\"Link not found\" device=lxcd7f7547b283f subsys=datapath-loader" in logs 1 times

  Found "2023-11-14T01:26:06.151005727Z level=warning msg=\"JoinEP: Failed to attach program(s)\" containerID=d8c7454117 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3634 error=\"getting interface lxcd7f7547b283f by name: Link not found\" file-path=3634_next/bpf_lxc.o identity=22463 ipv4=10.244.0.83 ipv6= k8sPodName=kube-system/hubble-ui-78b9fbc9cb-6qmlz subsys=datapath-loader veth=lxcd7f7547b283f" in logs 1 times

  Found "2023-11-14T01:26:06.151068905Z level=error msg=\"Error while rewriting endpoint BPF program\" containerID=d8c7454117 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3634 error=\"getting interface lxcd7f7547b283f by name: Link not found\" identity=22463 ipv4=10.244.0.83 ipv6= k8sPodName=kube-system/hubble-ui-78b9fbc9cb-6qmlz subsys=endpoint" in logs 1 times

  Found "2023-11-14T01:26:06.151421122Z level=error msg=\"endpoint regeneration failed\" containerID=d8c7454117 datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3634 error=\"getting interface lxcd7f7547b283f by name: Link not found\" identity=22463 ipv4=10.244.0.83 ipv6= k8sPodName=kube-system/hubble-ui-78b9fbc9cb-6qmlz subsys=endpoint" in logs 1 times

cilium-junits.zip
cilium-sysdumps.zip

@michi-covalent
Copy link
Contributor Author

hit this in #2136

https://github.com/cilium/cilium-cli/actions/runs/7039057768/job/19157302289?pr=2136

cilium-junits.zip
cilium-sysdumps.zip

@giorio94
Copy link
Member

Another one: https://github.com/cilium/cilium-cli/actions/runs/7259322671/job/19776234471?pr=2194

@giorio94
Copy link
Member

giorio94 commented Jan 9, 2024

Another one: https://github.com/cilium/cilium-cli/actions/runs/7459445768/job/20295468816?pr=2191

@thorn3r
Copy link
Contributor

thorn3r commented Jan 31, 2024

Hit in https://github.com/cilium/cilium-cli/actions/runs/7714719229/attempts/2?pr=2185
PR: #2185

@sayboras sayboras mentioned this issue Jan 31, 2024
Closed
2 tasks
@michi-covalent
Copy link
Contributor Author

another one 😢 https://github.com/cilium/cilium-cli/actions/runs/8068163331/job/22040261074

michi-covalent added a commit that referenced this issue Feb 27, 2024
@michi-covalent
connectivity: Add "Link not found" in the exception list
be5680e 
This "Link not found" message shows up frequently, causing the
connectivity test to fail. Add the message to the error / warning log
exceptions until #1894 is resolved.

Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
@michi-covalent michi-covalent mentioned this issue Feb 27, 2024
Closed
michi-covalent added a commit that referenced this issue Feb 27, 2024
@michi-covalent
connectivity: Add "Link not found" in the exception list
9f5dcc2 
This "Link not found" message shows up frequently, causing the
connectivity test to fail. Add the message to the error / warning log
exceptions until #1894 is resolved.

Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
michi-covalent added a commit that referenced this issue Feb 27, 2024
@michi-covalent
connectivity: Add "Link not found" in the exception list
542fbb8 
This error shows up frequently, causing the connectivity test to fail.
Add the message to the error log exceptions until #1894 is resolved.

Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
michi-covalent added a commit that referenced this issue Feb 28, 2024
@michi-covalent
connectivity: Add "Link not found" in the exception list
29d5ad0 
This error shows up frequently, causing the connectivity test to fail.
Add the message to the error log exceptions until #1894 is resolved.
More specifically, ignore these two messages:

    level=error msg="Link not found" device=... subsys=datapath-loader
    level=warning msg="JoinEP: Failed to attach program(s)" error="getting interface by name: Link not found"

Signed-off-by: Michi Mutsuzaki <michi@isovalent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake Issues tracking failing (integration or unit) tests. needs/triage This issue requires triaging to establish the root cause.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thorn3r @michi-covalent @giorio94