Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: RuntimePrivilegedUnitTests: arping failed in TestArpPingHandling #14125

Closed
pchaigno opened this issue Nov 23, 2020 · 12 comments · Fixed by #14501
Closed

CI: RuntimePrivilegedUnitTests: arping failed in TestArpPingHandling #14125

pchaigno opened this issue Nov 23, 2020 · 12 comments · Fixed by #14501
Assignees
@brb @aanm
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! needs/triage This issue requires triaging to establish severity and next steps.
Projects
CI Force

Comments

@pchaigno
Copy link
Member

https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/2657/testReport/junit/(root)/Suite-runtime/RuntimePrivilegedUnitTests_Run_Tests/
372c1e59_RuntimePrivilegedUnitTests_Run_Tests.zip

level=error msg="arping failed" error=timeout interface=veth0 ipAddr=9.9.9.250 subsys=linux-datapath
node_linux_test.go:1038:
 c.Assert(found, check.Equals, true)
... obtained bool = false
... expected bool = true

START: node_linux_test.go:133: linuxPrivilegedIPv4OnlyTestSuite.TearDownTest
PASS: node_linux_test.go:133: linuxPrivilegedIPv4OnlyTestSuite.TearDownTest	0.052s

FAIL: node_linux_test.go:950: linuxPrivilegedIPv4OnlyTestSuite.TestArpPingHandling
@pchaigno pchaigno added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels Nov 23, 2020
@brb brb self-assigned this Nov 23, 2020
@pchaigno pchaigno mentioned this issue Nov 23, 2020
Merged
@brb
Copy link
Member

brb commented Nov 23, 2020

The test was recently introduced by #14070.

@brb brb added the needs/triage This issue requires triaging to establish severity and next steps. label Nov 23, 2020
@pchaigno pchaigno added this to To Do (1.9 - Daily Flakes) in CI Force Nov 26, 2020
@pchaigno
Copy link
Member Author

I bumped this to daily flakes since I'm seeing it several times per day, just now in e.g. #14113.

This was referenced Nov 27, 2020
Merged
Merged
@brb
Copy link
Member

brb commented Nov 27, 2020

#14201 might fix this.

@tklauser
Copy link
Member

Hit this on #14197 (https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/2765/) and #14200 (https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/2770/) today.

@pchaigno pchaigno mentioned this issue Nov 28, 2020
Merged
@sayboras sayboras mentioned this issue Nov 29, 2020
Merged
@gandro gandro mentioned this issue Nov 30, 2020
Merged
@jibi jibi mentioned this issue Dec 3, 2020
Merged
@pchaigno pchaigno mentioned this issue Dec 4, 2020
Merged
@jibi jibi mentioned this issue Dec 4, 2020
Merged
This was referenced Dec 8, 2020
Merged
Merged
@aanm aanm mentioned this issue Dec 8, 2020
Merged
@aanm
Copy link
Member

aanm commented Dec 8, 2020

@brb still happening

@brb
Copy link
Member

brb commented Dec 9, 2020

Managed to reproduce it locally running in the loop, continuing investigation.

@brb
Copy link
Member

brb commented Dec 9, 2020

Switching arping implementation to #13112 didn't resolve the issue. Also, I see with tcpdump that the arping response, which waiting times out according to the failure log, is received by the host.

@qmonnet qmonnet mentioned this issue Dec 9, 2020
Merged
3 tasks
This was referenced Dec 9, 2020
Merged
Merged
@pchaigno pchaigno mentioned this issue Dec 10, 2020
Merged
@tklauser tklauser mentioned this issue Dec 11, 2020
Merged
@fristonio fristonio mentioned this issue Dec 14, 2020
Merged
@tklauser tklauser mentioned this issue Dec 21, 2020
Merged
@brb
Copy link
Member

brb commented Dec 21, 2020

Tried running tests with strace for the whole day - no luck reproducing the issue.

@tklauser tklauser mentioned this issue Dec 22, 2020
Merged
@fristonio fristonio mentioned this issue Dec 23, 2020
Merged
@ungureanuvladvictor
Copy link
Member

Hit this in #14478 -- https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/3137/testReport/junit/(root)/Suite-runtime/RuntimePrivilegedUnitTests_Run_Tests/

@ungureanuvladvictor
Copy link
Member

Happened again in https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/3145/testReport/junit/(root)/Suite-runtime/RuntimePrivilegedUnitTests_Run_Tests/ for the same PR as ^^.

@aanm aanm mentioned this issue Dec 29, 2020
Merged
@jrajahalme
Copy link
Member

Still happening: https://jenkins.cilium.io/job/Cilium-PR-Runtime-4.9/3176/testReport/junit/(root)/Suite-runtime/RuntimePrivilegedUnitTests_Run_Tests/

@jrajahalme jrajahalme mentioned this issue Dec 30, 2020
Merged
@brb brb assigned aanm Jan 4, 2021
CI Force automation moved this from To Do (1.9 - Daily Flakes) to Fixed / Done Jan 4, 2021
@jaffcheng jaffcheng mentioned this issue Jan 12, 2021
Merged
brb added a commit that referenced this issue Jan 13, 2021
@brb
arp: Add retries to arping
cbb040e 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
@brb brb mentioned this issue Jan 13, 2021
Merged
brb added a commit that referenced this issue Jan 14, 2021
@brb
arp: Add retries to arping
612d41f 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
aanm pushed a commit that referenced this issue Jan 18, 2021
@brb @aanm
arp: Add retries to arping
2284c76 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
brb added a commit to brb/arping that referenced this issue Jan 18, 2021
@brb
Add retries to arping
3a59ad9 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: cilium/cilium#14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
brb added a commit to brb/arping that referenced this issue Jan 18, 2021
@brb
Add retries to arping
18f32ac 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: cilium/cilium#14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
brb added a commit to brb/arping that referenced this issue Jan 18, 2021
@brb
Add retries to arping
68be1a3 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: cilium/cilium#14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
@brb brb mentioned this issue Jan 18, 2021
Merged
brb added a commit to brb/arping that referenced this issue Jan 18, 2021
@brb
Add retries to arping
00c19fd 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: cilium/cilium#14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
aanm pushed a commit to cilium/arping that referenced this issue Jan 18, 2021
@brb @aanm
Add retries to arping
d0a4684 
It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: cilium/cilium#14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
@tklauser tklauser mentioned this issue Jan 18, 2021
Closed
brb added a commit that referenced this issue Apr 28, 2021
@brb
arp: Add retries to arping
122b31e 
[ upstream commit 2284c76 ]

It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
brb added a commit that referenced this issue Apr 28, 2021
@brb
arp: Add retries to arping
c4c126f 
[ upstream commit 2284c76 ]

It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
tklauser added a commit that referenced this issue Apr 28, 2021
@brb @tklauser
arp: Add retries to arping
d526495 
[ upstream commit 2284c76 ]

It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Co-authored-by: Tobias Klauser <tobias@cilium.io>
Signed-off-by: Martynas Pumputis <m@lambda.lt>
rolinh pushed a commit that referenced this issue Apr 29, 2021
@brb @tklauser @rolinh
arp: Add retries to arping
daaf1f6 
[ upstream commit 2284c76 ]

It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Co-authored-by: Tobias Klauser <tobias@cilium.io>
Signed-off-by: Martynas Pumputis <m@lambda.lt>
rolinh pushed a commit that referenced this issue Apr 29, 2021
@brb @rolinh
arp: Add retries to arping
7364d28 
[ upstream commit 2284c76 ]

It has been observed that sometimes arping fails with "i/o timeout".
Further investigation [1] has shown that this happen due to the kernel
not sending packets. Therefore, to mitigate the issue,try multiple times
to send the request if the timeout error is encountered.

[1]: #14125 (comment)

Signed-off-by: Martynas Pumputis <m@lambda.lt>
@tklauser tklauser mentioned this issue May 19, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brb brb

@aanm aanm

Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! needs/triage This issue requires triaging to establish severity and next steps.
Projects
No open projects
CI Force
  
Fixed / Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pkg/datapath: fix arp ping handling aanm/cilium
6 participants
@brb @tklauser @pchaigno @ungureanuvladvictor @aanm @jrajahalme