Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: Multicluster: pod-to-service: Flow validation failed #16291

Closed
pchaigno opened this issue May 25, 2021 · 4 comments
Closed

CI: Multicluster: pod-to-service: Flow validation failed #16291

pchaigno opened this issue May 25, 2021 · 4 comments
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
CI Force

Comments

@pchaigno
Copy link
Member

Happened in a PR changing only the way flags are validated, where Jenkins tests are passing (give or take a few flakes): https://github.com/cilium/cilium/runs/2655001058.
logs_103649.zip
cilium-sysdump-out.zip.zip

---------------------------------------------------------------------------------------------------------------------
🔌 [pod-to-service] Testing cilium-test/client-8655c47fd7-6xxpg -> echo-other-node:8080 (ClusterIP):8080...
---------------------------------------------------------------------------------------------------------------------
❌ curl command "curl -w %***local_ip***:%***local_port*** -> %***remote_ip***:%***remote_port*** = %***response_code***\n --show-error --silent --fail --show-error --connect-timeout 5 --output /dev/null echo-other-node:8080" failed: command terminated with exit code 28
ℹ️  curl error: curl: (28) Connection timed out after 5000 milliseconds

❌ Flow validation failed for pod cilium-test/client-8655c47fd7-6xxpg: 2 failures (first: 0, last: 0, matched: 3, nlog: 5)
✅ DNS request found for pod cilium-test/client-8655c47fd7-6xxpg
✅ DNS response found for pod cilium-test/client-8655c47fd7-6xxpg
✅ SYN found for pod cilium-test/client-8655c47fd7-6xxpg
❌ SYN-ACK and(ip(dst=10.8.1.115),or(tcp(srcPort=8080),tcp(srcPort=8080)),tcpflags(syn,ack)) not found for pod cilium-test/client-8655c47fd7-6xxpg
❌ FIN and(or(and(ip(src=10.8.1.115),or(tcp(dstPort=8080),tcp(dstPort=8080))),and(ip(dst=10.8.1.115),or(tcp(srcPort=8080),tcp(srcPort=8080)))),tcpflags(ack,fin)) not found for pod cilium-test/client-8655c47fd7-6xxpg
📄 Flow logs of pod cilium-test/client-8655c47fd7-6xxpg:
✅May 24 10:07:52.175: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 from-endpoint FORWARDED (UDP)
❓May 24 10:07:52.175: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 to-stack FORWARDED (UDP)
❓May 24 10:07:52.175: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 to-network FORWARDED (UDP)
❓May 24 10:07:52.175: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 from-endpoint FORWARDED (UDP)
❓May 24 10:07:52.175: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 to-stack FORWARDED (UDP)
❓May 24 10:07:52.175: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 to-network FORWARDED (UDP)
❓May 24 10:07:52.177: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 from-network FORWARDED (UDP)
❓May 24 10:07:52.177: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 from-network FORWARDED (UDP)
❓May 24 10:07:52.177: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 from-stack FORWARDED (UDP)
❓May 24 10:07:52.177: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 to-endpoint FORWARDED (UDP)
❓May 24 10:07:52.177: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 from-stack FORWARDED (UDP)
❓May 24 10:07:52.177: cilium-test/client-8655c47fd7-6xxpg:56403 -> kube-system/kube-dns-5d54b45645-m2hds:53 to-endpoint FORWARDED (UDP)
✅May 24 10:07:52.177: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 from-endpoint FORWARDED (UDP)
❓May 24 10:07:52.177: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 to-stack FORWARDED (UDP)
❓May 24 10:07:52.177: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 to-network FORWARDED (UDP)
❓May 24 10:07:52.177: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 from-endpoint FORWARDED (UDP)
❓May 24 10:07:52.177: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 to-stack FORWARDED (UDP)
❓May 24 10:07:52.177: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 to-network FORWARDED (UDP)
❓May 24 10:07:52.178: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 from-network FORWARDED (UDP)
❓May 24 10:07:52.178: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 from-network FORWARDED (UDP)
❓May 24 10:07:52.178: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 from-stack FORWARDED (UDP)
❓May 24 10:07:52.178: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 to-endpoint FORWARDED (UDP)
❓May 24 10:07:52.178: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 from-stack FORWARDED (UDP)
❓May 24 10:07:52.178: kube-system/kube-dns-5d54b45645-m2hds:53 -> cilium-test/client-8655c47fd7-6xxpg:56403 to-endpoint FORWARDED (UDP)
✅May 24 10:07:52.178: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 from-endpoint FORWARDED (TCP Flags: SYN)
❓May 24 10:07:52.178: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 to-stack FORWARDED (TCP Flags: SYN)
❓May 24 10:07:52.178: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 to-network FORWARDED (TCP Flags: SYN)
❓May 24 10:07:53.225: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 from-endpoint FORWARDED (TCP Flags: SYN)
❓May 24 10:07:53.225: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 to-stack FORWARDED (TCP Flags: SYN)
❓May 24 10:07:53.225: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 to-network FORWARDED (TCP Flags: SYN)
❓May 24 10:07:55.273: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 from-endpoint FORWARDED (TCP Flags: SYN)
❓May 24 10:07:55.273: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 to-stack FORWARDED (TCP Flags: SYN)
❓May 24 10:07:55.273: cilium-test/client-8655c47fd7-6xxpg:44134 -> cilium-test/echo-other-node:8080 to-network FORWARDED (TCP Flags: SYN)
❌ [pod-to-service] cilium-test/client-8655c47fd7-6xxpg (10.8.1.115) -> echo-other-node:8080 (ClusterIP) (echo-other-node:8080)
@pchaigno pchaigno added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels May 25, 2021
@pchaigno pchaigno mentioned this issue Jun 1, 2021
Merged
nathanjsweet pushed a commit that referenced this issue Jun 2, 2021
@pchaigno @nathanjsweet
.github: Disable flow validation in flaky tests
49ce8e9 
The flow validation of cilium connectivity test has been reported to be
flaky in several of the ci-xxx tests [1, 2, 3]. To reduce CI noise, we
can disable flow validation until the flakes are fixed.

1 - #16292.
2 - #16291.
3 - #16096.
Signed-off-by: Paul Chaignon <paul@cilium.io>
@stale
Copy link

stale bot commented Jul 30, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jul 30, 2021
@soggiest
Copy link
Contributor

I ran into this same error during a Multicluster run:
CI Run: https://github.com/cilium/cilium-cli/runs/4373094252?check_suite_focus=true
Cilium Sysdump: https://github.com/cilium/cilium-cli/suites/4510490190/artifacts/120542303

@pchaigno
Copy link
Member Author

pchaigno commented Dec 1, 2021

@soggiest Could you open a bug report on the cilium-cli repository? Was it a one-off failure or was it consistently failing?

@soggiest
Copy link
Contributor

soggiest commented Dec 1, 2021

It's failed a couple of times one me, not consistently. I'll open a new ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
No open projects
CI Force
  
Awaiting triage
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pchaigno @soggiest