Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: ConformanceGKE: pod-to-world-toFQDNs: Flow validation failed #16096

Closed
pchaigno opened this issue May 11, 2021 · 1 comment
Closed

CI: ConformanceGKE: pod-to-world-toFQDNs: Flow validation failed #16096

pchaigno opened this issue May 11, 2021 · 1 comment
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me!
Projects
CI Force

Comments

@pchaigno
Copy link
Member

pchaigno commented May 11, 2021 •
edited

Happened in a PR with unrelated changes (touching only test/): https://github.com/cilium/cilium/runs/2555946860
logs_96673.zip
cilium-sysdump-out.zip(1).zip

---------------------------------------------------------------------------------------------------------------------
🔌 [pod-to-world-toFQDNs] Testing cilium-test/client-5f479986f6-j6ckd -> google.com:443...
---------------------------------------------------------------------------------------------------------------------
⌛ The following command is expected to fail...
✅ curl command "curl -w %***local_ip***:%***local_port*** -> %***remote_ip***:%***remote_port*** = %***response_code***\n --show-error --silent --fail --show-error --connect-timeout 5 --output /dev/null https://google.com" failed as expected: command terminated with exit code 28
❌ Flow validation failed for pod cilium-test/client-5f479986f6-j6ckd: 3 failures (first: 0, last: 0, matched: 2, nlog: 5)
✅ DNS request found for pod cilium-test/client-5f479986f6-j6ckd
❌ DNS proxy and(ip(dst=10.16.2.126),or(udp(srcPort=53),tcp(srcPort=53)),dns(query=google.com.,rcode=0)) not found for pod cilium-test/client-5f479986f6-j6ckd
✅ DNS response found for pod cilium-test/client-5f479986f6-j6ckd
❌ SYN and(ip(src=10.16.2.126),tcp(dstPort=443),tcpflags(syn)) not found for pod cilium-test/client-5f479986f6-j6ckd
❌ Drop and(ip(src=10.16.2.126),tcp(dstPort=443),drop) not found for pod cilium-test/client-5f479986f6-j6ckd
📄 Flow logs of pod cilium-test/client-5f479986f6-j6ckd:
✅May 11 13:43:57.758: cilium-test/client-5f479986f6-j6ckd:33596 -> kube-system/kube-dns-6465f78586-x6n2h:53 from-endpoint FORWARDED (UDP)
❓May 11 13:43:57.758: cilium-test/client-5f479986f6-j6ckd:33596 -> kube-system/kube-dns-6465f78586-x6n2h:53 L3-L4 FORWARDED (UDP)
❓May 11 13:43:57.758: cilium-test/client-5f479986f6-j6ckd:33596 -> kube-system/kube-dns-6465f78586-x6n2h:53 to-proxy FORWARDED (UDP)
❓May 11 13:43:57.758: cilium-test/client-5f479986f6-j6ckd:33596 -> kube-system/kube-dns-6465f78586-x6n2h:53 dns-request FORWARDED (DNS Query google.com.cilium-test.svc.cluster.local. A)
❓May 11 13:43:57.760: cilium-test/client-5f479986f6-j6ckd:33596 -> kube-system/kube-dns-6465f78586-x6n2h:53 from-endpoint FORWARDED (UDP)
❓May 11 13:43:57.760: cilium-test/client-5f479986f6-j6ckd:33596 -> kube-system/kube-dns-6465f78586-x6n2h:53 to-proxy FORWARDED (UDP)
❓May 11 13:43:57.760: cilium-test/client-5f479986f6-j6ckd:33596 -> kube-system/kube-dns-6465f78586-x6n2h:53 dns-request FORWARDED (DNS Query google.com.cilium-test.svc.cluster.local. AAAA)
✅May 11 13:43:57.761: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:33596 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy google.com.cilium-test.svc.cluster.local. A))
❓May 11 13:43:57.761: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:33596 from-proxy FORWARDED (UDP)
❓May 11 13:43:57.761: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:33596 to-endpoint FORWARDED (UDP)
❓May 11 13:43:57.761: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:33596 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy google.com.cilium-test.svc.cluster.local. AAAA))
❓May 11 13:43:57.761: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:33596 from-proxy FORWARDED (UDP)
❓May 11 13:43:57.761: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:33596 to-endpoint FORWARDED (UDP)
❓May 11 13:43:57.762: cilium-test/client-5f479986f6-j6ckd:37957 -> kube-system/kube-dns-6465f78586-x6n2h:53 from-endpoint FORWARDED (UDP)
❓May 11 13:43:57.762: cilium-test/client-5f479986f6-j6ckd:37957 -> kube-system/kube-dns-6465f78586-x6n2h:53 L3-L4 FORWARDED (UDP)
❓May 11 13:43:57.762: cilium-test/client-5f479986f6-j6ckd:37957 -> kube-system/kube-dns-6465f78586-x6n2h:53 to-proxy FORWARDED (UDP)
❓May 11 13:43:57.762: cilium-test/client-5f479986f6-j6ckd:37957 -> kube-system/kube-dns-6465f78586-x6n2h:53 from-endpoint FORWARDED (UDP)
❓May 11 13:43:57.762: cilium-test/client-5f479986f6-j6ckd:37957 -> kube-system/kube-dns-6465f78586-x6n2h:53 to-proxy FORWARDED (UDP)
❓May 11 13:43:57.762: cilium-test/client-5f479986f6-j6ckd:37957 -> kube-system/kube-dns-6465f78586-x6n2h:53 dns-request FORWARDED (DNS Query google.com.svc.cluster.local. A)
❓May 11 13:43:57.762: cilium-test/client-5f479986f6-j6ckd:37957 -> kube-system/kube-dns-6465f78586-x6n2h:53 dns-request FORWARDED (DNS Query google.com.svc.cluster.local. AAAA)
❓May 11 13:43:57.763: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:37957 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy google.com.svc.cluster.local. A))
❓May 11 13:43:57.763: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:37957 from-proxy FORWARDED (UDP)
❓May 11 13:43:57.763: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:37957 to-endpoint FORWARDED (UDP)
❓May 11 13:43:57.763: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:37957 dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy google.com.svc.cluster.local. AAAA))
❓May 11 13:43:57.763: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:37957 from-proxy FORWARDED (UDP)
❓May 11 13:43:57.763: kube-system/kube-dns-6465f78586-x6n2h:53 -> cilium-test/client-5f479986f6-j6ckd:37957 to-endpoint FORWARDED (UDP)
❓May 11 13:43:57.763: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 from-endpoint FORWARDED (UDP)
❓May 11 13:43:57.763: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❓May 11 13:43:57.763: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❓May 11 13:43:57.764: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 from-endpoint FORWARDED (UDP)
❓May 11 13:43:57.764: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❓May 11 13:43:57.764: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❓May 11 13:44:00.266: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 from-endpoint FORWARDED (UDP)
❓May 11 13:44:00.266: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❓May 11 13:44:00.266: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❓May 11 13:44:00.266: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 from-endpoint FORWARDED (UDP)
❓May 11 13:44:00.266: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❓May 11 13:44:00.266: cilium-test/client-5f479986f6-j6ckd:43141 -> kube-system/kube-dns-6465f78586-d8tjq:53 Policy denied DROPPED (UDP)
❌ [pod-to-world-toFQDNs] cilium-test/client-5f479986f6-j6ckd (10.16.2.126) -> google.com (google.com)
@pchaigno pchaigno added area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me! labels May 11, 2021
@pchaigno pchaigno mentioned this issue May 11, 2021
Merged
@pchaigno pchaigno mentioned this issue Jun 1, 2021
Merged
nathanjsweet pushed a commit that referenced this issue Jun 2, 2021
@pchaigno @nathanjsweet
.github: Disable flow validation in flaky tests
49ce8e9 
The flow validation of cilium connectivity test has been reported to be
flaky in several of the ci-xxx tests [1, 2, 3]. To reduce CI noise, we
can disable flow validation until the flakes are fixed.

1 - #16292.
2 - #16291.
3 - #16096.
Signed-off-by: Paul Chaignon <paul@cilium.io>
This was referenced Jun 29, 2021
Closed
Merged
@aanm aanm mentioned this issue Jul 3, 2021
Merged
@pchaigno
Copy link
Member Author

pchaigno commented Jul 5, 2021

Likely fixed, we've had a lot of flow validation improvements in cilium-cli's recent versions.

@pchaigno pchaigno closed this as completed Jul 5, 2021
@twpayne twpayne mentioned this issue Aug 9, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/CI Continuous Integration testing issue or flake ci/flake This is a known failure that occurs in the tree. Please investigate me!
Projects
No open projects
CI Force
  
Awaiting triage
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pchaigno