Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DNS proxy intercepts TFTP response and send a DNS error response to TFTP server thousands of times #12241

Closed
jrajahalme opened this issue Jun 23, 2020 · 2 comments · Fixed by #12248
Closed

DNS proxy intercepts TFTP response and send a DNS error response to TFTP server thousands of times #12241

jrajahalme opened this issue Jun 23, 2020 · 2 comments · Fixed by #12248
Assignees
@jrajahalme
Labels
kind/bug This is a bug in the Cilium logic.
Projects
CI Force

Comments

@jrajahalme
Copy link
Member

Found in logs of a failed CI test run (attached):

2020-06-15T21:12:19.262680843Z level=debug msg="Load 1-min: 0.24 5-min: 0.53 15min: 0.29" subsys=loadinfo type=background
2020-06-15T21:12:19.262705013Z level=debug msg="Memory: Total: 7974 Used: 868 (10.88%) Free: 4576 Buffers: 118 Cached: 2411" subsys=loadinfo type=background
2020-06-15T21:12:19.262709878Z level=debug msg="Swap: Total: 0 Used: 0 (0.00%) Free: 0" subsys=loadinfo type=background
2020-06-15T21:12:19.277248092Z level=debug msg="Handling request for /healthz" subsys=health-server
2020-06-15T21:12:20.335073725Z level=debug msg="Controller func execution time: 1.81µs" name=k8s-heartbeat subsys=controller uuid=690f6c74-af4c-11ea-aace-08002731e852
2020-06-15T21:12:20.435336539Z level=debug msg="Controller func execution time: 52.404µs" name=metricsmap-bpf-prom-sync subsys=controller uuid=69107f8b-af4c-11ea-aace-08002731e852
2020-06-15T21:12:20.460905785Z level=debug msg="GET /endpoint request" params="{HTTPRequest:0xc000c55900 Labels:[]}" subsys=daemon
2020-06-15T21:12:22.568019889Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
2020-06-15T21:12:22.575770403Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
2020-06-15T21:12:22.577000721Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
2020-06-15T21:12:22.577305779Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
...
2020-06-15T21:12:37.58690114Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
2020-06-15T21:12:37.586980691Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
2020-06-15T21:12:37.587170772Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
2020-06-15T21:12:37.58872813Z level=debug msg="dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" subsys=fqdn/dnsproxy
2020-06-15T21:12:37.888053984Z level=debug msg="GET /config request" params="{HTTPRequest:0xc0015e4400}" subsys=daemon
2020-06-15T21:12:39.269112166Z level=debug msg="Load 1-min: 1.98 5-min: 0.90 15min: 0.42" subsys=loadinfo type=background
2020-06-15T21:12:39.269133362Z level=debug msg="Memory: Total: 7974 Used: 893 (11.21%) Free: 4546 Buffers: 118 Cached: 2415" subsys=loadinfo type=background
2020-06-15T21:12:39.269151279Z level=debug msg="Swap: Total: 0 Used: 0 (0.00%) Free: 0" subsys=loadinfo type=background
2020-06-15T21:12:39.269153706Z level=debug msg="NAME cilium-agent STATUS S PID 1 CPU: 5.52% MEM: 1.20% CMDLINE: cilium-agent --config-dir=/tmp/cilium/config-map MEM-EXT: RSS: 95 VMS: 751 Data: 0 Stack: 0 Locked: 0 Swap: 0" subsys=loadinfo type=background
2020-06-15T21:12:39.269156178Z level=debug msg="NAME cilium STATUS S PID 780 CPU: 21.58% MEM: 0.61% CMDLINE: cilium monitor -vv MEM-EXT: RSS: 48 VMS: 741 Data: 0 Stack: 0 Locked: 0 Swap: 0" subsys=loadinfo type=background
2020-06-15T21:12:39.300088612Z level=debug msg="Handling request for /healthz" subsys=health-server

The same message repeats 54339 times:

$ grep -c "dnsproxy: Wrote DNS response (12/12 bytes) from 192.168.36.11:33241 to 192.168.36.12:31985" pod-kube-system-cilium-gfpcl-cilium-agent.log
pod-kube-system-cilium-gfpcl-cilium-agent.log:54339

Ref:
4a93942f_K8sServicesTest_Checks_service_across_nodes_with_L4_policy_Tests_NodePort_with_L4_Policy.zip
@jrajahalme jrajahalme added the kind/bug This is a bug in the Cilium logic. label Jun 23, 2020
@jrajahalme
Copy link
Member Author

This DNS response is most likely a minimal error message (size 12 bytes).

The ports are interesting:

Source port 33241 is the DNS proxy listening TPROXY port on k8s2, while a POD on k8s1 is choosing the same port as the ephemeral source port. The POD is accessing a service via host port 192.168.36.12:31985 in k8s2, which gets to the TFTP server on k8s1 (192.168.36.11), on port 69. TFTP server responds and apparently the response gets back to k8s2, in which the host port NATting is reversed, so that the TFTP response packet has the source 192.168.36.12:31985 and destination 192.168.36.11:33241. Now, the port 33241 is a transparent listen port of the DNS proxy in k8s2 (192.168.36.12), but somehow the packet gets routed to the local stack in k8s2, even though the packet is never marked with a local routing mark by a TPROXY rule.

Next the packet sent by the DNS proxy gets to one of the TFTP servers again, which then send a response, which gets to the DNS proxy, which sends a response, etc., this loop continues for 54339 times, until the TFTP server(s) stop responding to the DNS formatted packet that is garbage for TFTP and apparently not logged.

@jrajahalme
Copy link
Member Author

This iptables rule is marking the TFTP response packets for local delivery to the DNS proxy:

[54339:2554107] -A CILIUM_PRE_mangle -m socket --transparent --nowildcard -m comment --comment "cilium: any->pod redirect proxied traffic to host proxy" -j MARK --set-xmark 0x200/0xffffffff

--transparent and --nowildcard should mean that the socket match is done only against accepted sockets with full 5-tuple. This seems to work for TCP, but UDP does not have 'accepted connections' so apparently the socket match here matches the transparent listener on 0.0.0.0:33241.

@jrajahalme jrajahalme changed the title DNS proxy loops sending same response thousands of times DNS proxy intercepts TFTP response and send a DNS error response to TFTP server thousands of times Jun 23, 2020
jrajahalme added a commit that referenced this issue Jun 23, 2020
@jrajahalme
iptables: Remove '--nowildcard' from socket match
bb07b3e 
'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
@jrajahalme jrajahalme mentioned this issue Jun 23, 2020
Merged
@jrajahalme jrajahalme added this to In Progress (Cilium) in CI Force Jun 24, 2020
@jrajahalme jrajahalme self-assigned this Jun 24, 2020
jrajahalme added a commit that referenced this issue Jun 25, 2020
@jrajahalme
test: Add TFTP / DNS collision test.
6d21b01 
Add a test case where a TFTP client in k8s1 uses the DNS proxy port of
k8s2 as it's ephemeral local (source) port number. This exposes a
problem with the iptables rules used in proxy redirection in k8s2, as
the response TFTP packets get intercepted by a socket match rule.

As of now, this test case fails, but the fix in a following commit
fixes the underlying problem.

Related: #12241
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
jrajahalme added a commit that referenced this issue Jun 25, 2020
@jrajahalme
iptables: Remove '--nowildcard' from socket match
47707a3 
'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
jrajahalme added a commit that referenced this issue Jun 29, 2020
@jrajahalme
test: Add TFTP / DNS collision test.
a7f76f7 
Add a test case where a TFTP client in k8s1 uses the DNS proxy port of
k8s2 as it's ephemeral local (source) port number. This exposes a
problem with the iptables rules used in proxy redirection in k8s2, as
the response TFTP packets get intercepted by a socket match rule.

As of now, this test case fails, but the fix in a following commit
fixes the underlying problem.

Related: #12241
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
jrajahalme added a commit that referenced this issue Jun 29, 2020
@jrajahalme
iptables: Remove '--nowildcard' from socket match
bd620cb 
'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
CI Force automation moved this from In Progress (Cilium) to Fixed / Done Jun 29, 2020
joestringer pushed a commit that referenced this issue Jun 29, 2020
@jrajahalme @joestringer
test: Add TFTP / DNS collision test.
069ff48 
Add a test case where a TFTP client in k8s1 uses the DNS proxy port of
k8s2 as it's ephemeral local (source) port number. This exposes a
problem with the iptables rules used in proxy redirection in k8s2, as
the response TFTP packets get intercepted by a socket match rule.

As of now, this test case fails, but the fix in a following commit
fixes the underlying problem.

Related: #12241
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
joestringer pushed a commit that referenced this issue Jun 29, 2020
@jrajahalme @joestringer
iptables: Remove '--nowildcard' from socket match
ca767ee 
'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
christarazi pushed a commit that referenced this issue Jun 30, 2020
@jrajahalme @christarazi
test: Add TFTP / DNS collision test.
5cbfecb 
[ upstream commit 069ff48 ]

Add a test case where a TFTP client in k8s1 uses the DNS proxy port of
k8s2 as it's ephemeral local (source) port number. This exposes a
problem with the iptables rules used in proxy redirection in k8s2, as
the response TFTP packets get intercepted by a socket match rule.

As of now, this test case fails, but the fix in a following commit
fixes the underlying problem.

Related: #12241
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
Signed-off-by: Chris Tarazi <chris@isovalent.com>
christarazi pushed a commit that referenced this issue Jun 30, 2020
@jrajahalme @christarazi
iptables: Remove '--nowildcard' from socket match
abcf05b 
[ upstream commit ca767ee ]

'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
Signed-off-by: Chris Tarazi <chris@isovalent.com>
joestringer pushed a commit that referenced this issue Jun 30, 2020
@jrajahalme @joestringer
test: Add TFTP / DNS collision test.
2f7b7e4 
[ upstream commit 069ff48 ]

Add a test case where a TFTP client in k8s1 uses the DNS proxy port of
k8s2 as it's ephemeral local (source) port number. This exposes a
problem with the iptables rules used in proxy redirection in k8s2, as
the response TFTP packets get intercepted by a socket match rule.

As of now, this test case fails, but the fix in a following commit
fixes the underlying problem.

Related: #12241
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
Signed-off-by: Chris Tarazi <chris@isovalent.com>
joestringer pushed a commit that referenced this issue Jun 30, 2020
@jrajahalme @joestringer
iptables: Remove '--nowildcard' from socket match
465af42 
[ upstream commit ca767ee ]

'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
Signed-off-by: Chris Tarazi <chris@isovalent.com>
christarazi pushed a commit that referenced this issue Jun 30, 2020
@jrajahalme @christarazi
iptables: Remove '--nowildcard' from socket match
2c2a100 
[ upstream commit ca767ee ]

'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
Signed-off-by: Chris Tarazi <chris@isovalent.com>
joestringer pushed a commit that referenced this issue Jun 30, 2020
@jrajahalme @joestringer
iptables: Remove '--nowildcard' from socket match
abd782e 
[ upstream commit ca767ee ]

'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
Signed-off-by: Chris Tarazi <chris@isovalent.com>
jrajahalme added a commit that referenced this issue Jul 1, 2020
@jrajahalme
iptables: Remove '--nowildcard' from socket match
820aa59 
[ upstream commit ca767ee ]

'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
joestringer pushed a commit that referenced this issue Jul 1, 2020
@jrajahalme @joestringer
iptables: Remove '--nowildcard' from socket match
2fe8348 
[ upstream commit ca767ee ]

'--no-wildcard' allows the socket match to find zero-bound (listening)
sockets, which we do not want, as this may intercept (reply) traffic
intended for other nodes when an ephemeral source port number
allocated in one node happens to be the same as the allocated proxy
port number in 'this' node (the node doing the iptables socket match
changed here).

Fixes: #12241
Related: #8864
Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
@brb brb mentioned this issue Jul 6, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jrajahalme jrajahalme

Labels
kind/bug This is a bug in the Cilium logic.
Projects
No open projects
CI Force
  
Fixed / Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

iptables: Remove '--nowildcard' from socket match cilium/cilium
1 participant
@jrajahalme