Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix DNS egress L7 policy in ENI mode #8837

Merged
merged 1 commit into from
Aug 10, 2019
Merged

Fix DNS egress L7 policy in ENI mode #8837

merged 1 commit into from
Aug 10, 2019

Conversation

joestringer
Copy link
Member

@joestringer joestringer commented Aug 8, 2019 •
edited

ENI mode was disabling L7 proxy entirely previously due to an overeager check in the datapath to avoid redirecting to the proxy in cases where the endpoint devices are IPVLAN devices.

These fixes enable the following cases in ENI mode:

  • Egress L7 policy plus no ingress policy

Fixes: #8813

Future work:

This change is Reviewable

@joestringer joestringer added pending-review sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. labels Aug 8, 2019
@joestringer joestringer requested review from a team August 8, 2019 04:57
@joestringer
Copy link
Member Author

test-me-please

@coveralls
Copy link

coveralls commented Aug 8, 2019 •
edited

Coverage Status

Coverage decreased (-0.009%) to 44.169% when pulling dea7a69 on joestringer:submit/eni-proxy into a234302 on cilium:master.

borkmann
borkmann reviewed Aug 8, 2019
View reviewed changes
Copy link
Member

@borkmann borkmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't ENI mode simply define ENABLE_HOST_REDIRECT then?
L7 with ipvlan is not supported (https://cilium.readthedocs.io/en/stable/gettingstarted/ipvlan/) at this point.

@tgraf
Copy link
Member

tgraf commented Aug 8, 2019

@joestringer I remember that for the lyft-cni chaining, it was required to specifically route via cilium_host in order to get the redirect to work. It's interesting that this is no longer the case apparently. Could also be specific to kernel versions. I remember @jrfastab saw similar issues related to the encryption work.

@ianvernon ianvernon added this to To Merge To Master in 1.6.0-rc6 Aug 8, 2019
@joestringer
Copy link
Member Author

joestringer commented Aug 9, 2019 •
edited

Enabling host redirect fixes DNS egress proxy iirc, but had other problems. Long and short is that right now if I just enable host redirect it causes ingress proxy at destination to receive SYN, reply with SYN/ACK, then fail to receive the ACK (and retransmit SYN/ACK ad infinitum). SYN/ACK reaches source node which responds with ACK, which arrives to eth1 of destination but I haven't been able to trace it further than that yet.

EDIT: Here's a rough sense of the netfilter stack paths for this configuration:

[240175.580516] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=253 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                               
[240175.594056] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=253 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                               
[240175.607625] CILIUM_PRE_nat:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=253 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                               
[240175.621522] CILIUM_FORWARD:     IN=eth1 OUT=lxc61b3fb1d7642 MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=252 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                
[240175.637058] CILIUM_POST_mangle: IN= OUT=lxc61b3fb1d7642 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=252 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                                                                  
[240175.649602] CILIUM_POST_nat:    IN= OUT=lxc61b3fb1d7642 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=252 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                                                                  
[240175.662060] CILIUM_PRE_raw:     IN=cilium_host OUT= MAC=66:70:85:22:a1:91:aa:6a:86:da:04:ac:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=251 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0 MARK=0xe83b0200                        
[240175.678050] CILIUM_PRE_mangle:  IN=cilium_host OUT= MAC=66:70:85:22:a1:91:aa:6a:86:da:04:ac:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=251 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0 MARK=0xe83b0200                        
[240175.694148] INPUT:              IN=cilium_host OUT= MAC=66:70:85:22:a1:91:aa:6a:86:da:04:ac:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=251 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0 MARK=0x200                             
[240175.709985] CILIUM_OUTPUT_raw:  IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240175.722579] OUTPUT:             IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240175.735015] CILIUM_POST_mangle: IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240175.747619] CILIUM_PRE_raw:     IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240175.762845] CILIUM_PRE_mangle:  IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240175.778255] CILIUM_FORWARD:     IN=cilium_net OUT=eth1 MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                     
[240175.793722] CILIUM_POST_mangle: IN= OUT=eth1 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                                                                             
[240175.807333] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65418 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0                                                 
[240175.820700] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65418 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0                                                 
[240175.834169] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65419 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240175.847866] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65419 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240176.464183] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65420 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240176.478037] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65420 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240176.730940] CILIUM_OUTPUT_raw:  IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240176.743434] OUTPUT:             IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240176.756019] CILIUM_POST_mangle: IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240176.768717] CILIUM_PRE_raw:     IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240176.784031] CILIUM_PRE_mangle:  IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240176.799456] CILIUM_FORWARD:     IN=cilium_net OUT=eth1 MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                     
[240176.814885] CILIUM_POST_mangle: IN= OUT=eth1 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                                                                             
[240176.828098] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65421 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0                                                 
[240176.947137] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65421 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0

@joestringer
datapath: Enable host redirect in ENI mode
dea7a69 
Signed-off-by: Joe Stringer <joe@cilium.io>
@joestringer joestringer changed the title Fix some L7 policy cases in ENI mode Fix DNS egress L7 policy in ENI mode Aug 9, 2019
@joestringer
Copy link
Member Author

I've updated the PR to simplify it, this is easier to justify and it fixes the specific case that was reported.

Getting ingress policy to work in ENI mode across multiple nodes will require additional work.

@joestringer
Copy link
Member Author

test-me-please

@ianvernon ianvernon merged commit a96d7f4 into cilium:master Aug 10, 2019
@joestringer joestringer deleted the submit/eni-proxy branch August 10, 2019 16:27
@ianvernon ianvernon moved this from To Merge To Master to To Backport in 1.6.0-rc6 Aug 12, 2019
@ianvernon ianvernon mentioned this pull request Aug 12, 2019
Merged
@ianvernon ianvernon moved this from To Backport to Backport Pending in 1.6.0-rc6 Aug 12, 2019
@ianvernon ianvernon moved this from Backport Pending to Done in 1.6.0-rc6 Aug 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@borkmann borkmann borkmann approved these changes

@ianvernon ianvernon ianvernon approved these changes

Assignees
No one assigned
Labels
sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
No open projects
1.6.0-rc6
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

DSN resolution does not work with Cillium agent in ENI IPAM allocation mode
5 participants
@joestringer @coveralls @tgraf @borkmann @ianvernon