Remove chain jump rules as part of cleanup

[grampelberg]

This does a couple things:

• Adds delete for the jump rules to potentially address All linkerd components in Init:CrashLoopBackOff linkerd2#3563
• Moves cleanup to run before everything instead of during configuration for personal sanity
• Simplifies the nsenter code a little so that the log output includes the commands run
• Improves readability of log output
• Uses iptables-save to get all rules instead of just the nat table
• Fixes a personal pet peeve of if/else statements.

Here's the new output:

Tracing this script execution as [1573775205]
current state
------------------------------------------------------------
:; iptables-save


cleanup
------------------------------------------------------------
:; iptables -t nat -D OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-prerouting/1573775205
iptables v1.6.0: Couldn't load target `PROXY_INIT_OUTPUT':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

:; iptables -t nat -D PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1573775205
iptables v1.6.0: Couldn't load target `PROXY_INIT_REDIRECT':No such file or directory

Try `iptables -h' or 'iptables --help' for more information.

:; iptables -t nat -F PROXY_INIT_OUTPUT
iptables: No chain/target/match by that name.

:; iptables -t nat -X PROXY_INIT_OUTPUT
iptables: No chain/target/match by that name.

:; iptables -t nat -F PROXY_INIT_REDIRECT
iptables: No chain/target/match by that name.

:; iptables -t nat -X PROXY_INIT_REDIRECT
iptables: No chain/target/match by that name.



configuration
------------------------------------------------------------
Will ignore port 4190 on chain PROXY_INIT_REDIRECT
Will ignore port 4191 on chain PROXY_INIT_REDIRECT
Will redirect all INPUT ports to proxy
Ignoring uid 2102
Will ignore port 443 on chain PROXY_INIT_OUTPUT
Redirecting all OUTPUT to 4140


adding rules
------------------------------------------------------------
:; iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/1573775205
:; iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --destination-port 4190 -j RETURN -m comment --comment proxy-init/ignore-port-4190/1573775205
:; iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --destination-port 4191 -j RETURN -m comment --comment proxy-init/ignore-port-4191/1573775205
:; iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port/1573775205
:; iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1573775205
:; iptables -t nat -N PROXY_INIT_OUTPUT -m comment --comment proxy-init/redirect-common-chain/1573775205
:; iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -o lo ! -d 127.0.0.1/32 -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-non-loopback-local-traffic/1573775205
:; iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -j RETURN -m comment --comment proxy-init/ignore-proxy-user-id/1573775205
:; iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN -m comment --comment proxy-init/ignore-loopback/1573775205
:; iptables -t nat -A PROXY_INIT_OUTPUT -p tcp --destination-port 443 -j RETURN -m comment --comment proxy-init/ignore-port-443/1573775205
:; iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 4140 -m comment --comment proxy-init/redirect-all-outgoing-to-proxy-port/1573775205
:; iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1573775205


end state
------------------------------------------------------------
:; iptables-save
# Generated by iptables-save v1.6.0 on Thu Nov 14 23:46:45 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:PROXY_INIT_OUTPUT - [0:0]
:PROXY_INIT_REDIRECT - [0:0]
-A PREROUTING -m comment --comment "proxy-init/install-proxy-init-prerouting/1573775205" -j PROXY_INIT_REDIRECT
-A OUTPUT -m comment --comment "proxy-init/install-proxy-init-output/1573775205" -j PROXY_INIT_OUTPUT
-A PROXY_INIT_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 2102 -m comment --comment "proxy-init/redirect-non-loopback-local-traffic/1573775205" -j PROXY_INIT_REDIRECT
-A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -m comment --comment "proxy-init/ignore-proxy-user-id/1573775205" -j RETURN
-A PROXY_INIT_OUTPUT -o lo -m comment --comment "proxy-init/ignore-loopback/1573775205" -j RETURN
-A PROXY_INIT_OUTPUT -p tcp -m tcp --dport 443 -m comment --comment "proxy-init/ignore-port-443/1573775205" -j RETURN
-A PROXY_INIT_OUTPUT -p tcp -m comment --comment "proxy-init/redirect-all-outgoing-to-proxy-port/1573775205" -j REDIRECT --to-ports 4140
-A PROXY_INIT_REDIRECT -p tcp -m tcp --dport 4190 -m comment --comment "proxy-init/ignore-port-4190/1573775205" -j RETURN
-A PROXY_INIT_REDIRECT -p tcp -m tcp --dport 4191 -m comment --comment "proxy-init/ignore-port-4191/1573775205" -j RETURN
-A PROXY_INIT_REDIRECT -p tcp -m comment --comment "proxy-init/redirect-all-incoming-to-proxy-port/1573775205" -j REDIRECT --to-ports 4143
COMMIT
# Completed on Thu Nov 14 23:46:45 2019

[grampelberg]

The cleanup rules here never worked. Just cleaning our own rules up will add some complexity and potentially paper over more fundamental underlying issues. @adleong and I chatted and agreed that the correct solution is to detect if any of our rules still exist. If they do exist, error out in a reasonable fashion and produce a sane error message to help the users dig into what's going on.

[adleong]

This updated structure and output format is great!

As we discussed, I think making this error out if existing Linkerd rules are detected is a good idea.

[alpeb]

@grampelberg Nice refactor 👍
FWIW, the cleanup rules didn't work because, apparently, you have to feed the exactly same comment:

$ iptables-save | grep A.OUTPUT
-A OUTPUT -m comment --comment "proxy-init/install-proxy-init-output/1574720803" -j PROXY_INIT_OUTPUT
$ iptables -t nat -D OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-prerouting/1574303985
iptables: No chain/target/match by that name.
$ iptables -t nat -D OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1574720803   
# success

So that'd require parsing the iptables-save output to extract those timestamps.

[grampelberg]

@alpeb yeah, there's no guarantee those are the only two rules as well. It'd be pretty easy with bash and way more complex than I'd like with how this is currently written.

[cpretzer]

Tested this locally against main and it's working as expected.

[cpretzer]

@alpeb What do you think is the right next step on this one? The comments above are correct in that this implementation is incomplete and chain rules can be left dangling or will be duplicated with different timestamps, which sounds like a different PR that will require some thought around the implementation.

[alpeb]

@cpretzer I think we can rescue all the code cleanup made in this PR, removing the rules cleanup added int ConfigureFirewall(). Then in a different PR indeed, we can tackle erroring whenever any of the linkerd rules are detected to exist already.

[cpretzer]

@alpeb this is updated with the feedback you provided

[alpeb]

Thanks @cpretzer (and @grampelberg) , this looks good to me 👍