Unable to detect if pod passed network policies

[SerialVelocity]

What happened?
Before 1.0.0-rc4, network traffic was "accepted" if network policies were matched. Now, they just return. This makes it very difficult to differentiate between whether the network policies were matched and passed or were unmatched.

What did you expect to happen?
Traffic to be accepted if it made it through the network policy or for a way to detect that it matched a network policy

How can we reproduce the behavior you experienced?
Steps to reproduce the behavior:

1. Set iptables forward policy to drop
2. Try and figure out how to unmatched traffic (such as host traffic and traffic that didn't match a network policy)

**Screenshots / Architecture Diagrams / Network Topologies **
None

** System Information (please complete the following information):**

• Kube-Router Version (kube-router --version): 1.0.1
• Kube-Router Parameters:

    - --hostname-override=172.28.0.3
    - --kubeconfig=/etc/kubernetes/kube-proxy-kubeconfig
    - --run-router=false
    - --run-firewall=true
    - --run-service-proxy=true
    - --service-cluster-ip-range=172.30.128.0/17
    - --metrics-port=20249

• Kubernetes Version (kubectl version) : 1.18.6
• Cloud Type: On premise
• Kubernetes Deployment Type: Custom
• Kube-Router Deployment Type: static pod
• Cluster Size: 9 nodes

** Logs, other output, metrics **
None

Additional context

It would be nice if there was a flag to either go back to the old behaviour, have a post-processing table name, or a way to set a mark if the network policies were matched.

[aauren]

@SerialVelocity this was done on purpose in this PR #915 to resolve this issue #916 where if the source and destination were on the same node, network policy would not work according to the upstream spec as traffic would be accepted as soon as it conformed one of the policies regardless of whether it conformed to all policies.

Because of this, we replaced the ACCEPT policy with the RETURN policy so that the packet would continue to traverse all applicable chains before being accepted. Since this is a matter of conforming to the intent of the upstream network policy spec, it is something that we are unlikely to change.

However, we do put a logging statement at the end of every chain before we drop or reject a packet (see: https://github.com/cloudnativelabs/kube-router/blob/master/pkg/controllers/netpol/network_policy_controller.go#L835). So you should be able to get some observability by setting up ulogd on your system and then logging these packets. There are some brief instructions on how to do that here: https://github.com/cloudnativelabs/kube-router/blob/master/docs/Observability.md

[SerialVelocity]

Hi @aauren, thanks for the quick reply.

I'm fine with it being a RETURN statement, I just want to detect whether the packet matched a policy or not. By default, my hosts are locked down but before I could get around this by adding a network policy to kube-router which would "accept" the traffic. Now, that is extremely hard to do since I can't tell if a pod matched a policy or not. I guess what I'm asking for is a way to set a mark if a network policy was matched. (like make every policy set 0x3000 but only check and clear 0x1000 so there is now a 0x2000 mark left meaning a policy was matched)

[aauren]

This might be possible. We'd have to be careful how we implement it so that we don't end up having to mark twice for every packet we want to accept which would essentially double the amount of rules that we add which would have performance issues.

I feel like I see some potential crossover between this and #692 which we are probably going to need to handle in 1.2. In #692 we have the problem of our iptables rules being too lenient where all packets (at least in the FORWARD table) are ACCEPTed after they pass through the policy chains. However, the iptables rule change in #695 probably is probably too locked down in only accepting related or established flows. It's possible that adding a mark that is not removed may be helpful in allowing this ACCEPT policy to be more intelligent about the packets it's accepting.

Given the current state where all packets in the FORWARD table are accepted if they make it through the policy chains, maybe you can add some more information here about what flows we're missing in your situation and why you need additional ACCEPT statements?

[SerialVelocity]

Packets in the FORWARD chain are only accepted if the policy is set to ACCEPT. This can be insecure depending on what else is running on the hosts.

e.g.

• If the policy is ACCEPT you might accidentally expose too much if you forget to put iptables rules in place.
• If the policy is REJECT you won't get traffic if you forget the iptables rules.

For that reason, I've set the policy for my INPUT/FORWARD chains to REJECT.

[aauren]

Can you run the following command on one of your hosts: iptables -nvL FORWARD and post the output to this thread?

[SerialVelocity]

$ sudo iptables -nvL FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 127M  309G KUBE-ROUTER-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-router netpol */
 100M  250G DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0

[aauren]

If I had to guess, it seems like you're not running kube-router with the --run-router option enabled? Which is fine, but then kube-router takes a hands off view of routing traffic to your pods. I think that what would be helpful, is if you went back and updated your issue description with the full details from our bug template: https://github.com/cloudnativelabs/kube-router/blob/master/.github/ISSUE_TEMPLATE/bug_report.md

This gives us a much better understanding of the context that our users are coming from. Once you do that, please ping back on the issue and I'll take a look again.

[SerialVelocity]

@aauren Done!

[aauren]

Thanks @SerialVelocity for adding that information.

I agree that there is an issue here. Now that we've changed the default to be RETURN there are no rules to explicitly ACCEPT the traffic that makes it through all of the network policies when kube-router is not run with --run-router enabled. Additionally, since Docker is run on the host, unless people change Docker to run with --iptables=false which I suspect many people don't, then they have a default policy of DROP on their FORWARD chain. This combination would be very problematic.

@murali-reddy I suspect that there is at least a bit of intersection between this issue and #692 but I think that this is a use-case that should be covered. Likely it will involve marking of some sort which would also give you what you hope to accomplish, however, I think that user's shouldn't have to create that rule for themselves. kube-router should be responsible for accepting the packet if it makes it through all of the firewall rules.