reopen the egress policy debate :)

[jayunit100]

Per sig network - the idea of mesh like features for choke points / egress policies, seems to have gotten a second wind.

maybe we should reopen #9

@rikatz :)

[rikatz]

just clarifying , I won't write a KEP until 2025 :P

[astoycos]

/assign @astoycos

[astoycos]

This came up in the sig-network-policy API meeting on Monday May 28th 2022 (I recommend watching the recording if you were not able to make it)

I will give a better summary of the discussion here in a bit to try and get the conversation going.

[astoycos]

Originally the design for Admin Network Policy did include N/S use cases such as the one described in #9. This was removed due to some of the same issues NetworkPolicy faced with IPblock selectors (i.e The pre/post SNAT ambiguity, etc), the fact that the proposal was already extremely complicated and lastly that we believed designing for N/S use cases would be better completed in a separate/new object (something like an Egress Firewall object). Even now in the Admin Network Policy API PR we still explicitly leave the selection of N/S traffic out of the API.

However, there is some rational for re-opening this conversation while the implementation of the API is still active,

1. Implementing rules that explicitly only reference E/W traffic may be inefficient and even impossible for some CNIs
   • Our existing rational for dealing with this issue was to push forward with v1alpha1, see if many CNIs have issues implementing a functionally correct/scalable solution, and if there are many issues revisit the N/S question in v1alpha2
2. Based on the functionality of the existing networkPolicy resources (which include the ipBlock Selector), we believe it will be a bit "strange" for users to not be able to truly express similar desires at a cluster level with ANP. Specifically we forsee many users wanting to be able to completely block workloads from ALL traffic(N/S/E/W), while also allowing restricted egress access to specific destinations
3. Creation of a separate object (such as EGFW) could take quite a bit of time, leaving users unable to satisfy all their use-cases in-tree... And if 1. actually occurs we would end up having to address N/S traffic selection in later versions of the ANP API anyways

The way I see it we have a few options moving forward,

1. Do nothing, continue as planned, leaving N/S traffic selection out of the ANP API with the possibility of having to address it in later API versions
2. Embrace a total isolation model in the ANP API, meaning if a workload is affected by a general Deny rule it applies to ALL (N/S/E/W) traffic, and add an egress traffic selector allowing rules to be applied which would allow the workload to send egress traffic to specific remote destinations. (this explicitly leaves out a method of selecting ingress traffic, since in-cluster workloads are not generally directly exposed to remote sources and other mechanisms, such as the gatewayAPI, can be used to control ingress traffic to workloads. This is also inline with many of the actual customer use-cases that exist today. )
3. Leave the ANP API be for now and push hard to start the work on a new object to satisfy the additional N/S use cases

Please let me know what you all think!

[astoycos]

/assign @danwinship @thockin @caseydavenport

[danwinship]

Implementing rules that explicitly only reference E/W traffic may be inefficient and even impossible for some CNIs

This was brought up in the ANP KEP, but NetworkPolicy already requires plugins to be able to distinguish E/W from N/S traffic (namespaceSelector: {} selects all E/W traffic and no N/S traffic), so either (a) this is not a real issue, or (b) some plugins are already unable to implement NetworkPolicy correctly/efficiently.

I brought up in the meeting (or tried to bring up; I was having trouble describing it) that if this really is a problem, then one way to deal with it might be to say that plugins are allowed to get confused about hosts "near the cluster", but they aren't allowed to get confused about hosts "far away from the cluster". So, like, if your plugin allocates pod IPs from the same subnet as node IPs, then a rule saying "allow egress to all pods" might end up allowing egress to nodes too, but it could never allow egress to api.example.com). This is not ideal, but if it is a problem for you, then don't use a plugin that has this problem!

(Although that's also a bad example because one of the things I brought up in my old ClusterEgressFirewall proto-KEP that you linked to is that admins really want a way to reliably refer specifically to "all node IPs".)

[thockin]

As discussed today, I suspected we would need to reopen this, but let's get v0 committed and then iterate on this

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten