-
Notifications
You must be signed in to change notification settings - Fork 334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubernetes network policy support #48
Closed
Closed
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
ba3672c
Kubernetes client function for namespaces and policies
salv-orlando 7b541ca
Overlay mode: routines for translating policies into ACLs
salv-orlando b37e743
Add namespace and policy watchers, update pod watcher
salv-orlando d0cc701
Create types for Service and Endpoint events
salv-orlando 463fb98
Process network policies for pods
salv-orlando e85a11f
Network policy integration documentation
salv-orlando File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
apiVersion: extensions/v1beta1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: tcp80-443-foo | ||
spec: | ||
ingress: | ||
- ports: | ||
- port: "80" | ||
protocol: TCP | ||
- port: "443" | ||
protocol: TCP | ||
from: | ||
- podSelector: | ||
matchLabels: | ||
app: foo | ||
podSelector: | ||
matchLabels: | ||
app: meh |
23 changes: 23 additions & 0 deletions
23
docs/examples/networkpolicies/tcp80-443_from_foopods_all_from_barns.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
apiVersion: extensions/v1beta1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: tcp80-443-foo-bar | ||
spec: | ||
ingress: | ||
- ports: | ||
- port: "80" | ||
protocol: TCP | ||
- port: "443" | ||
protocol: TCP | ||
from: | ||
- podSelector: | ||
matchLabels: | ||
app: foo | ||
- ports: | ||
from: | ||
- namespaceSelector: | ||
matchLabels: | ||
ns: bar | ||
podSelector: | ||
matchLabels: | ||
app: meh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
apiVersion: extensions/v1beta1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: tcp80-443-meh | ||
spec: | ||
ingress: | ||
- ports: | ||
- port: "80" | ||
protocol: TCP | ||
- port: "443" | ||
protocol: TCP | ||
podSelector: | ||
matchLabels: | ||
app: meh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
apiVersion: extensions/v1beta1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: tcp80-443 | ||
spec: | ||
ingress: | ||
- ports: | ||
- port: "80" | ||
protocol: TCP | ||
- port: "443" | ||
protocol: TCP | ||
podSelector: | ||
matchLabels: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
apiVersion: extensions/v1beta1 | ||
kind: NetworkPolicy | ||
metadata: | ||
name: whitelist | ||
spec: | ||
ingress: | ||
- ports: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,155 @@ | ||
Kubernetes Network Policies Support | ||
=================================== | ||
|
||
Support is disabled by default, and should be enabled launching the watcher | ||
`--watch-policies` option. When launched with this option the watcher process | ||
will monitor Kubernetes namespaces, and for each namespace will monitor | ||
network policies. | ||
|
||
Network policy OVN integration has been tested against Kubernetes v1.4, where | ||
network policies are a beta API. The integration will not work with Kubernetes | ||
v1.3, where network policies where in alpha state and implemented via | ||
ThirdPartyResources. | ||
|
||
The watcher process will only implement network policies on those namespaces | ||
whose default isolation behavior is set to __DefaultDeny__. Any other value | ||
for the namespace isolation annotation will be ignored, and the namespace | ||
will be treated as non-isolated. | ||
|
||
The isolation annotation for an isolated namespace should have the following | ||
value: | ||
|
||
``` | ||
net.beta.kubernetes.io/network-policy: | | ||
{ | ||
"ingress": { | ||
"isolation": "DefaultDeny" | ||
} | ||
} | ||
``` | ||
|
||
It is also worth noting that: | ||
|
||
- The value of the property must be in JSON format. The Kubernetes API | ||
server expects a string and will not perform any JSON validation. | ||
The watcher processes will however only try to parse this string as | ||
JSON. Any failure in parsing will result in the namespace being | ||
considered non-isolated | ||
- As the server only expects a string, this value should be passed as such | ||
when annotating with kubectl, and quotes must be properly escaped. More | ||
information are available on the [Kubernetes docs page for network | ||
policies](http://kubernetes.io/docs/user-guide/networkpolicies/# | ||
configuring-namespace-isolation-policy) | ||
|
||
Theory of operation | ||
-------------------- | ||
|
||
This section briefly discusses how the network policy integration for OVN | ||
operates with regards to pod, namespace, and policy events. | ||
|
||
Policy workflow for pod operations | ||
----------------------------------- | ||
|
||
Network policies are enforced via OVN ACLs. The ACLs are generated and enforced | ||
before the pod is started. ACLs are generated only for network policies whose pod | ||
selector matches the one being started. A distinct ACL is generated for each | ||
rule in the network policy object. In particular: | ||
- Port and protocols in the 'ports' clause for a policy are converted into | ||
`tcp.dst` and `udp.dst` expressions in the match rule; | ||
- The IP addresses for pods matched by pod or namespace selector are place in | ||
a unique address set for the rule. The address set is then referenced in the | ||
`ip.src` expression of the match rule; | ||
- The pod's logical port is always part of the ACL match; therefore each ACL | ||
always applies to a single pod. | ||
- Every ACL has the same priority, which is anyway higher than the baseline | ||
'drop' ACL which is added for every pod created in isolated namespaces. | ||
|
||
Network Policies are implemented synchronously with creation of the pod's | ||
logical port. When the CNI plugin receives network information for a given | ||
pod, the watcher process already programmed ACLs corresponding to network | ||
policies for that pod. | ||
|
||
Upon pod deletion, every ACL configured for the pod is destroyed, and the | ||
pod's IP address removed from the policy's address sets. | ||
|
||
Pod labels affect policy rules' from clauses. Whenever a change is detected | ||
in pod labels, address sets for policies might need to be recalculated. The | ||
current logic - inefficiently - recalculates all address sets every time a | ||
change in pod labels is detected. | ||
|
||
_Note:_ The current integration does not recalculate address sets upon | ||
namespace events, which might affect policy rules' namespace selectors. | ||
This is a limitation that will be addressed in the near future. | ||
|
||
Policy workflow for namespace operations | ||
----------------------------------------- | ||
|
||
The workflow above applies - obviously - only if the namespace where the pod | ||
is being created is isolated. If the pod is instead created in a | ||
non-isolated namespace, an ACL for explicitly white listing traffic directed | ||
to that pod is created. | ||
|
||
Whenever namespace isolation is switched on or off, ACLs for every pod in | ||
the namespace are recalculated. This process is however not synchronous with | ||
the namespace operation, as that operation simply changes the value of | ||
the namespace isolation annotation on the Kubernetes API server. | ||
|
||
_Note_: Unfortunately Kubernetes (as of version 1.4) does not offer any | ||
facility to assess whether policy processing has been completed or not, and | ||
in the first case whether it completed successfully or not. | ||
|
||
|
||
Workflow for policy operations | ||
------------------------------- | ||
|
||
Network policies rules are translated in 'pseudo-ACL', which are simply | ||
in-memory data structures that contain abstract data for creating match rules | ||
for OVN ACLs. | ||
When a policy is created, its pod selector is analyzed to check for which pods | ||
the policy must be enforced. Then pseudo-ACLs for those policies are | ||
translated into actual ACLs building a match rule which also includes the OVN | ||
logical port for the pod in the match expression. | ||
|
||
When a network policy is deleted, all of its address sets are destroyed, | ||
together with the ACLs for all pods that match the policy pod selector. | ||
|
||
_Note_: As policy objects are accessed only through a namespace scope, the | ||
watcher process maintains a distinct watcher thread for every namespace. | ||
For this reason the logs will show a creation of a distinct network policy | ||
watcher for every namespace in the Kubernetes cluster. | ||
|
||
Implementation details | ||
---------------------- | ||
|
||
- The Kubernetes API only allows for retrieving network policies within the | ||
scope of a given namespace. Unlike pods or services, there is no API for | ||
retrieving policies for every namespace. For this reason a distinct watcher | ||
thread is started for each namespace. This is performed by the policy | ||
processor thread when handling the namespace ADDED event. The same policy | ||
processor also destroys the policy watcher when then namespace is deleted. | ||
- The policy processor mentioned above processes events generated from pods, | ||
namespace, and network policy instances and ensures they are translated in | ||
to the appropriate OVN ACLs. In particular, the policy processor has the | ||
following responsibilities: | ||
- ensuring traffic is white listed for pods in non-isolated namespaces | ||
- ensuring a default 'drop' ACL is added for each pod in an isolated | ||
namespace | ||
- translating Kubernetes network policies into a set of 'pseudo-ACLs', | ||
an intermediate representation for an OVN ACL, which essentially is just | ||
a tuple describing a rule's priority, the target ports, the source IPs, | ||
and the desired action (so far that would always be 'allow-related') | ||
- determining which policies apply to a pod and translating pseudo-ACLs | ||
into actual ACLs upon pod creation | ||
- removing all ACLs for a pod upon pod deletion | ||
- creating and maintaining OVN address sets for the IP addresses of pods | ||
that match the from clause of network policies rules | ||
- re-calculate ACLs upon transitions in the namespace isolation property | ||
- starting network policy watchers for every namespace, regardless of their | ||
isolation status. | ||
- When a pod gets created, the following things happen: | ||
1. For each network policy check whether its pod selector matches the pod. | ||
If it does, or if the network policy has an empty pod selector, | ||
add ingress control ACLs based on the rules in that network policy object. | ||
2. For each network policy, verify whether this pod matches any 'from' | ||
clause in the policy rules. If it does match, then this pod IP address | ||
must be added to address sets for matching rules. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a little unrelated, but I think the exceptions class does not work correctly. You likely need a call in init of OvnK8sException
super(Exception, self).init(self.message)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point. I'll check and push a separate patch in the morning