Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add and enable two egressAllowRules to ensure DNS access #3179

Merged
merged 2 commits into from
Aug 2, 2023
Merged

Add and enable two egressAllowRules to ensure DNS access #3179

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
53f61c9
Add and enable two egressAllowRules to ensure DNS access
consideRatio Aug 1, 2023
97e7acf
docs: add/update versionadded notes
consideRatio Aug 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
29 changes: 22 additions & 7 deletions jupyterhub/templates/_helpers-netpol.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,20 +30,35 @@
{{- define "jupyterhub.networkPolicy.renderEgressRules" -}}
{{- $root := index . 0 }}
{{- $netpol := index . 1 }}
{{- if $netpol.egressAllowRules.dnsPortsPrivateIPs }}
# Allow outbound connections to the DNS port in the private IP ranges
{{- if or (or $netpol.egressAllowRules.dnsPortsCloudMetadataServer $netpol.egressAllowRules.dnsPortsKubeSystemNamespace) $netpol.egressAllowRules.dnsPortsPrivateIPs }}
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- port: 53
protocol: UDP
- port: 53
protocol: TCP
to:
{{- if $netpol.egressAllowRules.dnsPortsCloudMetadataServer }}
# Allow outbound connections to DNS ports on the cloud metadata server
- ipBlock:
cidr: {{ $root.Values.singleuser.cloudMetadata.ip }}/32
{{- end }}
{{- if $netpol.egressAllowRules.dnsPortsKubeSystemNamespace }}
# Allow outbound connections to DNS ports on pods in the kube-system
# namespace
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
{{- end }}
{{- if $netpol.egressAllowRules.dnsPortsPrivateIPs }}
# Allow outbound connections to DNS ports on destinations in the private IP
# ranges
- ipBlock:
cidr: 10.0.0.0/8
- ipBlock:
cidr: 172.16.0.0/12
- ipBlock:
cidr: 192.168.0.0/16
{{- end }}
{{- end }}

{{- if $netpol.egressAllowRules.nonPrivateIPs }}
Expand All @@ -53,7 +68,7 @@
cidr: 0.0.0.0/0
except:
# As part of this rule, don't:
# - allow outbound connections to private IP
# - allow outbound connections to private IPs
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
Expand Down
78 changes: 73 additions & 5 deletions jupyterhub/values.schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -651,7 +651,6 @@ properties:
```

```{versionadded} 2.0.0
All `egressAllowRules` are new in JupyterHub Helm chart 2.0.0.
```
properties:
cloudMetadataServer:
Expand All @@ -667,6 +666,55 @@ properties:
IP ranges but makes an exception for the cloud metadata
server, leaving this as the definitive configuration to allow
access to the cloud metadata server.
dnsPortsCloudMetadataServer:
type: boolean
description: |
Defaults to `true` for all network policies.

When enabled this rule allows the respective pod(s) to
establish outbound connections to the cloud metadata server
via port 53.

Relying on this rule should go hand in hand with disabling
[`singleuser.cloudMetadata.blockWithIptables`](schema_singleuser.cloudMetadata.blockWithIptables).

Known situations when this rule can be relevant:

- In GKE clusters with Cloud DNS that is reached at the
cloud metadata server's non-private IP.

```{note}
This chart doesn't know how to identify the DNS server that
pods will rely on due to variations between how k8s clusters
have been setup. Due to that, multiple rules are enabled by
default to ensure DNS connectivity.
```

```{versionadded} 3.0.0
```
dnsPortsKubeSystemNamespace:
type: boolean
description: |
Defaults to `true` for all network policies.

When enabled this rule allows the respective pod(s) to
establish outbound connections to pods in the kube-system
namespace via port 53.

Known situations when this rule can be relevant:

- GKE, EKS, AKS, and other clusters relying directly on
`kube-dns` or `coredns` pods in the `kube-system` namespace.

```{note}
This chart doesn't know how to identify the DNS server that
pods will rely on due to variations between how k8s clusters
have been setup. Due to that, multiple rules are enabled by
default to ensure DNS connectivity.
```

```{versionadded} 3.0.0
```
dnsPortsPrivateIPs:
type: boolean
description: |
Expand All @@ -675,10 +723,23 @@ properties:
When enabled this rule allows the respective pod(s) to
establish outbound connections to private IPs via port 53.

Note that we can't reliably identify the k8s internal DNS
server due to variations between k8s clusters. Due to that,
this rule which is critical for core functionality, can be
disabled for a more refined custom rule.
Known situations when this rule can be relevant:

- GKE clusters relying on a DNS server indirectly via a a node
local DNS cache at an unknown private IP.

```{note}
This chart doesn't know how to identify the DNS server that
pods will rely on due to variations between how k8s clusters
have been setup. Due to that, multiple rules are enabled by
default to ensure DNS connectivity.

```{warning}
This rule is not expected to work in clusters relying on
Cilium to enforce the NetworkPolicy rules (includes GKE
clusters with Dataplane v2), this is due to a [known
limitation](https://github.com/cilium/cilium/issues/9209).
```
nonPrivateIPs:
type: boolean
description: |
Expand Down Expand Up @@ -713,6 +774,13 @@ properties:
If possible, try to avoid setting this to true as it gives
broad permissions that could be specified more directly via
the [`.egress`](schema_singleuser.networkPolicy.egress).

```{warning}
This rule is not expected to work in clusters relying on
Cilium to enforce the NetworkPolicy rules (includes GKE
clusters with Dataplane v2), this is due to a [known
limitation](https://github.com/cilium/cilium/issues/9209).
```
interNamespaceAccessLabels:
enum: [accept, ignore]
description: |
Expand Down
8 changes: 8 additions & 0 deletions jupyterhub/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ hub:
egress: []
egressAllowRules:
cloudMetadataServer: true
dnsPortsCloudMetadataServer: true
dnsPortsKubeSystemNamespace: true
dnsPortsPrivateIPs: true
nonPrivateIPs: true
privateIPs: true
Expand Down Expand Up @@ -231,6 +233,8 @@ proxy:
egress: []
egressAllowRules:
cloudMetadataServer: true
dnsPortsCloudMetadataServer: true
dnsPortsKubeSystemNamespace: true
dnsPortsPrivateIPs: true
nonPrivateIPs: true
privateIPs: true
Expand Down Expand Up @@ -278,6 +282,8 @@ proxy:
egress: []
egressAllowRules:
cloudMetadataServer: true
dnsPortsCloudMetadataServer: true
dnsPortsKubeSystemNamespace: true
dnsPortsPrivateIPs: true
nonPrivateIPs: true
privateIPs: true
Expand Down Expand Up @@ -354,6 +360,8 @@ singleuser:
egress: []
egressAllowRules:
cloudMetadataServer: false
dnsPortsCloudMetadataServer: true
dnsPortsKubeSystemNamespace: true
dnsPortsPrivateIPs: true
nonPrivateIPs: true
privateIPs: false
Expand Down
2 changes: 2 additions & 0 deletions tools/templates/lint-and-validate-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,8 @@ hub:
enabled: true
egressAllowRules:
cloudMetadataServer: true
dnsPortsCloudMetadataServer: true
dnsPortsKubeSystemNamespace: true
dnsPortsPrivateIPs: true
nonPrivateIPs: true
privateIPs: false
Expand Down