-
-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Network policies break daskhub #445
Comments
Hi, apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dask-network-policy
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
- from:
- podSelector:
matchLabels:
app: jupyterhub
egress:
- ports:
- port: 6443
- port: 53
protocol: UDP
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
- to:
- podSelector:
matchLabels:
app: jupyterhub
|
@Ph0tonic Thank you for confirming and providing a more restrictive Mar 15 12:00:07.234: jhub01/hub-6cdf59cc94-x4zmf:54686 (ID:154794) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-fghxd:8000 (ID:135336) Policy denied DROPPED (TCP Flags: SYN)
Mar 15 12:00:08.278: jhub01/hub-6cdf59cc94-x4zmf:54686 (ID:154794) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-fghxd:8000 (ID:135336) Policy denied DROPPED (TCP Flags: SYN) If I allow egress port 8000 (via the 6443 workaround) for the hub pod then I receive blocks from the proxy pod to the traefik pod: Mar 15 13:11:07.854: jhub01/proxy-84cd6496dc-64jvb:47066 (ID:138248) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-cqkbg:8000 (ID:135336) Policy denied DROPPED (TCP Flags: SYN)
Mar 15 13:11:08.886: jhub01/proxy-84cd6496dc-64jvb:47066 (ID:138248) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-cqkbg:8000 (ID:135336) Policy denied DROPPED (TCP Flags: SYN) If I modify the proxy networkpolicy and allow egress for port 8000 then I receive blocks from the singleuser pod to the traefik pod: Mar 15 13:14:07.052: jhub01/jupyter-test:49324 (ID:132715) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-cqkbg:8000 (ID:135336) Policy denied DROPPED (TCP Flags: SYN)
Mar 15 13:14:08.109: jhub01/jupyter-test:49324 (ID:132715) <> jhub01/traefik-jhub01-dask-gateway-7665b69c66-cqkbg:8000 (ID:135336) Policy denied DROPPED (TCP Flags: SYN) I then modified the singleuser networkpolicy to allow egress for port 8000 and everything works. Since this chart is designed to deploy two sub-charts, I think it should address the networkpolicies to allow correct communications between the two deployments. The root issue is with the jupyterhub egress networkpolicies. I tried to address this with a helm values file: jupyterhub:
hub:
networkPolicy:
egress:
- ports:
- port: 6443
- port: 8000
chp:
networkPolicy:
egress:
- ports:
- port: 8000
singleuser:
networkPolicy:
egress:
- ports:
- port: 8000 However, this chart doesn't understand the |
Ho, sorry, indeed I only mentioned the config for dask-gateway, here is my config for Jupyterhub Helm Chart: values:
proxy:
chp:
networkPolicy:
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
ports:
- port: 8000
singleuser:
networkPolicy:
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
ports:
- port: 8000
hub:
networkPolicy:
egress:
- ports:
- port: 6443
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
ports:
- port: 8000 And the additional config as previously provided: apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: dask-network-policy
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
- from:
- podSelector:
matchLabels:
app: jupyterhub
egress:
- ports:
- port: 6443
- port: 53
protocol: UDP
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
- to:
- podSelector:
matchLabels:
app: jupyterhub However, I agree that it would be nice to have those config as default. |
@Ph0tonic Thank you! With the values you provided I have managed to make dask-gateway function with jupyterhub. For others that may come upon this issue, these values work for the daskhub chart: jupyterhub:
proxy:
chp:
networkPolicy:
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
ports:
- port: 8000
singleuser:
networkPolicy:
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
ports:
- port: 8000
hub:
networkPolicy:
egress:
- ports:
- port: 6443
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: dask-gateway
ports:
- port: 8000 I do think these values should be defined by default when deploying the daskhub chart so dask-gateway works with jupyterhub. |
Describe the issue:
It appears the default network policies from the jupyterhub helm chart breaks communication with dask-gateway and the kube-apiserver.
I deployed daskhub with default values onto a vanilla K8s cluster with a CNI that supports network policies (cilium).
helm upgrade --install --create-namespace --namespace jhub01 jhub01 dask/daskhub
With this deployment, the jupyterhub pod will not spawn and I receive this output:
![image](https://private-user-images.githubusercontent.com/43788363/305511058-4a484716-b02f-4f58-a7d5-293f1922fcac.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyMTc4MjcsIm5iZiI6MTcyNTIxNzUyNywicGF0aCI6Ii80Mzc4ODM2My8zMDU1MTEwNTgtNGE0ODQ3MTYtYjAyZi00ZjU4LWE3ZDUtMjkzZjE5MjJmY2FjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDE5MDUyN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRlNzYxY2U0MmExMTIxMGQxYmYxZDg1MGM3NDc3NTY2MGJlYmE2MjM5YTVkYWJhNjc2YWI3MDEzODFhMzI2NWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.vbdsgbYvGq0x7ejfqb9x2TLvDi1BMGgve7F0dcKTUZU)
Using hubble, I am able to see the packets are being dropped via network policy:
hubble observe -n jhub01 -t drop -f
If I allow access to the kube-apiserver (reference ticket below), the pod will then spawn but I still get drops for dask-gateway communication:
Here is the list of network policies defined for the whole cluster:
kubectl get networkpolicies.networking.k8s.io -A
Everything works when I deploy this network policy into the namespace:
Anything else we need to know?:
Bare Metal - K8s Server Version: v1.29.1
CRI-O Version: v1.29.1
Cilium Version: v1.15.1
This issue is related and is why we are seeing drops for the kube-apiserver:
hub
pod unable to establish connections to k8s api-server etc on port 6443 with Cilium jupyterhub/zero-to-jupyterhub-k8s#3202If I allow access to the kube-apiserver I then hit this issue:
I think that the daskhub chart should deploy network policies to allow the jupyterhub pod to communicate with dask-gateway. Or perhaps something about the correct network policies should be documented since the default values don't allow dask-gateway communication?
Environment:
The text was updated successfully, but these errors were encountered: