Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Model Registry networkpolicy #2724

Merged

Conversation

tarilabs
Copy link
Member

Raising as draft following indication on #2701 but not yet sure how to manually verify: I've tested with 1.9.0-rc.0 on a local cluster, both with and without this network policy, I can reach the service both externally the cluster (authenticated call) and from another pod in a different namespace (namespace "default" using wget from an hello-world pod).

Which issue is resolved by this Pull Request:
Resolves #2701

Description of your changes:

Checklist:

  • Unit tests pass:
    Make sure you have installed kustomize == 5.2.1+
    1. make generate-changed-only
    2. make test

Signed-off-by: Matteo Mortari <matteo.mortari@gmail.com>
@dhirajsb
Copy link

lgtm, but need to test it manually in a deployment

@juliusvonkohout
Copy link
Member

@lampajr please add explicit port numbers for the service.

@lampajr
Copy link
Member

lampajr commented May 22, 2024

@lampajr please add explicit port numbers for the service.

Hi @juliusvonkohout, given that we should make accessible all ports that are exposed by the model registry pod (@tarilabs can you confirm that?) I think that it is acceptable as it is (without any port specification, similarly to what other components are already doing)

@tarilabs
Copy link
Member Author

I would rather have this pov: this networkpolicy proposal aligns with other examples as suggested by action item out of previous meeting (notes).

In short we need 2 "endpoints": REST API (8080) and gRPC (9090).

Could someone kindly comment regardings how to A/B test it, per original message, please?
That is not clear to me, and sorry if asking a banal question

@juliusvonkohout
Copy link
Member

juliusvonkohout commented May 22, 2024

I would rather have this pov: this networkpolicy proposal aligns with other examples as suggested by action item out of previous meeting (notes).

In short we need 2 "endpoints": REST API (8080) and gRPC (9090).

Could someone kindly comment regardings how to A/B test it, per original message, please?
That is not clear to me, and sorry if asking a banal question

It is rather trivial and best practices to list those two ports in the networkpolicy. A networkpolicy is just a firewall rule. If you install a full Kubeflow and can still reach both ports it is fine. For standalone tests I would add a deny all networkpolicy and this policy here on top to test.

@tarilabs
Copy link
Member Author

I've added 31345c9
but I need to test it on a vanilla KF cluster, hence keeping this as draft for now.
Would this align better to expectations, please?

@tarilabs tarilabs force-pushed the tarilabs-20250520-networkpolicy branch from 31345c9 to b06906f Compare May 23, 2024 11:24
Signed-off-by: tarilabs <matteo.mortari@gmail.com>
@tarilabs tarilabs force-pushed the tarilabs-20250520-networkpolicy branch from b06906f to 5778fca Compare May 23, 2024 11:25
@juliusvonkohout
Copy link
Member

juliusvonkohout commented May 23, 2024

Next to my 2 comments I would like to add that gpt4 can write you such policies. Just input your service and deployments and it will figure it out.

Signed-off-by: Matteo Mortari <matteo.mortari@gmail.com>
@tarilabs
Copy link
Member Author

tarilabs commented Jun 3, 2024

Tested with:

diff --git a/common/networkpolicies/base/kustomization.yaml b/common/networkpolicies/base/kustomization.yaml
index 3592bc9a..33bf626c 100644
--- a/common/networkpolicies/base/kustomization.yaml
+++ b/common/networkpolicies/base/kustomization.yaml
@@ -16,6 +16,7 @@ resources:
   - minio.yaml
   - ml-pipeline-ui.yaml
   - ml-pipeline.yaml
+  - model-registry.yaml
   - poddefaults.yaml
   - pvcviewer-webhook.yaml
   - seldon.yaml
diff --git a/common/networkpolicies/base/model-registry.yaml b/common/networkpolicies/base/model-registry.yaml
new file mode 100644
index 00000000..801a2145
--- /dev/null
+++ b/common/networkpolicies/base/model-registry.yaml
@@ -0,0 +1,33 @@
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: model-registry
+  namespace: kubeflow
+spec:
+  podSelector:
+    matchExpressions:
+      - key: component
+        operator: In
+        values:
+          - model-registry-server
+  ingress:
+    - from:
+        - namespaceSelector:
+            matchExpressions:
+              - key: app.kubernetes.io/part-of
+                operator: In
+                values:
+                  - kubeflow-profile
+        - namespaceSelector:
+            matchExpressions:
+              - key: kubernetes.io/metadata.name
+                operator: In
+                values:
+                  - istio-system
+      ports:
+        - protocol: TCP
+          port: 8080
+        - protocol: TCP
+          port: 9090
+  policyTypes:
+    - Ingress

per latest reviews.

Without network policy:

Screenshot from 2024-06-03 14-58-26

WITH network policy:

Screenshot from 2024-06-03 15-02-20

Notes on testing environment used

Since I've tested locally on Minikube, I needed to ensure "calico".
For example:

minikube start --cni calico --memory 8192 --cpus 6 # ...

Caution

the --cni calico need be supplied at the FIRST start, it cannot be applied to an already-existing minikube

outputs as (on my linux box)

$ minikube start --listen-address=0.0.0.0 --cni calico --memory 8192 --cpus 6
😄  minikube v1.33.1 on Fedora 40
✨  Automatically selected the docker driver. Other choices: qemu2, ssh
📌  Using Docker driver with root privileges

...

🔗  Configuring Calico (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
...

(notice the calico line explicitly output)

Additional notes

This might actually be analogous as experienced by end-user with:

@juliusvonkohout
Copy link
Member

/approve

@juliusvonkohout
Copy link
Member

/lgtm

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot added the lgtm label Jun 3, 2024
@google-oss-prow google-oss-prow bot merged commit 0805068 into kubeflow:master Jun 3, 2024
4 checks passed
@tarilabs
Copy link
Member Author

tarilabs commented Jun 3, 2024

Thank you @juliusvonkohout 🙏 🚀

@tiansiyuan
Copy link
Contributor

@tarilabs I tested it on Microk8s 1.28. It works.

juliusvonkohout pushed a commit that referenced this pull request Jun 4, 2024
* add Model Registry networkpolicy

Signed-off-by: Matteo Mortari <matteo.mortari@gmail.com>

* implement review feedback

Signed-off-by: tarilabs <matteo.mortari@gmail.com>

* implement code review feedback

Signed-off-by: Matteo Mortari <matteo.mortari@gmail.com>

---------

Signed-off-by: Matteo Mortari <matteo.mortari@gmail.com>
Signed-off-by: tarilabs <matteo.mortari@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Kubeflow Model Registry Network Policies
5 participants