✨ Adding DNS network policies #2423

lionelvillard · 2022-11-28T22:55:53Z

Summary

Restrict access to DNS pods to workspace associated to them. Also, make sure DNS pods have only access to CoreDNS pods.

This is enforced by create this networking policy, one per workspace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: Name
  namespace: Namespace
spec:
  podSelector:
    matchLabels:
      app: Name
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              internal.workload.kcp.io/cluster: Cluster # workspace key
      ports:
        - protocol: TCP
          port: 5353
        - protocol: UDP
          port: 5353
  egress:
    # Only give access to coredns in kube-system
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
        - podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
     # Give access to the API server to watch its associated configmap
    - to:
# one ipBlock per IP (dynamically filled)
      - ipBlock:
          cidr: APIServerIP/32
# one ports per endpoint port (dynamically filled)
      ports:
        - protocol: TCP
           port: 6443

Related issue(s)

Fix #1988

Related issue for cleaning up DNS-related resources: kcp-dev/contrib-tmc#80

openshift-ci · 2022-11-28T22:55:57Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

davidfestal · 2022-11-29T13:45:30Z

Partial fix for #1988

Could you provide more details about why it is a partial fix. What is implemented and what is not ?

pkg/cliplugins/workload/plugin/syncer.yaml

lionelvillard · 2023-01-11T18:37:14Z

not sure if it's a flake:

oroutine 26251 [running]:
        k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2c1d260?, 0x38ec8d0})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/runtime/runtime.go:75 +0x99
        k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00e49b990?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/runtime/runtime.go:49 +0x75
        panic({0x2c1d260, 0x38ec8d0})
        	/usr/local/go/src/runtime/panic.go:884 +0x212
        reflect.Value.Index({0x2b87f20?, 0xc004693a40?, 0xa?}, 0x2cf0540?)
        	/usr/local/go/src/reflect/value.go:1412 +0x16d
        k8s.io/apimachinery/pkg/runtime.sliceToUnstructured({0x2b8bfa0?, 0xc00d680898?, 0xc008306fd8?}, {0x2cf0540?, 0xc003c986d0?, 0x2dd8d60?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:763 +0x785
        k8s.io/apimachinery/pkg/runtime.toUnstructured({0x2b8bfa0?, 0xc00d680898?, 0x2?}, {0x2cf0540?, 0xc003c986d0?, 0x98?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:688 +0x6cc
        k8s.io/apimachinery/pkg/runtime.structToUnstructured({0x30226e0?, 0xc00d680780?, 0xc00da9a5a0?}, {0x2d4c880?, 0xc00822af38?, 0x2b52b20?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:843 +0x905
        k8s.io/apimachinery/pkg/runtime.toUnstructured({0x30226e0?, 0xc00d680780?, 0xc00d680780?}, {0x2d4c880?, 0xc00822af38?, 0xc00da9a570?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:692 +0x7f4
        k8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).ToUnstructured(0x54536d0, {0x31fc540?, 0xc00d680780})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:586 +0x3ba
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.toUnstructured({0x31fc540, 0xc00d680780})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_reconcile_unstructured.go:213 +0x6c
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).reconcileObject(0xc000f5d300, {0x3924818, 0xc00da9a2a0}, {0xc00e4860a5?, 0x1?}, {{0x326d8a3, 0x19}, {0x3238f30, 0x2}, {0x3249a2d, ...}}, ...)
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_reconcile.go:177 +0x565
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).reconcile(0xc000f5d300, {0x3924818, 0xc00da9a2a0}, {0xc00e486070, 0x70})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_reconcile.go:97 +0xe1a
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).processNextWorkItem(0xc000f5d300, {0x3924818, 0xc007d03320})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_controller.go:193 +0x2b1
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).startWorker(0xc00980ec00?, {0x3924818, 0xc007d03320})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_controller.go:180 +0x39
        k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:188 +0x25
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x2?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:155 +0x3e
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x38f5900, 0xc00980ec00}, 0x1, 0xc005b2d2c0)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:156 +0xb6
        k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0010147e0?, 0x3b9aca00, 0x0, 0x0?, 0xc002222000?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:133 +0x89
        k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x3924818, 0xc007d03320}, 0xc005af30e0, 0x1a7e305?, 0xc001bcce01?, 0x0?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:188 +0x99
        k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x3924818?, 0xc007d03320?}, 0xc0010147e0?, 0x0?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:99 +0x2b
        created by github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).Start
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_controller.go:174 +0x345
        panic: reflect: slice index out of range [recovered]
        	panic: reflect: slice index out of range
        
        goroutine 26251 [running]:
        k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00e49b990?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/runtime/runtime.go:56 +0xd7
        panic({0x2c1d260, 0x38ec8d0})
        	/usr/local/go/src/runtime/panic.go:884 +0x212
        reflect.Value.Index({0x2b87f20?, 0xc004693a40?, 0xa?}, 0x2cf0540?)
        	/usr/local/go/src/reflect/value.go:1412 +0x16d
        k8s.io/apimachinery/pkg/runtime.sliceToUnstructured({0x2b8bfa0?, 0xc00d680898?, 0xc008306fd8?}, {0x2cf0540?, 0xc003c986d0?, 0x2dd8d60?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:763 +0x785
        k8s.io/apimachinery/pkg/runtime.toUnstructured({0x2b8bfa0?, 0xc00d680898?, 0x2?}, {0x2cf0540?, 0xc003c986d0?, 0x98?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:688 +0x6cc
        k8s.io/apimachinery/pkg/runtime.structToUnstructured({0x30226e0?, 0xc00d680780?, 0xc00da9a5a0?}, {0x2d4c880?, 0xc00822af38?, 0x2b52b20?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:843 +0x905
        k8s.io/apimachinery/pkg/runtime.toUnstructured({0x30226e0?, 0xc00d680780?, 0xc00d680780?}, {0x2d4c880?, 0xc00822af38?, 0xc00da9a570?})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:692 +0x7f4
        k8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).ToUnstructured(0x54536d0, {0x31fc540?, 0xc00d680780})
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/runtime/converter.go:586 +0x3ba
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.toUnstructured({0x31fc540, 0xc00d680780})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_reconcile_unstructured.go:213 +0x6c
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).reconcileObject(0xc000f5d300, {0x3924818, 0xc00da9a2a0}, {0xc00e4860a5?, 0x1?}, {{0x326d8a3, 0x19}, {0x3238f30, 0x2}, {0x3249a2d, ...}}, ...)
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_reconcile.go:177 +0x565
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).reconcile(0xc000f5d300, {0x3924818, 0xc00da9a2a0}, {0xc00e486070, 0x70})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_reconcile.go:97 +0xe1a
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).processNextWorkItem(0xc000f5d300, {0x3924818, 0xc007d03320})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_controller.go:193 +0x2b1
        github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).startWorker(0xc00980ec00?, {0x3924818, 0xc007d03320})
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_controller.go:180 +0x39
        k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:188 +0x25
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x2?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:155 +0x3e
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x38f5900, 0xc00980ec00}, 0x1, 0xc005b2d2c0)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:156 +0xb6
        k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0010147e0?, 0x3b9aca00, 0x0, 0x0?, 0xc002222000?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:133 +0x89
        k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x3924818, 0xc007d03320}, 0xc005af30e0, 0x1a7e305?, 0xc001bcce01?, 0x0?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:188 +0x99
        k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x3924818?, 0xc007d03320?}, 0xc0010147e0?, 0x0?)
        	/go/pkg/mod/github.com/kcp-dev/kubernetes/staging/src/k8s.io/apimachinery@v0.0.0-20230109113100-c493866a854f/pkg/util/wait/wait.go:99 +0x2b
        created by github.com/kcp-dev/kcp/pkg/reconciler/cache/replication.(*controller).Start
        	/go/src/github.com/kcp-dev/kcp/pkg/reconciler/cache/replication/replication_controller.go:174 +0x345

lionelvillard · 2023-01-11T18:37:28Z

/test e2e
/test e2e-multiple-runs

lionelvillard · 2023-01-11T19:10:14Z

/test e2e
/test e2e-multiple-runs

lionelvillard · 2023-01-12T19:56:41Z

/hold

I'm adding e2e tests.

lionelvillard · 2023-01-13T19:01:58Z

/unhold

davidfestal · 2023-02-08T17:51:21Z

@sttts Do you want to approve ?

sttts · 2023-02-08T18:00:06Z

/approve

openshift-ci · 2023-02-08T18:00:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/cliplugins/OWNERS~~ [sttts]
~~pkg/syncer/OWNERS~~ [sttts]
~~test/OWNERS~~ [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

davidfestal · 2023-02-08T18:55:30Z

/lgtm

davidfestal · 2023-02-08T19:32:33Z

/restest

davidfestal · 2023-02-08T19:37:24Z

/retest

davidfestal · 2023-02-08T21:46:26Z

/retest

davidfestal · 2023-02-09T08:42:06Z

/lgtm

davidfestal · 2023-02-09T13:27:57Z

/lgtm

lionelvillard · 2023-02-09T15:28:51Z

/test e2e-sharded

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 28, 2022

lionelvillard marked this pull request as ready for review November 29, 2022 13:34

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 29, 2022

openshift-ci bot requested review from davidfestal and jmprusi November 29, 2022 13:34

davidfestal added the area/transparent-multi-cluster Related to scheduling of workloads into pclusters. label Dec 2, 2022

lionelvillard force-pushed the dns-network-policies branch from a642622 to 4e79820 Compare January 4, 2023 15:08

sttts reviewed Jan 5, 2023

View reviewed changes

pkg/cliplugins/workload/plugin/syncer.yaml Show resolved Hide resolved

lionelvillard force-pushed the dns-network-policies branch 4 times, most recently from a4f5b63 to 75610fc Compare January 11, 2023 17:49

lionelvillard force-pushed the dns-network-policies branch from 75610fc to e8ab723 Compare January 11, 2023 20:41

lionelvillard changed the title ~~✨ Adding DNS network policies - Part 1~~ ✨ Adding DNS network policies Jan 12, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 12, 2023

lionelvillard force-pushed the dns-network-policies branch 7 times, most recently from d52f788 to 527ee46 Compare January 13, 2023 18:32

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 13, 2023

openshift-ci bot assigned davidfestal Feb 8, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 8, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 8, 2023

lionelvillard added 12 commits February 8, 2023 16:56

Adding dns network policies - Part 1

df2c102

fix unit test

e562310

add networkpolicies in the fake pcluster

5d7672c

add e2e test

4f500f8

cluster test requires kind

8a824eb

add tenant-id label and use it in network policies

6016ada

check tenantid is set and correct (when upgrading kcp)

5d0246d

reenable test against fake cluster

b8c2617

better error message

fc683ba

create kubernetes endpoint in fake cluster.

1e47c2b

fix unit test

81870cb

changes after review

656ad4d

lionelvillard force-pushed the dns-network-policies branch from f242693 to 656ad4d Compare February 8, 2023 21:56

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Feb 8, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 9, 2023

openshift-merge-robot merged commit ee81cfe into kcp-dev:main Feb 9, 2023

kcp-ci-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Adding DNS network policies #2423

✨ Adding DNS network policies #2423

lionelvillard commented Nov 28, 2022 •

edited

openshift-ci bot commented Nov 28, 2022

davidfestal commented Nov 29, 2022

lionelvillard commented Jan 11, 2023

lionelvillard commented Jan 11, 2023

lionelvillard commented Jan 11, 2023

lionelvillard commented Jan 12, 2023

lionelvillard commented Jan 13, 2023

davidfestal commented Feb 8, 2023

sttts commented Feb 8, 2023

openshift-ci bot commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 9, 2023

davidfestal commented Feb 9, 2023

lionelvillard commented Feb 9, 2023

✨ Adding DNS network policies #2423

✨ Adding DNS network policies #2423

Conversation

lionelvillard commented Nov 28, 2022 • edited

Summary

Related issue(s)

openshift-ci bot commented Nov 28, 2022

davidfestal commented Nov 29, 2022

lionelvillard commented Jan 11, 2023

lionelvillard commented Jan 11, 2023

lionelvillard commented Jan 11, 2023

lionelvillard commented Jan 12, 2023

lionelvillard commented Jan 13, 2023

davidfestal commented Feb 8, 2023

sttts commented Feb 8, 2023

openshift-ci bot commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 8, 2023

davidfestal commented Feb 9, 2023

davidfestal commented Feb 9, 2023

lionelvillard commented Feb 9, 2023

lionelvillard commented Nov 28, 2022 •

edited