Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add customizations for kyverno.io/v1/Policy #3498

Merged
merged 1 commit into from
May 15, 2023

Conversation

yike21
Copy link
Member

@yike21 yike21 commented May 7, 2023

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
Add third-party resources kyverno.io/v1/Policy into Resource Interpreter framework.

Which issue(s) this PR fixes:
Part of #3331

Special notes for your reviewer:
@Poor12 @chaunceyjiang

Does this PR introduce a user-facing change?:
NONE

`Resource Interpreter`: Support `kyverno.io/v1/Policy`

@karmada-bot karmada-bot added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label May 7, 2023
@karmada-bot karmada-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 7, 2023
@yike21
Copy link
Member Author

yike21 commented May 7, 2023

The test report base on karmadactl interpret is below:

### Check the customizations in file
$ karmadactl interpret -f customizations.yaml --check
I0507 19:37:20.283558 3795998 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:37:20.283786 3795998 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:37:20.294560 3795998 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:37:20.294701 3795998 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
-----------------------------------
SOURCE: declarative-configuration-policy
TARGET: kyverno.io/v1 Policy   
RULERS:
    Retain:                    UNSET
    InterpretReplica:          UNSET
    ReviseReplica:             UNSET
    InterpretStatus:           PASS
    AggregateStatus:           PASS
    InterpretHealth:           PASS
    InterpretDependency:       UNSET

### Execute the statusReflection rule
$ karmadactl interpret -f customizations.yaml --operation interpretStatus --observed-file testdata/observed-helmrelease.yaml 
I0507 19:38:31.522604 3796978 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:38:31.523013 3796978 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:38:31.532197 3796978 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:38:31.532288 3796978 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/1] status:
autogen:
    rules:
        - exclude: []
          generate: []
          match:
            any:
                - resources:
                    kinds:
                        - DaemonSet
                        - Deployment
                        - Job
                        - StatefulSet
                        - ReplicaSet
                        - ReplicationController
          name: autogen-require-pod-purpose-label
          validate:
            message: You must have label `purpose` with value `production` set on all new Pod in test-policy Namespace.
            pattern:
                spec:
                    template:
                        metadata:
                            labels:
                                purpose: production
        - exclude: []
          generate: []
          match:
            any:
                - resources:
                    kinds:
                        - CronJob
          name: autogen-cronjob-require-pod-purpose-label
          validate:
            message: You must have label `purpose` with value `production` set on all new Pod in test-policy Namespace.
            pattern:
                spec:
                    jobTemplate:
                        spec:
                            template:
                                metadata:
                                    labels:
                                        purpose: production
conditions:
    - lastTransitionTime: "2023-05-07T09:19:06Z"
      message: ""
      reason: Succeeded
      status: "True"
      type: Ready
ready: true
rulecount:
    generate: 0
    mutate: 0
    validate: 1
    verifyimages: 0

### Execute the healthInterpretation rule 
$ karmadactl interpret -f customizations.yaml --operation interpretHealth --observed-file testdata/observed-policy.yaml
I0507 19:40:30.357568 3798601 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:40:30.357701 3798601 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:40:30.366363 3798601 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 19:40:30.366533 3798601 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/1] healthy:
true 

### Execute the statusAggregation rule
$ karmadactl interpret -f customizations.yaml --operation aggregateStatus --desired-file testdata/desired-policy.yaml --status-file testdata/status-file.yaml 
I0507 20:08:08.735195 3821710 deploy.go:75] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 20:08:08.735510 3821710 enable_option.go:63] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 20:08:08.753128 3821710 cmdinit.go:149] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
I0507 20:08:08.753275 3821710 register.go:160] No default release version found. build version: version.Info{GitVersion:"", GitCommit:"c8720e81c7f4badb2a2e07ed7218391b3ea7d912", GitTreeState:"dirty", BuildDate:"2023-04-12T09:43:27Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
---
# [1/1] aggregatedStatus:
apiVersion: kyverno.io/v1
kind: Policy
metadata:
    name: sample
    namespace: test-policy
spec:
    rules:
        - match:
            any:
                - resources:
                    kinds:
                        - Pod
          name: require-pod-purpose-label
          validate:
            message: You must have label `purpose` with value `production` set on all new Pod in test-policy Namespace.
            pattern:
                metadata:
                    labels:
                        purpose: production
    validationFailureAction: Enforce
status:
    autogen:
        rules:
            - match:
                any:
                    - resources:
                        kinds:
                            - DaemonSet
                            - Deployment
                            - Job
                            - StatefulSet
                            - ReplicaSet
                            - ReplicationController
              name: autogen-require-pod-purpose-label
              validate:
                message: You must have label `purpose` with value `production` set on all new Pod in test-policy Namespace.
                pattern:
                    spec:
                        template:
                            metadata:
                                labels:
                                    purpose: production
            - match:
                any:
                    - resources:
                        kinds:
                            - CronJob
              name: autogen-cronjob-require-pod-purpose-label
              validate:
                message: You must have label `purpose` with value `production` set on all new Pod in test-policy Namespace.
                pattern:
                    spec:
                        jobTemplate:
                            spec:
                                template:
                                    metadata:
                                        labels:
                                            purpose: production
    conditions:
        - lastTransitionTime: "2023-05-07T09:19:06Z"
          message: member2=, member3=
          reason: Succeeded
          status: "True"
          type: Ready
    ready: true
    rulecount:
        generate: 0
        mutate: 0
        validate: 2
        verifyimages: 0

@codecov-commenter
Copy link

codecov-commenter commented May 7, 2023

Codecov Report

Merging #3498 (c1e97d4) into master (99f3c4f) will increase coverage by 0.00%.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@           Coverage Diff           @@
##           master    #3498   +/-   ##
=======================================
  Coverage   52.62%   52.62%           
=======================================
  Files         213      213           
  Lines       19600    19600           
=======================================
+ Hits        10314    10315    +1     
+ Misses       8734     8733    -1     
  Partials      552      552           
Flag Coverage Δ
unittests 52.62% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

@Poor12
Copy link
Member

Poor12 commented May 8, 2023

It seems that #3390 still happens.
/cc @XiShanYongYe-Chang for check.

@XiShanYongYe-Chang
Copy link
Member

It seems that #3390 still happens. /cc @XiShanYongYe-Chang for check.

Does the current pr contain related fix commit?

@chaunceyjiang
Copy link
Member

Does the current pr contain related fix commit?

https://github.com/yike21/karmada/blob/bundle-kyverno-policy/pkg/scheduler/event_handler.go#L191

Yes.

@chaunceyjiang
Copy link
Member


 1 scheduler.go:485] Schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(0)
 1 generic_scheduler.go:123] Cluster "member1" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 generic_scheduler.go:123] Cluster "member2" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 generic_scheduler.go:123] Cluster "member3" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 scheduler.go:498] failed to schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(0): 0/3 clusters are available: 3 cluster(s) didn't match the placement cluster affinity constraint.
 1 scheduler.go:485] Schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(1)
 1 generic_scheduler.go:123] Cluster "member1" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 generic_scheduler.go:123] Cluster "member2" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 generic_scheduler.go:123] Cluster "member3" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 scheduler.go:498] failed to schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(1): 0/3 clusters are available: 3 cluster(s) didn't match the placement cluster affinity constraint.
 1 scheduler.go:485] Schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(2)
 1 generic_scheduler.go:123] Cluster "member1" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 generic_scheduler.go:123] Cluster "member2" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
1 generic_scheduler.go:123] Cluster "member3" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 scheduler.go:498] failed to schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(2): 0/3 clusters are available: 3 cluster(s) didn't match the placement cluster affinity constraint.
 1 scheduler.go:485] Schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(3)
 1 generic_scheduler.go:123] Cluster "member2" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 generic_scheduler.go:123] Cluster "member3" is not fit, reason: cluster(s) didn't match the placement cluster affinity constraint
 1 generic_scheduler.go:78] Feasible clusters found: [member1]
 1 generic_scheduler.go:149] Plugin ClusterAffinity scores on karmadatest-pzbhc/deploy-x5b4l => [{member1 0}]
 1 generic_scheduler.go:149] Plugin ClusterLocality scores on karmadatest-pzbhc/deploy-x5b4l => [{member1 0}]
 1 generic_scheduler.go:84] Feasible clusters scores: [{member1 0}]


@chaunceyjiang
Copy link
Member

2023-05-08T02:27:31.985571585Z stderr F I0508 02:27:31.985505       1 generic_scheduler.go:84] Feasible clusters scores: [{member1 0}]
2023-05-08T02:27:31.997886527Z stderr F I0508 02:27:31.997697       1 event.go:294] "Event occurred" object="karmadatest-pzbhc/deploy-x5b4l-deployment" fieldPath="" kind="ResourceBinding" apiVersion="work.karmada.io/v1alpha2" type="Warning" reason="ScheduleBindingFailed" message="failed to schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(0): 0/3 clusters are available: 3 cluster(s) didn't match the placement cluster affinity constraint."
2023-05-08T02:27:31.998234828Z stderr F I0508 02:27:31.998030       1 event.go:294] "Event occurred" object="karmadatest-pzbhc/deploy-x5b4l" fieldPath="" kind="Deployment" apiVersion="apps/v1" type="Warning" reason="ScheduleBindingFailed" message="failed to schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(0): 0/3 clusters are available: 3 cluster(s) didn't match the placement cluster affinity constraint."
2023-05-08T02:27:31.999071931Z stderr F I0508 02:27:31.998980       1 event.go:294] "Event occurred" object="karmadatest-pzbhc/deploy-x5b4l-deployment" fieldPath="" kind="ResourceBinding" apiVersion="work.karmada.io/v1alpha2" type="Warning" reason="ScheduleBindingFailed" message="failed to schedule ResourceBinding(karmadatest-pzbhc/deploy-x5b4l-deployment) with clusterAffiliates index(1): 0/3 clusters are available: 3 cluster(s) didn't match the placement cluster affinity constraint."
``

@chaunceyjiang
Copy link
Member

chaunceyjiang commented May 8, 2023

	ClusterAffinities: []policyv1alpha1.ClusterAffinityTerm{
		{
			AffinityName:    "group1",
			ClusterAffinity: policyv1alpha1.ClusterAffinity{LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{member1LabelKey: "ok"}}},
		},
		{
			AffinityName:    "group2",
			ClusterAffinity: policyv1alpha1.ClusterAffinity{LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{member2LabelKey: "ok"}}},
		},
		{
			AffinityName:    "group3",
			ClusterAffinity: policyv1alpha1.ClusterAffinity{LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{"no-exist-cluster": "ok"}}},
		},
		{
			AffinityName:    "group4",
			ClusterAffinity: policyv1alpha1.ClusterAffinity{ClusterNames: []string{"member1"}},
		},
	}})



	ginkgo.It("propagate deployment and then update the cluster label", func() {
		// 1. wait for deployment present on member1 cluster
		framework.WaitDeploymentPresentOnClusterFitWith("member1", deployment.Namespace, deployment.Name, func(deployment *appsv1.Deployment) bool { return true })

		// 2. update member1 cluster label to make it's unmatched with the policy
		framework.UpdateClusterLabels(karmadaClient, "member1", map[string]string{member1LabelKey: "not-ok"})

		framework.WaitDeploymentDisappearOnCluster("member1", deployment.Namespace, deployment.Name)

		// 3. wait for deployment present on member2 cluster
		framework.WaitDeploymentPresentOnClusterFitWith("member2", deployment.Namespace, deployment.Name, func(deployment *appsv1.Deployment) bool { return true })



		// 4. update member2 cluster label to make it's unmatched with the policy
		framework.UpdateClusterLabels(karmadaClient, "member2", map[string]string{member2LabelKey: "not-ok"})
		framework.WaitDeploymentDisappearOnCluster("member2", deployment.Namespace, deployment.Name)

		// 5. wait for deployment present on member1 cluster
		framework.WaitDeploymentPresentOnClusterFitWith("member1", deployment.Namespace, deployment.Name, func(deployment *appsv1.Deployment) bool { return true })
	})

After step 2 is executed, before step 4 is executed, AffinityName has already become group4.

/cc @XiShanYongYe-Chang

Signed-off-by: yike21 <yike21@qq.com>
@Poor12
Copy link
Member

Poor12 commented May 12, 2023

/lgtm
Hold a while for #3492

@karmada-bot karmada-bot added the lgtm Indicates that a PR is ready to be merged. label May 12, 2023
@chaunceyjiang
Copy link
Member

/approve

Thanks!

@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chaunceyjiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@karmada-bot karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 15, 2023
@karmada-bot karmada-bot merged commit d6470bc into karmada-io:master May 15, 2023
@yike21 yike21 deleted the bundle-kyverno-policy branch May 15, 2023 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants