Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sysched: add the implementation of SySched #568

Merged
merged 4 commits into from
Mar 6, 2024

Conversation

salmanyam
Copy link
Contributor

What type of PR is this?

/kind feature

What this PR does / why we need it:

Implementation of SySched, a system call-aware scheduler. The KEP of this PR is KEP #399.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Thank you very much for reviewing this PR.
Please let us know if there are questions or comments.
Your comments and feedback are appreciated.

Does this PR introduce a user-facing change?

Introduce new Scoring plugin called SySched that will rank feasible nodes based on the relative risks of pods' system call usage.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Mar 30, 2023
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Mar 30, 2023

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. do-not-merge/contains-merge-commits labels Mar 30, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @salmanyam. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Mar 30, 2023
@salmanyam
Copy link
Contributor Author

@Huang-Wei This is the implementation of KEP #399, a system call-aware scheduler plugin called SySched. Could you please take a look to see if there are any concerns or comments? Thank you!

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed do-not-merge/contains-merge-commits cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Mar 30, 2023
@salmanyam salmanyam changed the title Sysched: add the implementation of SySched (KEP #399) Sysched: add the implementation of SySched Mar 31, 2023
@Huang-Wei
Copy link
Contributor

Thanks for the PR. I'm a bit swamped recently. So if any reviewers have free cycles, please assign to yourself.

/help

Copy link
Contributor

@PiotrProkop PiotrProkop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My first pass on this PR. Didn't dive deep into the logic.

Comment on lines 17 to 18
#serviceAccountName: default
#schedulerName: scheduler-plugins-scheduler
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Comment on lines 17 to 18
#serviceAccountName: default
#schedulerName: scheduler-plugins-scheduler
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Comment on lines 17 to 18
#serviceAccountName: default
#schedulerName: scheduler-plugins-scheduler
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Comment on lines 19 to 20
#serviceAccountName: default
#schedulerName: scheduler-plugins-scheduler
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Comment on lines 24 to 27
#resources:
# requests:
# cpu: 100m
# memory: 100Mi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

W := 1
score := len(syscalls) - tot_crit
score = score + W*tot_crit
klog.Info(score, " ", tot_crit)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this function will always return same score. tot_crit is always 0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, our plugin does not support syscall weights. We are working on it and update the weighting in our next version. We leave that portion of code intentionally. Also, we have put a note there regarding it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a comment explaining that weight is hard-coded for now.


nodeIP := node.Status.Addresses[0].Address

podSyscalls, _ := sc.getSyscalls(pod)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always check errors

}

func (sc *SySched) updateHostSyscalls(pod *v1.Pod) {
syscall, _ := sc.getSyscalls(pod)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always check errors

syscalls := make(map[string]bool)

for _, p := range pods {
syscall, _ := sc.getSyscalls(p)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always check errors

if p.Name == pod.Name {
sc.HostToPods[nodeIP] = remove(sc.HostToPods[nodeIP], i)
sc.HostSyscalls[nodeIP] = sc.recomputeHostSyscalls(sc.HostToPods[nodeIP])
c, _ := sc.getHostSyscalls(nodeIP)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always check errors

@Huang-Wei
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 21, 2023
@salmanyam
Copy link
Contributor Author

Thank you very much, @PiotrProkop! We have addressed all the comments. Please let us know if you have more comments or concerns.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 19, 2023
@k8s-ci-robot k8s-ci-robot removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. do-not-merge/contains-merge-commits labels May 19, 2023
@salmanyam
Copy link
Contributor Author

Hi @Huang-Wei, we have rebased our code as the bot triggered need-rebase. Please let us know if there's anything needed from our end. Thanks!

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 2, 2023

// Score invoked at the score extension point.
func (sc *SySched) Score(ctx context.Context, cs *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
node, err := sc.handle.ClientSet().CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to get cached NodeInfo from scheduler framework, which is the source of truth to do scheduling. (if reading from API server, the "cached" info would not be there)

nodeInfo, err := sc.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We noticed the cached state is not always up-to-date, especially when there are activities on other schedulers in the system or when our scheduler comes up after a bunch of pods and nodes have already appeared, hence, we opted not to use the SnapshotSharedLister.

We tried inserting some comments in the code to explain our rationale. Please let us know if this makes sense.

return 0, nil
}

nodeIP := node.Status.Addresses[0].Address
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why using a node IP instead of node.Name as the canonical identifier to refer to a Node?

type SySched struct {
handle framework.Handle
clientSet v1alpha1.SPOV1Alpha1Interface
HostToPods map[string][]*v1.Pod
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the frameworkHandle already holds this: sc.handle.SnapshotSharedLister().

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As explained above, we keep a local map of HostToPods in our plugin rather than rely on the cached state to reliably keep track of all pods on the cluster, regardless of which scheduler it used.

handle framework.Handle
clientSet v1alpha1.SPOV1Alpha1Interface
HostToPods map[string][]*v1.Pod
HostSyscalls map[string]map[string]bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add a comment explaining what's keyed, and what's valued?

Commented in another thread, in Golang we don't use map[string]bool to represent a set; instead, we use map[string]struct{}. And you even don't need to implement you self - reuse Set as well as its Untion/Intersect functions.

clientSet v1alpha1.SPOV1Alpha1Interface
HostToPods map[string][]*v1.Pod
HostSyscalls map[string]map[string]bool
CritSyscalls map[string][]string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also comment it.

}
cfg.Profiles[0].Plugins.Score = schedapi.PluginSet{
Enabled: []schedapi.Plugin{{Name: sysched.Name}},
//Disabled: []schedapi.Plugin{{Name: "*"}},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be disabling it. Otherwise score of other plugins may impact the eventual result.

Comment on lines 186 to 168
_, err = SPOCreate(extClient, &fullseccompSPOCR, metav1.CreateOptions{})
if err != nil {
t.Fatal(err)
}
_, err = SPOCreate(extClient, &badSPOCR, metav1.CreateOptions{})
if err != nil {
t.Fatal(err)
}
_, err = SPOCreate(extClient, &good1SPOCR, metav1.CreateOptions{})
if err != nil {
t.Fatal(err)
}
_, err = SPOCreate(extClient, &good2SPOCR, metav1.CreateOptions{})
if err != nil {
t.Fatal(err)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

De-duple the code by using a for loop.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you make the test more table-driven? you can refer to other unit tests like pkg/capacityscheduling/capacityscheduling_test.go

package sysched

import (
//"fmt"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove useless comments please (applies elsewhere).

Comment on lines 10 to 28
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime/schema"
informers "k8s.io/client-go/informers"
clientsetfake "k8s.io/client-go/kubernetes/fake"
"k8s.io/client-go/kubernetes/scheme"
restclient "k8s.io/client-go/rest"
restfake "k8s.io/client-go/rest/fake"
"k8s.io/kubernetes/pkg/scheduler/framework"
"k8s.io/kubernetes/pkg/scheduler/framework/plugins/defaultbinder"
"k8s.io/kubernetes/pkg/scheduler/framework/plugins/queuesort"
frameworkruntime "k8s.io/kubernetes/pkg/scheduler/framework/runtime"
st "k8s.io/kubernetes/pkg/scheduler/testing"
"net/http"
pluginconfig "sigs.k8s.io/scheduler-plugins/apis/config"
v1alpha1 "sigs.k8s.io/scheduler-plugins/pkg/sysched/clientset/v1alpha1"
"sigs.k8s.io/security-profiles-operator/api/seccompprofile/v1beta1"
"strings"
"testing"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sort the imports by groups: (applies elsewhere)

  • native imports (followed an empty line)
  • upstream imports (followed by an empty line)
  • imports in this repo

@mvle
Copy link
Contributor

mvle commented Sep 21, 2023

Thanks @Huang-Wei for taking the time to review our patch. We should be able to address all of your concerns in the next PR.

@netlify
Copy link

netlify bot commented Oct 24, 2023

Deploy Preview for kubernetes-sigs-scheduler-plugins canceled.

Name Link
🔨 Latest commit 4b6ea59
🔍 Latest deploy log https://app.netlify.com/sites/kubernetes-sigs-scheduler-plugins/deploys/65e78e81366eb800087eefb3

@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2023
@mvle
Copy link
Contributor

mvle commented Nov 2, 2023

@Huang-Wei Thanks for all your comments so far. We've updated the code to cover most of the comments, with some exceptions as explained in the comments above. Main items we have not changed:

  1. SnapshotSharedLister for getting nodeInfo and HostToPods (opted not to use for reasons stated above).
  2. custom clientset (WIP. Might need a PR to update SPO's code).

Please let us know your thoughts on the changes.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 10, 2023
@mvle
Copy link
Contributor

mvle commented Jan 17, 2024

@Huang-Wei Hi! Wondering if you'll have a chance to take a look at this PR in the coming days? Looking forward to see if we can get this closer to being merged.

@Huang-Wei
Copy link
Contributor

@mvle could you rebase to resolve the CI failure? I will take a look afterwards.

@mvle
Copy link
Contributor

mvle commented Jan 27, 2024

@Huang-Wei we've rebased. when you get a chance, please provide feedback.

@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 27, 2024
Copy link
Contributor

@Huang-Wei Huang-Wei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you eliminate the in-house clientset? We already have mature semantics to interact with any kind of CR objects.

Also, given this PR introduced a CRD, could you also update hack/verify-crdgen.sh to include the source types.go of security-profiles-operator.x-k8s.io_seccompprofiles.yaml?

# Generate CRD
api_paths="./apis/scheduling/v1alpha1/...;./vendor/github.com/k8stopologyawareschedwg/noderesourcetopology-api/pkg/apis/...;./vendor/github.com/diktyo-io/appgroup-api/pkg/apis/...;./vendor/github.com/diktyo-io/networktopology-api/pkg/apis/..."

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you follow other crds to make it symbolic linked from ../seccompprofile/crd.yaml?:

pr/568 ⇒  ll manifests/crds
total 12K
lrwxr-xr-x 1 weih   20 Apr 14  2023 appgroup.diktyo.x-k8s.io_appgroups.yaml -> ../appgroup/crd.yaml
lrwxr-xr-x 1 weih   27 Apr 14  2023 networktopology.diktyo.x-k8s.io_networktopologies.yaml -> ../networktopology/crd.yaml
lrwxr-xr-x 1 weih   30 Apr 14  2023 scheduling.x-k8s.io_elasticquotas.yaml -> ../capacityscheduling/crd.yaml
lrwxr-xr-x 1 weih   24 Apr 14  2023 scheduling.x-k8s.io_podgroups.yaml -> ../coscheduling/crd.yaml
-rw-r--r-- 1 weih 9.6K Jan 27 09:51 seccompprofiles.security-profiles-operator.x-k8s.io_sysched.yaml
lrwxr-xr-x 1 weih   32 May 25  2022 topology.node.k8s.io_noderesourcetopologies.yaml -> ../noderesourcetopology/crd.yaml

@@ -0,0 +1,322 @@
/*
Copyright 2020 The Kubernetes Authors.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2024

// Currently, this function does not change score as the weight is set to 1, and
// no critical syscalls are given as input
func (sc *SySched) calcScore(syscalls []string) int {
tot_crit := 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the comment was given on tot_crit := 0, not the function name.

RestClient rest.Interface
}

func NewForConfig(c *rest.Config) (*SPOV1Alpha1Client, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this as this should have been abstracted via generic (dynamic) clientset. You can just use the generic clientset in pkg/sysched/sysched.go w/o composing pkg/sysched/clientset at all.

Example:

scheme := runtime.NewScheme()
_ = clientscheme.AddToScheme(scheme)
_ = v1.AddToScheme(scheme)
_ = v1alpha1.AddToScheme(scheme)
client, err := client.New(handle.KubeConfig(), client.Options{Scheme: scheme})

Comment on lines 142 to 149
err := client.RestClient.
Get().
Namespace("default").
Resource("seccompprofiles").
Name(name).
VersionedParams(&opts, scheme.ParameterCodec).
Do(context.TODO()).
Into(&result)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto. We don't directly manipulate the client in this manner.

Comment on lines 76 to 78
- apiGroups: ["security-profiles-operator.x-k8s.io"]
resources: ["seccompprofiles", "profilebindings"]
verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you comment it out? for newly introduced plugin, we don't enable it until it's tested enough.

Comment on lines 114 to 116
- apiGroups: ["security-profiles-operator.x-k8s.io"]
resources: ["seccompprofiles", "profilebindings"]
verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. Comment them out.

@@ -17,7 +17,7 @@ controller:
# as they need extra RBAC privileges on metrics.k8s.io.

plugins:
enabled: ["Coscheduling","CapacityScheduling","NodeResourceTopologyMatch","NodeResourcesAllocatable"]
enabled: ["Coscheduling","CapacityScheduling","NodeResourceTopologyMatch","NodeResourcesAllocatable", "SySched"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. Don't enable it by default.

@salmanyam
Copy link
Contributor Author

Hi @Huang-Wei, we have addressed all the comments and removed the in-house clientset. Please take a look and let us know your feedback. Thanks!

@salmanyam
Copy link
Contributor Author

Hi @Huang-Wei, when you get a chance, please provide us feedback.

Copy link
Contributor

@Huang-Wei Huang-Wei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work. It's very close to merge, just a new nits.

BTW: could you squash the commits into 3:

  1. API files (apis/) - manually added by you
  2. API codegen files (apis/../zz_*.go)
  3. Other logic files - i.e., all files exclude 1,2 and 4
  4. vendor/, go.mod, go.sum

Comment on lines 35 to 38
- name: SySched
args:
defaultProfileNamespace: "default"
defaultProfileName: "full-seccomp"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment these out.

Comment on lines 18 to 20
- apiGroups: ["security-profiles-operator.x-k8s.io"]
resources: ["seccompprofiles", "profilebindings"]
verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment these out (to be consistent with default values.yaml in Helm)

Comment on lines 62 to 64
- apiGroups: ["security-profiles-operator.x-k8s.io"]
resources: ["seccompprofiles", "profilebindings"]
verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment these out (to be consistent with default values.yaml in Helm)


type SySched struct {
handle framework.Handle
//clientSet v1alpha1.SPOV1Alpha1Interface
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this line.

// NOTE: not used at this time
// Key: category of critical system call, e.g., CVE based, admin policy, etc.
// Value: list of system calls
//CritSyscalls map[string][]string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. Remove this line.

// SCMP_ACT_TRACE --> ActTrace, seccomp.ActNotify
if element.Action == seccomp.ActAllow || element.Action == seccomp.ActLog {
syscalls = syscalls.Union(sets.New[string](element.Names...))
//syscalls = unionList(syscalls, element.Names)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. Let's remove this line.

// NOTE: weight W is hardcoded for now
// TODO: add critical/cve syscalls
// TODO: adjust weight W for critical/cve syscalls
tot_crit := 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you change it to totCrit? to be Go-idiomatic.

return 0, nil
}

//diffSyscalls := setSubtract(hostSyscalls, podSyscalls)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line please.

Manually update the API files for the implementation of SySched,
a system call-aware scheduler(KEP kubernetes-sigs#399). SySched is a new scoring
plugin that ranks feasible nodes based on the relative risks of
pods' system call usage.
@salmanyam
Copy link
Contributor Author

Hi @Huang-Wei, we made the changes accordingly. Please take a look when you a chance. Thanks a lot!

@Huang-Wei
Copy link
Contributor

@salmanyam thanks!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 6, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, salmanyam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 6, 2024
@salmanyam
Copy link
Contributor Author

Thanks a lot, @Huang-Wei! We greatly appreciate your time and effort on this PR.

@k8s-ci-robot k8s-ci-robot merged commit cd3e4fb into kubernetes-sigs:master Mar 6, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants