CRI: expose optional runtime features

@thockin

Meta-issue for CRI: #28789

For some features, kubelet checks if the underlying runtime supports such features and reject the pod if necessary. One example is the sysctl support. Right now, kubelet simply checks if (1) the runtime is docker, and (2) if docker version is higher than the minimum requirement. With CRI, checking runtime versions directly is no longer viable. One way to solve this is to ask runtime to expose a set of features it supports through CRI. E.g., an Info call can be added to CRI to retrieve a list of supported features.

Thoughts? Objections?

/cc @kubernetes/sig-node @thockin

Note: sysctl is still an experimental feature and relies on pod annotations in the k8s API. However, the pod admission logic is already live in kubelet.

@yujuhong

@yujuhong Why checking runtime versions is not viable?

@yujuhong

+1 for this.

@yujuhong Why checking runtime versions is not viable?

@yifan-gu Do you mean extend Version() API to include features? I think that's better than adding a new API.

@yujuhong

Do we need this as an explicit API? Why not just pass sysctl to the
runtime, and if it doesn't support them, it can either reject the request
or log-and-run-anyway...

On Thu, Sep 15, 2016 at 8:34 PM, Pengfei Ni notifications@github.com
wrote:

+1 for this.

@yujuhong https://github.com/yujuhong Why checking runtime versions is
not viable?

@yifan-gu https://github.com/yifan-gu Do you mean extend Version() API
to include features? I think that's better than adding a new API.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32803 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVADZyi6Kvw70L3uZyzxDwdn4Gxtsks5qqg4-gaJpZM4J-JSY
.

@thockin

Do we need this as an explicit API? Why not just pass sysctl to the
runtime, and if it doesn't support them, it can either reject the request
or log-and-run-anyway...

@thockin One benefit is failing early before calling runtime API if features are not supported.

@yifan-gu

@yifan-gu Do you mean extend Version() API to include features? I think that's better than adding a new API.

Not saying including features in versions. My thought is the version itself is sufficient to provide information on what features are supported or not (e.g. docker 1.1x support feature a, b, c, etc)

Once the runtime tells us the version, we can infer what features are supported or not, right?

@yifan-gu

Not saying including features in versions. My thought is the version itself is sufficient to provide information on what features are supported or not (e.g. docker 1.1x support feature a, b, c, etc)

@yifan-gu Versions of different runtime mean different things. if we just let runtime return its version, kubelet must have some runtime specific logic to figure out what docker 1.1x supports, what rkt 1.0x supports etc.

@yifan-gu

@yifan-gu Versions of different runtime mean different things. if we just let runtime return its version, kubelet must have some runtime specific logic to figure out what docker 1.1x supports, what rkt 1.0x supports etc.

Fair enough.

@thockin

I am not sure that is a compelling enough reason on its own.

On Sep 15, 2016 8:45 PM, "Pengfei Ni" notifications@github.com wrote:

Do we need this as an explicit API? Why not just pass sysctl to the
runtime, and if it doesn't support them, it can either reject the request
or log-and-run-anyway...

@thockin https://github.com/thockin One benefit is failing early before
calling runtime API if features are not supported.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32803 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVDMVeLxY7GiEzWqhGQNPu-B-7wSxks5qqhDngaJpZM4J-JSY
.

Do we need this as an explicit API? Why not just pass sysctl to the
runtime, and if it doesn't support them, it can either reject the request
or log-and-run-anyway...

kubelet right now "rejects" a pod if the runtime doesn't support sysctl. To achieve the same effect, we'll need to either a) ask runtime to return a list of supported features or b) add a new "error" indicating the pod is not supported so that kubelet can reject the pod.
Handling version skew explicitly. If sysctl is added to version 1.1, an older server/runtime, e.g., 1.0, would simply ignore the newly added fields and not return errors. This means that kubelet will have to check what CRI version the runtime is using and whether a particular feature is supported in this version.

On Mon, Sep 19, 2016 at 2:38 PM, Yu-Ju Hong notifications@github.com wrote:

Do we need this as an explicit API? Why not just pass sysctl to the
runtime, and if it doesn't support them, it can either reject the request
or log-and-run-anyway...

kubelet right now "rejects" a pod if the runtime doesn't support sysctl. To achieve the same effect, we'll need to either a) ask runtime to return a list of supported features or b) add a new "error" indicating the pod is not supported so that kubelet can reject the pod.

This is what I was thinking

Handling version skew explicitly. If sysctl is added to version 1.1, an older server/runtime, e.g., 1.0, would simply ignore the newly added fields and not return errors. This means that kubelet will have to check what CRI version the runtime is using and whether a particular feature is supported in this version.

Kubelet should NOT be checking versions of plugins. It must be
neutral. The question is which behavior is correct? Rejecting pods
that use unsupported features or ignoring the features? Note this has
implications for defaulting logic, too. If we reject, we can never
default a new field lest we risk the kubelet rejecting it with version
skew (which empirically is the normal state of many clusters).

Ignoring features may be ok in some cases, and I think this is what we do in k8s today (older kubelets ignore new features).

On the other hand, if the feature is related to security (e.g., apparmor?), letting the runtime silently ignore it seems dangerous.

Can we somehow close the loop? Maybe pass back the runtime's pod, then do
a diff or something? I guess I am OK with feature flags, but I am worried
about the defaulting problem.

On Mon, Sep 19, 2016 at 4:23 PM, Yu-Ju Hong notifications@github.com
wrote:

Ignoring features may be ok in some cases, and I think this is what we do
in k8s today (older kubelets ignore new features).

On the other hand, if the feature is related to security (e.g.,
apparmor?), letting the runtime silently ignore it seems dangerous.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32803 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVBnW2saz84eQ9wmBhhy_bvhD2LuPks5qrxlfgaJpZM4J-JSY
.

@jessfraz

@jessfraz Any comments on this?

Yeah I think /info or something like it should return so the admission controllers can deny or not for features like they do with docker versions

#64005 is adding a similar GetRuntimeConfigInfo() interface in CRI, which returns the UID/GID mapping config of runtimes. This could be combined together in one info API.

@yujuhong

@yujuhong @Random-Liu @mikebrow @mrunalp For the list of features, any more ideas of this list:

enum RuntimeCapability {
    CAPABILITY_SYSCTL = 0;
    CAPABILITY_SECCOMP = 1;
    CAPABILITY_SELINUX  = 2;
    CAPABILITY_APPARMOR = 3;
}

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot

@fejta-bot: Closing this issue.

Details

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Merge pull request kubernetes#34830 from feiskyer/sysctl

chore(cli): generate conversion from cdk.json to cli arguments (kuber…

Navigation Menu

Repository navigation

CRI: expose optional runtime features #32803

38 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CRI: expose optional runtime features #32803

Description

Activity

yifan-gu commented on Sep 15, 2016

feiskyer commented on Sep 16, 2016

thockin commented on Sep 16, 2016

feiskyer commented on Sep 16, 2016

yifan-gu commented on Sep 16, 2016

Random-Liu commented on Sep 16, 2016

yifan-gu commented on Sep 16, 2016

thockin commented on Sep 17, 2016

yujuhong commented on Sep 19, 2016

thockin commented on Sep 19, 2016

yujuhong commented on Sep 19, 2016

thockin commented on Sep 19, 2016

38 remaining items

feiskyer commented on May 29, 2018

jessfraz commented on May 29, 2018

feiskyer commented on May 29, 2018

feiskyer commented on May 29, 2018

fejta-bot commented on Aug 27, 2018

fejta-bot commented on Sep 26, 2018

fejta-bot commented on Oct 26, 2018

k8s-ci-robot commented on Oct 26, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions