Should the kubelet perform a version check on the container runtime?

[pmorie]

Proper functioning of the kubelet and node depends upon an expected version of whatever container runtime the kubelet uses. For example, the proposal for QoS tiers being explored in #12035 will rely on the CPU quota feature introduced in docker 1.7, and I believe there is some code that depends upon docker 1.6. Another example: eventually mount propagation mode control in docker will be required to correctly containerize the kubelet. Should we have a version check in the kubelet for each container runtime to ensure the minimum usable version is present, and at least warn the user if not?

@derekwaynecarr @smarterclayton @bgrant0607

[thockin]

We have a mostly-unwritten rule of "3 docker revisions", but we definitely
need to do a better job at documenting and enforcing and testing that.

On Fri, Jul 31, 2015 at 8:44 AM, Paul Morie notifications@github.com
wrote:

Proper functioning of the kubelet and node depends upon an expected
version of whatever container runtime the kubelet uses. For example, the
proposal for QoS tiers being explored in #12035
#12035 will rely
on the CPU quota feature introduced in docker 1.7, and I believe there is
some code that depends upon docker 1.6. Another example: eventually mount
propagation mode control in docker will be required to correctly
containerize the kubelet. Should we have a version check in the kubelet for
each container runtime to ensure the minimum usable version is present, and
at least warn the user if not?

@derekwaynecarr https://github.com/derekwaynecarr @smarterclayton
https://github.com/smarterclayton @bgrant0607
https://github.com/bgrant0607

—
Reply to this email directly or view it on GitHub
#12090.

[derekwaynecarr]

Should it be an input decision to the scheduler? For example, if I upgrade my cluster to docker 1.9, we cannot use new features until docker 1.12?

[thockin]

So far we have avoided depending on new features until 3 versions work. We
have added support for new features that were purely extensions (e.g.
--ipc=container) so that they work if you have a new docker version.

On Fri, Jul 31, 2015 at 9:09 AM, Derek Carr notifications@github.com
wrote:

Should it be an input decision to the scheduler? For example, if I upgrade
my cluster to docker 1.9, we cannot use new features until docker 1.12?

—
Reply to this email directly or view it on GitHub
#12090 (comment)
.

[dchen1107]

Kubelet does a docker version check already, and healthcheck will failed if the version, API version lower than X. But that is very basic, and not related to featureset in a given docker version. Maybe kubelet could maintain a small list of feature set for docker versions, so that kubelet could query if a feature is present. If it is required, kubelet could even publish the featureset through NodeStatus?

[smarterclayton]

That makes a lot of sense to me - we for instance will want to be able to
leverage 1.7 and 1.8 docker features prior to EOL for 1.6

On Jul 31, 2015, at 2:13 PM, Dawn Chen notifications@github.com wrote:

Kubelet does a docker version check already, and healthcheck will failed if
the version, API version lower than X. But that is very basic, and not
related to featureset in a given docker version. Maybe kubelet could
maintain a small list of feature set for docker versions, so that kubelet
could query if a feature is present. If it is required, kubelet could even
publish the featureset through NodeStatus?

—
Reply to this email directly or view it on GitHub
#12090 (comment)
.

[dchen1107]

cc/ @bgrant0607 on potential API extension.

[bgrant0607]

I don't want the scheduler, admission control, and other parts of the system to need to reason about what API features work with what Docker version. We'd need to be able to consolidate the logic into something akin to a feature map evaluated in a single place, such as admission control.

I'm not saying it's the best thing to do, but internally we push this problem onto users. Not all kernel and container-runtime behaviors are explicit in our API, and we could try our best to determine which features worked on each version, but we'd also sometimes be wrong or out of date (e.g., when Docker is updated independently of Kubernetes).

We're going to need a mechanism for Kubelet to export system attributes via labels. Users could then constrain against those labels. We'd need to finish the work on the more general label selector #341, at least for nodeSelector.

[pmorie]

@bgrant0607 Interesting. Here's a semi-related problem: which nodes have SELinux enabled?

[bgrant0607]

Notes from today's hangout:

• Conditionalize only inside Kubelet if possible
• Export feature attributes and runtime kind (Docker, Rocket) and version as node labels
• Potentially add constraints (nodeSelector) in admission control
• Users should always be able to add constraints as an escape hatch

I think the alternatives are:

1. represent all attributes as labels
2. represent semantically meaningful attributes in a custom way (e.g., an attributes map in the node) and others as labels
3. represent semantically meaningful attributes in a custom way and all attributes as labels (so some will both be attributes and labels)

I was suggesting (1), but see some advantages to (3). (2) is likely not practical, since that would make some constraints completely implicit.

[dchen1107]

cc/ @pwittrock

[bgrant0607]

See also #9044.

[bgrant0607]

And #4855

[dchen1107]

@pwittrock is on leave for now. Will pick up the work once he is back. Retarget this one to 1.2

[dchen1107]

Kubelet doesn't some minimal docker version check since 1.1 release. Kubelet also support --node-labels for 1.2 release now. I am closing this issue.