More general scheduling constraints

[thockin]

There have been a few folks who have asked about machine constraints for scheduling. Let's use this issue as a place to gather ideas and requirements.

@timothysc

[verdverm]

I have noticed the FirstFit (default?) scheduler co-locates pods when there are open machines available. Each of these machines has a single cpu.

It would be nice to use information about available cpu and a pod's expected cpu requirements

sed 's/cpu/other_machine_stat/'

[monnand]

Currently, kubelet could get stats from cAdvisor which would be useful for scheduler. It could provide different percentiles of CPU and memory usage of a container (including the root container, i.e. the machine).

[thockin]

That's just "scheduling", as opposed to machine constraints, though very
coarsely they feel similar :)

On Mon, Jul 7, 2014 at 1:32 PM, Tony Worm notifications@github.com wrote:

I have noticed the FirstFit (default?) scheduler co-locates pods when
there are open machines available. Each of these machines has a single cpu.

It would be nice to use information about available cpu and a pod's
expected cpu requirements

sed 's/cpu/other_machine_stat/'

Reply to this email directly or view it on GitHub
#367 (comment)
.

[timothysc]

Labels "seems" like the ideal place to enable a rank & requirements to define constraints. However labels would need to be regularly published by minions.

e.g.
rank = memory
requirements = gpu & clusterXYZ

I have a couple of concerns here:

1. This treads into the full scale scheduling world.
2. Config syntax = ?, DSL? ...

[thockin]

Let's worry about semantics before syntax. We have a similar issue open
for label selectors in general - we can discuss syntax there.

On Mon, Jul 7, 2014 at 1:42 PM, Timothy St. Clair notifications@github.com
wrote:

Labels "seems" like the ideal place to enable a rank & requirements to
define constraints. For example:

rank = memory
requirements = gpu & clusterXYZ

I have a couple of concerns here:

1. This treads into the full scale scheduling world.
2. Config syntax = ?, DSL? ...

Reply to this email directly or view it on GitHub
#367 (comment)
.

[timothysc]

FWIW I often view constraints as a SQL query on a nvp store.

SELECT Resources
FROM Pool
WHERE Requirements
ORDER BY Rank
...

The hardest part are the 'fields' in an nvp store.

[bgrant0607]

Scheduling based on resources and constraints are 2 significantly different issues.

We have several issues open about resource (and QoS) awareness: #147 , #160 , #168 , #274 , #317.

Constraint syntax/semantics: We should start with the proposed label selector mechanism, #341 .

[timothysc]

I'm ok with doing the selection from a set of offers/resources from the scheduler.

Provided the offers have enough NVP information to enable discrimination.

[thockin]

I don't know about NVP - where can I read more on it?
On Jul 8, 2014 7:45 PM, "Timothy St. Clair" notifications@github.com
wrote:

I'm ok with doing the selection from a set of offers/resources from the
scheduler.

Provided the offers have enough NVP information to enable discrimination.

Reply to this email directly or view it on GitHub
#367 (comment)
.

[bgrant0607]

Searching for "NVP SQL" or "name value pair SQL" or "key value pair SQL" comes up with lots of hits. Common arguments against are performance and loss of control over DB schema. But I'm getting the feeling that we're barking up the wrong forest.

@timothysc What are you trying to do? Right now, k8s has essentially no intelligent scheduling. However, that's not a desirable end state. If what you want is a scheduler, we should figure out how to support scheduling plugins and/or layers on top of k8s.

[thockin]

Name Value Pairs? Now I feel dumb :)

On Tue, Jul 8, 2014 at 7:51 PM, Tim Hockin thockin@google.com wrote:

I don't know about NVP - where can I read more on it?
On Jul 8, 2014 7:45 PM, "Timothy St. Clair" notifications@github.com
wrote:

I'm ok with doing the selection from a set of offers/resources from the
scheduler.

Provided the offers have enough NVP information to enable discrimination.

Reply to this email directly or view it on GitHub
#367 (comment)
.

[bgrant0607]

Something somewhat different than label selectors is per-attribute limits for spreading. Aurora is one system that supports this model:
https://aurora.incubator.apache.org/documentation/latest/configuration-reference/#specifying-scheduling-constraints

This is more relevant to physical rather than virtual deployments. I'd consider it a distinct mechanism from constraints. @timothysc If you'd like this, we should file a separate issue. However, I'd prefer a a new failure tolerance scheduling policy object that specifies a label selector to identify the set of instances to be spread. We could debate about how to describe what kind and/or how much spreading to apply, but I'd initially just leave it entirely up to the infrastructure.

[timothysc]

I completely agree its more relevant to physical rather then virtual deployments.

I was somewhat testing the possibility of enabling the capabilities for more general purpose scheduling, on par with a mini-Condor approach but it's not a requirement.

Aurora or Marathon -esk capabilities will fill the gap.
https://github.com/mesosphere/marathon/wiki/Constraints