Support mesos resource roles (public_slave) in k8sm scheduler

[sttts]

• As Alice (app dev), using DCOS CE, I want to be able to configure pods to be scheduled on public DCOS slaves.
• As Alice (app dev), using Mesos or DCOS EE, I want to be able to configure pods to be scheduled with resources restricted to specific resource roles. (Not all Mesos clusters have slaves partitioned as "public" using resource roles.)

Note: Alice should already be able to schedule pods using node selectors that map to mesos slave attributes, but only if the resources from those slaves are not restricted to specific roles.

[jdef]

it might be fun to run k8s service-loadbalancer on a public slave:

• https://github.com/kubernetes/contrib/tree/master/service-loadbalancer

[sttts]

Wondering what the dcos load balancer story is. Having something for k8s is nice, but it's a more general problem.

[jdef]

Right now it's "bring your own"

On Tue, Sep 8, 2015 at 3:45 AM, Dr. Stefan Schimanski <
notifications@github.com> wrote:

Wondering what the dcos load balancer story is. Having something for k8s
is nice, but it's a more general problem.

—
Reply to this email directly or view it on GitHub
#482 (comment)
.

[sttts]

Probably the role should be exported as a label to the node. Then the node label predicate can be used to select a role for a pod.

[jdef]

a framework should only be offered resources according to the role that the framework registered with. there's no need to label nodes w/ roles

[sttts]

can't you schedule some pods on private, some on public slaves? How does marathon do this then?

[sttts]

Reading the docs, it seems that you can set the framework role to "public_slave". Then you get offers for * and public_slave.

[jdef]

Yes, thanks for clarifying. That's because * means all roles.

[s-urbaniak]

FYI: While doing a first deep-dive I discovered that we are already aware of a mesos role flag in the scheduler.

ref https://github.com/kubernetes/kubernetes/search?q=MesosRole

I'll concentrate on the predicate part.

[jdef]

When we claim cpu,mem,port resources (procurement) we may need to set the
role of the Resource's we're using when building up the TaskInfo.

Consider that we also allocate resources for the Executor (and static
pods). If role needs to be set in the Resource's for ExecutorInfo then
we'll have some extra work to do. We probably only want to assign a role to
Resource's of the ExecutorInfo once, for a particular slave - the first
time it's created.

On Thu, Oct 1, 2015 at 8:52 AM, Sergiusz Urbaniak notifications@github.com
wrote:

FYI: While doing a first deep-dive I discovered that we are already aware
of a mesos role flag in the scheduler.

ref https://github.com/kubernetes/kubernetes/search?q=MesosRole

I'll concentrate on the predicate part.

—
Reply to this email directly or view it on GitHub
#482 (comment)
.

[s-urbaniak]

@jdef thanks for the hints, I'll need to read more code to get all the puzzle pieces

[s-urbaniak]

Re "Setting roles in the ExecutorInfo's resources": I am collecting collecting infos/docs from core devs since it's not documented very well.

[s-urbaniak]

As per discussion w/ @joerg84: The ExecutorInfo's resource roles are independent of launched task resource roles. Executor resources (and their roles) only define its own resource needs. An executor can launch tasks with different resource roles

[sttts]

We might pull this into 0.7.0 if it applies easily.