New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A "*" Constraint for unique should exist #846

Closed
florianleibert opened this Issue Dec 2, 2014 · 49 comments

Comments

Projects
None yet
@florianleibert
Member

florianleibert commented Dec 2, 2014

This constraint would ensure that a command is run on all slaves in the cluster.

@sabraham

This comment has been minimized.

Show comment
Hide comment
@sabraham

sabraham Dec 4, 2014

I am very interested in this kind of deployment (and more generally, running 1 and only 1 instance on each slave with a given attribute). In this case, I think the interface should disallow scaling the number of instances entirely, as the instance count should always be #(slaves with given attribute).

sabraham commented Dec 4, 2014

I am very interested in this kind of deployment (and more generally, running 1 and only 1 instance on each slave with a given attribute). In this case, I think the interface should disallow scaling the number of instances entirely, as the instance count should always be #(slaves with given attribute).

@normanjoyner

This comment has been minimized.

Show comment
Hide comment
@normanjoyner

normanjoyner Dec 4, 2014

👍 This is potentially a very useful constraint.

normanjoyner commented Dec 4, 2014

👍 This is potentially a very useful constraint.

@aseppala

This comment has been minimized.

Show comment
Hide comment
@aseppala

aseppala Dec 16, 2014

+1, this is definitely a common usecase. I would use it combined with UNIQUE hostname constraint to run exactly one instance on each host.

aseppala commented Dec 16, 2014

+1, this is definitely a common usecase. I would use it combined with UNIQUE hostname constraint to run exactly one instance on each host.

@villind

This comment has been minimized.

Show comment
Hide comment
@villind

villind commented Dec 16, 2014

+1

@ConnorDoyle

This comment has been minimized.

Show comment
Hide comment
@ConnorDoyle

ConnorDoyle Dec 16, 2014

Contributor

Makes sense, lots of support, let's do it.

Contributor

ConnorDoyle commented Dec 16, 2014

Makes sense, lots of support, let's do it.

@ConnorDoyle

This comment has been minimized.

Show comment
Hide comment
@ConnorDoyle

ConnorDoyle Dec 16, 2014

Contributor

The only strange part in the API is the instances key, which will have no bearing on the number of instances we actually launch. Any suggestions of how you'd like to see this handled? One option is add a onePerHost (mod naming) boolean option to the app definition and ignore the instances field if that is not supplied.

Contributor

ConnorDoyle commented Dec 16, 2014

The only strange part in the API is the instances key, which will have no bearing on the number of instances we actually launch. Any suggestions of how you'd like to see this handled? One option is add a onePerHost (mod naming) boolean option to the app definition and ignore the instances field if that is not supplied.

@ConnorDoyle ConnorDoyle added this to the 0.7.7 milestone Dec 16, 2014

@aseppala

This comment has been minimized.

Show comment
Hide comment
@aseppala

aseppala Dec 16, 2014

How about

"instances": "*",
"constraints": [["hostname", "UNIQUE"]]

aseppala commented Dec 16, 2014

How about

"instances": "*",
"constraints": [["hostname", "UNIQUE"]]
@tnn1t1s

This comment has been minimized.

Show comment
Hide comment
@tnn1t1s

tnn1t1s Dec 17, 2014

exactly. and this functionality becomes critical as the size of the cluster
becomes non-deterministic.

On Mon, Dec 1, 2014 at 8:38 PM, Florian Leibert notifications@github.com
wrote:

This constraint would ensure that a command is run on all slaves in the
cluster.


Reply to this email directly or view it on GitHub
#846.

tnn1t1s commented Dec 17, 2014

exactly. and this functionality becomes critical as the size of the cluster
becomes non-deterministic.

On Mon, Dec 1, 2014 at 8:38 PM, Florian Leibert notifications@github.com
wrote:

This constraint would ensure that a command is run on all slaves in the
cluster.


Reply to this email directly or view it on GitHub
#846.

@BrianHicks

This comment has been minimized.

Show comment
Hide comment
@BrianHicks

BrianHicks Dec 17, 2014

Contributor

👍 for "instances": "*" - would it just turn the constraint into a selector in general, though?

Contributor

BrianHicks commented Dec 17, 2014

👍 for "instances": "*" - would it just turn the constraint into a selector in general, though?

@defender

This comment has been minimized.

Show comment
Hide comment
@defender

defender commented Dec 19, 2014

+1

@andrewortman

This comment has been minimized.

Show comment
Hide comment
@andrewortman

andrewortman commented Dec 19, 2014

👍

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 19, 2014

Contributor

+1

Contributor

sttts commented Dec 19, 2014

+1

@prantaaho

This comment has been minimized.

Show comment
Hide comment
@prantaaho

prantaaho commented Dec 29, 2014

👍

@amiorin

This comment has been minimized.

Show comment
Hide comment
@amiorin

amiorin commented Dec 31, 2014

👍

@pigeonflight

This comment has been minimized.

Show comment
Hide comment

pigeonflight commented Jan 1, 2015

+1

@ConnorDoyle ConnorDoyle modified the milestones: 0.8.0, 0.8.1 Jan 20, 2015

@defender

This comment has been minimized.

Show comment
Hide comment
@defender

defender Feb 11, 2015

Hi ConnorDoyle
A "*" Constraint for unique should exist #846 do you think it will be released at 0.8.1, this bug was already reassign several times :)

thanks.

defender commented Feb 11, 2015

Hi ConnorDoyle
A "*" Constraint for unique should exist #846 do you think it will be released at 0.8.1, this bug was already reassign several times :)

thanks.

@drexin

This comment has been minimized.

Show comment
Hide comment
@drexin

drexin Feb 11, 2015

Contributor

We can't ensure that a task is running on all slaves in the cluster for following reasons:

  1. we can't guarantee that there are enough resources on every machine
  2. in a multi framework cluster, we don't know if we will ever receive offers for all the machines
  3. we could never tell if it has been deployed successfully, because we don't know about all the machines in the cluster

I think this would need direct support from Mesos.

Contributor

drexin commented Feb 11, 2015

We can't ensure that a task is running on all slaves in the cluster for following reasons:

  1. we can't guarantee that there are enough resources on every machine
  2. in a multi framework cluster, we don't know if we will ever receive offers for all the machines
  3. we could never tell if it has been deployed successfully, because we don't know about all the machines in the cluster

I think this would need direct support from Mesos.

@drexin drexin modified the milestones: 0.8.1, 0.8.2 Feb 24, 2015

@emilevauge

This comment has been minimized.

Show comment
Hide comment
@emilevauge

emilevauge commented Apr 9, 2015

+1

@drexin

This comment has been minimized.

Show comment
Hide comment
@drexin

drexin Apr 15, 2015

Contributor

Moving this to backlog, as it is not possible to implement this in a reliable manner without support from Mesos.

Contributor

drexin commented Apr 15, 2015

Moving this to backlog, as it is not possible to implement this in a reliable manner without support from Mesos.

@drexin drexin modified the milestones: Backlog, 0.8.2 Apr 15, 2015

@matt-deboer

This comment has been minimized.

Show comment
Hide comment
@matt-deboer

matt-deboer May 4, 2015

@drexin, you mentioned:

 3. we could never tell if it has been deployed successfully

Can't you just query the Mesos REST api for the cluster topology, and then keep track of whether the offers received (within some time-limit) covered all the hosts?

GET http://{mesos-master}:5050/registrar(1)/registry

You'd probably also have to have some way of listening for changes to the mesos registry such that you could deploy the "once-per-host" apps to newly added hosts.

For the other two,

  1. we can't guarantee enough resources on every machine
  2. in a multi framework cluster, we don't know if we will ever receive offers for all the machines

...aren't these true for any deployment? I.e., a deployment asks for N instances, there's no guarantee that there are N instances worth of resources available (and that we'll get offers for them)?

Maybe some of the resource contention problem could be alleviated by having some way to load these once-per-host apps on framework startup...(of course, that might require an ordered startup of frameworks--i.e., make marathon start first)?

Am I on the right track here?

matt-deboer commented May 4, 2015

@drexin, you mentioned:

 3. we could never tell if it has been deployed successfully

Can't you just query the Mesos REST api for the cluster topology, and then keep track of whether the offers received (within some time-limit) covered all the hosts?

GET http://{mesos-master}:5050/registrar(1)/registry

You'd probably also have to have some way of listening for changes to the mesos registry such that you could deploy the "once-per-host" apps to newly added hosts.

For the other two,

  1. we can't guarantee enough resources on every machine
  2. in a multi framework cluster, we don't know if we will ever receive offers for all the machines

...aren't these true for any deployment? I.e., a deployment asks for N instances, there's no guarantee that there are N instances worth of resources available (and that we'll get offers for them)?

Maybe some of the resource contention problem could be alleviated by having some way to load these once-per-host apps on framework startup...(of course, that might require an ordered startup of frameworks--i.e., make marathon start first)?

Am I on the right track here?

@wricardo

This comment has been minimized.

Show comment
Hide comment
@wricardo

wricardo commented May 27, 2015

👍

@Ferdiism Ferdiism added the shelf label May 29, 2015

@Ferdiism Ferdiism removed this from the Backlog milestone May 29, 2015

@aquamatthias aquamatthias modified the milestone: IceBox May 29, 2015

@aquamatthias aquamatthias removed the shelf label May 29, 2015

@rewiko

This comment has been minimized.

Show comment
Hide comment
@rewiko

rewiko commented Jul 13, 2015

+1

@kolloch

This comment has been minimized.

Show comment
Hide comment
@kolloch

kolloch Jul 24, 2015

Contributor

Basically, we would need to query the Mesos api for the available slaves regularly. We only get events for lost slaves. It's possible but this is the only use case so far.

But much more important: You somehow need to provision your slaves. If you make something that you want to run on all slaves part of your slave provisioning, you are sure that it runs on all slaves. Implementing this in Marathon is complicated AND by nature fragile.

Contributor

kolloch commented Jul 24, 2015

Basically, we would need to query the Mesos api for the available slaves regularly. We only get events for lost slaves. It's possible but this is the only use case so far.

But much more important: You somehow need to provision your slaves. If you make something that you want to run on all slaves part of your slave provisioning, you are sure that it runs on all slaves. Implementing this in Marathon is complicated AND by nature fragile.

@air

This comment has been minimized.

Show comment
Hide comment
@air

air Jul 29, 2015

Contributor

See #1264 for additional chat on this.

Contributor

air commented Jul 29, 2015

See #1264 for additional chat on this.

@BrickXu

This comment has been minimized.

Show comment
Hide comment
@BrickXu

BrickXu Nov 2, 2015

join this thread

BrickXu commented Nov 2, 2015

join this thread

@clehene

This comment has been minimized.

Show comment
Hide comment
@clehene

clehene Feb 10, 2016

Here's a the equivalent from Kubernetes - it's called "Daemon Sets"
Curious if Kubernetes on Mesos supports this.

As we describe Marathon as "the init or upstart daemon" it would make sense to have this feature ;)

@drexin I understand the complexity, I think the right way to go about this is to figure out what exactly is needed and discuss this with Mesos devs.

clehene commented Feb 10, 2016

Here's a the equivalent from Kubernetes - it's called "Daemon Sets"
Curious if Kubernetes on Mesos supports this.

As we describe Marathon as "the init or upstart daemon" it would make sense to have this feature ;)

@drexin I understand the complexity, I think the right way to go about this is to figure out what exactly is needed and discuss this with Mesos devs.

@philipnrmn

This comment has been minimized.

Show comment
Hide comment
@philipnrmn

philipnrmn Feb 12, 2016

Contributor

Some great documentation on using GROUP_BY and another feature request for this in #3220

Contributor

philipnrmn commented Feb 12, 2016

Some great documentation on using GROUP_BY and another feature request for this in #3220

@scatterbrain

This comment has been minimized.

Show comment
Hide comment
@scatterbrain

scatterbrain Mar 23, 2016

+1 really important for many kinds of services (monitoring etc)

scatterbrain commented Mar 23, 2016

+1 really important for many kinds of services (monitoring etc)

@graphex

This comment has been minimized.

Show comment
Hide comment
@graphex

graphex Mar 31, 2016

+1 a very useful addition. Yes, this could be done with less complexity to Marathon by provisioning all services on all slaves up front, but that just shifts the complexity and burden of updates and redeploys. If a frequently changing app needs to be on each host, having marathon manage redeployment/restarting/configuration in a consistent manner would be very helpful. Would be worth coordinating any necessary changes to Mesos, and appropriate separation of concerns. Is there a corresponding thread in Mesos for this?

graphex commented Mar 31, 2016

+1 a very useful addition. Yes, this could be done with less complexity to Marathon by provisioning all services on all slaves up front, but that just shifts the complexity and burden of updates and redeploys. If a frequently changing app needs to be on each host, having marathon manage redeployment/restarting/configuration in a consistent manner would be very helpful. Would be worth coordinating any necessary changes to Mesos, and appropriate separation of concerns. Is there a corresponding thread in Mesos for this?

@drewrobb

This comment has been minimized.

Show comment
Hide comment
@drewrobb

drewrobb Mar 31, 2016

Contributor

I have been doing this by setting an autoscaling rule against the metrics we populate for the number of running slaves against an app with a hostname:UNIQUE constraint that we want on every slave. (I also have a simple internally written autoscaling service for our marathon apps). We also configured each slave to have a small amount of resources reserved for a specific role, and that role is only used by apps that need to run on every slave. It has been working pretty nicely in practice.

Contributor

drewrobb commented Mar 31, 2016

I have been doing this by setting an autoscaling rule against the metrics we populate for the number of running slaves against an app with a hostname:UNIQUE constraint that we want on every slave. (I also have a simple internally written autoscaling service for our marathon apps). We also configured each slave to have a small amount of resources reserved for a specific role, and that role is only used by apps that need to run on every slave. It has been working pretty nicely in practice.

@mvanholsteijn

This comment has been minimized.

Show comment
Hide comment
@mvanholsteijn

mvanholsteijn Apr 8, 2016

Being able to define a application that needs to run on every slave is essential functionality of a scheduler.

In Nomad these are called system Jobs, in CoreOS Fleet these are called global units.

So, what will the Marathon equivalent be called and how do we activate them?

It would make our live really easy. We need it now.......

mvanholsteijn commented Apr 8, 2016

Being able to define a application that needs to run on every slave is essential functionality of a scheduler.

In Nomad these are called system Jobs, in CoreOS Fleet these are called global units.

So, what will the Marathon equivalent be called and how do we activate them?

It would make our live really easy. We need it now.......

@edurdias

This comment has been minimized.

Show comment
Hide comment
@edurdias

edurdias commented May 20, 2016

+1

@clehene

This comment has been minimized.

Show comment
Hide comment
@clehene

clehene May 20, 2016

Marathon shouldn't guarantee that resources exist.
We can guarantee resources in Mesos through reservations.

The 1-1 framework - role "binding" may become the next issue, though as we'd need / like to ensure resources by workloads (in Marathon) rather than have it for the entire framework.

clehene commented May 20, 2016

Marathon shouldn't guarantee that resources exist.
We can guarantee resources in Mesos through reservations.

The 1-1 framework - role "binding" may become the next issue, though as we'd need / like to ensure resources by workloads (in Marathon) rather than have it for the entire framework.

@vkhatri

This comment has been minimized.

Show comment
Hide comment
@vkhatri

vkhatri commented Jun 22, 2016

+1

@JamieCressey

This comment has been minimized.

Show comment
Hide comment
@JamieCressey

JamieCressey Oct 15, 2016

+1 this would be hugely beneficial for monitoring and log extraction etc.

JamieCressey commented Oct 15, 2016

+1 this would be hugely beneficial for monitoring and log extraction etc.

@mimmus

This comment has been minimized.

Show comment
Hide comment
@mimmus

mimmus Oct 26, 2016

We need this now 👍

mimmus commented Oct 26, 2016

We need this now 👍

@fksimon

This comment has been minimized.

Show comment
Hide comment
@fksimon

fksimon commented Nov 14, 2016

+1

@sybrandy

This comment has been minimized.

Show comment
Hide comment
@sybrandy

sybrandy Nov 15, 2016

@JamieCressey is correct. We're in the same situation where we want to make sure telegraf and logspout are running on all of the slaves for metrics and log collection. It would be great if Marathon/Mesos can ensure that we have instances of each on every node in case failed nodes come back online or we add additional nodes to the cluster.

sybrandy commented Nov 15, 2016

@JamieCressey is correct. We're in the same situation where we want to make sure telegraf and logspout are running on all of the slaves for metrics and log collection. It would be great if Marathon/Mesos can ensure that we have instances of each on every node in case failed nodes come back online or we add additional nodes to the cluster.

@jdef

This comment has been minimized.

Show comment
Hide comment
@jdef

jdef Nov 15, 2016

Member

Running daemons on every node in the cluster in an efficient, robust manner will almost certainly require assistance from Mesos. If you want this feature then I encourage you to vote for support in Mesos because without that we're a bit stranded in Marathon-land.

A workaround meanwhile is to use a hostname UNIQUE constraint with instances set to a greater number of agents than you ever expect to have in your cluster.

https://issues.apache.org/jira/browse/MESOS-6595

Member

jdef commented Nov 15, 2016

Running daemons on every node in the cluster in an efficient, robust manner will almost certainly require assistance from Mesos. If you want this feature then I encourage you to vote for support in Mesos because without that we're a bit stranded in Marathon-land.

A workaround meanwhile is to use a hostname UNIQUE constraint with instances set to a greater number of agents than you ever expect to have in your cluster.

https://issues.apache.org/jira/browse/MESOS-6595

@jdef jdef added the is blocked by label Nov 15, 2016

@silvamerica

This comment has been minimized.

Show comment
Hide comment
@silvamerica

silvamerica Nov 15, 2016

@jdef Does your workaround leave hanging deployments around, since Marathon will never be able to satisfy the number of instances specified?

silvamerica commented Nov 15, 2016

@jdef Does your workaround leave hanging deployments around, since Marathon will never be able to satisfy the number of instances specified?

@jdef

This comment has been minimized.

Show comment
Hide comment
@jdef

jdef Nov 15, 2016

Member

Using a greater number of instances is not ideal for interaction w/ the
deployment manager. If you're running a fairly static cluster then you
could set # instances to match the number of agents you expect to have. If
you expect to increase the number of agents in the cluster, then increase
instances to match just prior to adding the agent(s). It's a workaround,
not a "solution".

On Tue, Nov 15, 2016 at 4:31 PM, Nicholas Silva notifications@github.com
wrote:

@jdef https://github.com/jdef Does your workaround leave hanging
deployments around, since Marathon will never be able to satisfy the number
of instances specified?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#846 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACPVLC6ozQWMkivmjwXsXET35dHpIlOgks5q-iSjgaJpZM4DCvNI
.

Member

jdef commented Nov 15, 2016

Using a greater number of instances is not ideal for interaction w/ the
deployment manager. If you're running a fairly static cluster then you
could set # instances to match the number of agents you expect to have. If
you expect to increase the number of agents in the cluster, then increase
instances to match just prior to adding the agent(s). It's a workaround,
not a "solution".

On Tue, Nov 15, 2016 at 4:31 PM, Nicholas Silva notifications@github.com
wrote:

@jdef https://github.com/jdef Does your workaround leave hanging
deployments around, since Marathon will never be able to satisfy the number
of instances specified?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#846 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACPVLC6ozQWMkivmjwXsXET35dHpIlOgks5q-iSjgaJpZM4DCvNI
.

@gengmao

This comment has been minimized.

Show comment
Hide comment
@gengmao

gengmao Nov 16, 2016

@jdef probably a dumb question - why Marathon can't deploy a daemon (with instances=* and hostname UNIQUE) to any new agents offered by Mesos? What specific supports have to be provided by Mesos?

gengmao commented Nov 16, 2016

@jdef probably a dumb question - why Marathon can't deploy a daemon (with instances=* and hostname UNIQUE) to any new agents offered by Mesos? What specific supports have to be provided by Mesos?

@hampsterx

This comment has been minimized.

Show comment
Hide comment
@hampsterx

hampsterx commented Nov 17, 2016

+1

@dlaidlaw

This comment has been minimized.

Show comment
Hide comment
@dlaidlaw

dlaidlaw Nov 23, 2016

+1 1 per instance per role. If an agent has a specific role, and the acceptedResourceRoles matches then deploy there. This is like an autoscaling that is performed as soon as a new instance is detected and before any other deployments can happen on that new instance.

dlaidlaw commented Nov 23, 2016

+1 1 per instance per role. If an agent has a specific role, and the acceptedResourceRoles matches then deploy there. This is like an autoscaling that is performed as soon as a new instance is detected and before any other deployments can happen on that new instance.

@janisz

This comment has been minimized.

Show comment
Hide comment
@janisz

janisz Nov 23, 2016

Contributor

👎
What if there are no resources to run this app on every agent? If service needs to be running on all agents then there are better ways to put it there (upstart/systemd/supervisor + ansible/puppet/chef/salt).
Other orchestrators take advantage of "being the only framework" so they can schedule given tasks before node is available for others. In Mesos you can't do that. Easiest way to achieve it with Marathon is to follow this comment #846 (comment)
Finally mixing integer and string in one json field in the API is a mistake.

Contributor

janisz commented Nov 23, 2016

👎
What if there are no resources to run this app on every agent? If service needs to be running on all agents then there are better ways to put it there (upstart/systemd/supervisor + ansible/puppet/chef/salt).
Other orchestrators take advantage of "being the only framework" so they can schedule given tasks before node is available for others. In Mesos you can't do that. Easiest way to achieve it with Marathon is to follow this comment #846 (comment)
Finally mixing integer and string in one json field in the API is a mistake.

@dangra

This comment has been minimized.

Show comment
Hide comment
@dangra

dangra Nov 23, 2016

@janisz the resources for these services is something you usually plan and reserve ahead of time. I am not interested on "before any other deployments can happen", just running a service on all agents that meets the criteria is enough for me.

dangra commented Nov 23, 2016

@janisz the resources for these services is something you usually plan and reserve ahead of time. I am not interested on "before any other deployments can happen", just running a service on all agents that meets the criteria is enough for me.

@mimmus

This comment has been minimized.

Show comment
Hide comment
@mimmus

mimmus Nov 24, 2016

On CoreOS (systemd-based), I'm solving dropping-in a unit file to run and restart a monitoring Docker container but it would be great having a way to manage this by Marathon.

mimmus commented Nov 24, 2016

On CoreOS (systemd-based), I'm solving dropping-in a unit file to run and restart a monitoring Docker container but it would be great having a way to manage this by Marathon.

@tobilg

This comment has been minimized.

Show comment
Hide comment
@tobilg

tobilg Nov 24, 2016

Contributor

Question I see regarding this:

  • What should happen if there are agent nodes added to the cluster? Should Marathon automatically be able to schedule the app on these new nodes as well?

This would mean IMO that Marathon would need to to utilize the /slaves endpoint of the masters (I guess it does so already?), and compare the list of slave from the current call to the list of slaves from the last call... Not sure about the implications.

  • Concerning mixing string and integer in a property as @janisz said, I think this wouldn't be a good idea either.

This would effectively mean to change the type of the instances property to string, and then testing and potentially parsing it as integer. It could be discussed if the value -1 could replace *, but maybe there's a better idea.

Contributor

tobilg commented Nov 24, 2016

Question I see regarding this:

  • What should happen if there are agent nodes added to the cluster? Should Marathon automatically be able to schedule the app on these new nodes as well?

This would mean IMO that Marathon would need to to utilize the /slaves endpoint of the masters (I guess it does so already?), and compare the list of slave from the current call to the list of slaves from the last call... Not sure about the implications.

  • Concerning mixing string and integer in a property as @janisz said, I think this wouldn't be a good idea either.

This would effectively mean to change the type of the instances property to string, and then testing and potentially parsing it as integer. It could be discussed if the value -1 could replace *, but maybe there's a better idea.

@meichstedt

This comment has been minimized.

Show comment
Hide comment
Contributor

meichstedt commented Mar 7, 2017

@mesosphere mesosphere locked and limited conversation to collaborators Mar 27, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.