New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide for dependencies between tasks in a group #419

Open
mfischer-zd opened this Issue Nov 15, 2015 · 27 comments

Comments

Projects
None yet
@mfischer-zd

Tasks in a group sometimes need to be ordered to start up correctly.

For example, to support the Ambassador pattern, proxy containers (P[n]) used for outbound request routing by a dependent application may be started only after the dependent application (A) is started. This is because Docker needs to know the name of A to configure shared-container networking when launching P[n].

In the first approximation of the solution, ordering can be simple, e.g., by having the task list in a group be an array.

@ChrisHines

This comment has been minimized.

Show comment
Hide comment
@ChrisHines

ChrisHines Nov 16, 2015

Contributor

+1

Batch jobs need task dependencies to manage data-flow between tasks.

I would expand this to allow dependencies between tasks in a job rather than just a task group. Data flow is not constrained to a single node.

Contributor

ChrisHines commented Nov 16, 2015

+1

Batch jobs need task dependencies to manage data-flow between tasks.

I would expand this to allow dependencies between tasks in a job rather than just a task group. Data flow is not constrained to a single node.

@gsuyashs

This comment has been minimized.

Show comment
Hide comment
@gsuyashs

gsuyashs Nov 30, 2015

Any tentative date to add this enhancement feature in Nomad?

Any tentative date to add this enhancement feature in Nomad?

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Nov 30, 2015

Contributor

@suyash1983: we don't have any dates yet but it will most likely be something that comes in 0.4

Contributor

dadgar commented Nov 30, 2015

@suyash1983: we don't have any dates yet but it will most likely be something that comes in 0.4

@blalor

This comment has been minimized.

Show comment
Hide comment
@blalor

blalor Jan 15, 2016

Contributor

This is really going to be core to a number of use cases, like allowing EBS volumes to be mounted on a host before the application starts.

Contributor

blalor commented Jan 15, 2016

This is really going to be core to a number of use cases, like allowing EBS volumes to be mounted on a host before the application starts.

@c4milo

This comment has been minimized.

Show comment
Hide comment
@c4milo

c4milo Mar 4, 2016

Contributor

or pulling down certificates and secrets from Vault before starting a given service.

Contributor

c4milo commented Mar 4, 2016

or pulling down certificates and secrets from Vault before starting a given service.

@vrenjith

This comment has been minimized.

Show comment
Hide comment
@vrenjith

vrenjith Mar 7, 2016

Contributor

+1
This will be a great addition for large enterprise cluster deployments

Contributor

vrenjith commented Mar 7, 2016

+1
This will be a great addition for large enterprise cluster deployments

@steve-jansen

This comment has been minimized.

Show comment
Hide comment
@steve-jansen

steve-jansen Mar 16, 2016

Contributor

+1

Running consul-template alongside an app...

Contributor

steve-jansen commented Mar 16, 2016

+1

Running consul-template alongside an app...

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Mar 16, 2016

Contributor

@steve-jansen You can run consul-template currently. We do this in our Nomad deployment. Essentially you have a script that runs consul-template which produces your config, and then runs your binary

Contributor

dadgar commented Mar 16, 2016

@steve-jansen You can run consul-template currently. We do this in our Nomad deployment. Essentially you have a script that runs consul-template which produces your config, and then runs your binary

@steve-jansen

This comment has been minimized.

Show comment
Hide comment
@steve-jansen

steve-jansen Mar 16, 2016

Contributor

@dadgar nice, thanks for sharing. It's always great to hear how a company uses their own products. I imagine consul-template creates the env vars in the job config...

One catch for us, we aspire to have consul-template rewrite the config in response to changes in a Consul k/v, or more importantly, when Vault rotates a secret. We're trying to have consul-template signal the co-scheduled task to reload its config.

Indeed, my need for task dependencies is very narrow. In a perfect world for me, Nomad would have integration with Consul K/V and/or Vault. My need is config that updates dynamically for tasks. That would eliminate my need for task dependencies.

Contributor

steve-jansen commented Mar 16, 2016

@dadgar nice, thanks for sharing. It's always great to hear how a company uses their own products. I imagine consul-template creates the env vars in the job config...

One catch for us, we aspire to have consul-template rewrite the config in response to changes in a Consul k/v, or more importantly, when Vault rotates a secret. We're trying to have consul-template signal the co-scheduled task to reload its config.

Indeed, my need for task dependencies is very narrow. In a perfect world for me, Nomad would have integration with Consul K/V and/or Vault. My need is config that updates dynamically for tasks. That would eliminate my need for task dependencies.

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Mar 16, 2016

Contributor

@steve-jansen consul-template can supervise the process and restart/signal it whenever the config changes. @sethvargo to verify

Contributor

dadgar commented Mar 16, 2016

@steve-jansen consul-template can supervise the process and restart/signal it whenever the config changes. @sethvargo to verify

@sethvargo

This comment has been minimized.

Show comment
Hide comment
@sethvargo

sethvargo Mar 16, 2016

Contributor

Hi @steve-jansen

Vault currently does not support blocking queries, which is what Consul (and CT) use to give you that "real-time" trigger when something changes. CT will renew secrets at lease_duration/2.0, but it's not currently possible to trigger a change in Vault and have that immediately notify another process. There is an open issue in Vault for blocking queries. Please note this was an intentional design decision in Vault for performance reasons, and not a bug in CT or Vault.

CT will start the process, but it is not a supervisor or monitor. CT has the expectation that the given command will return within 30s, typically by delegating to some supervisor. If you're running in pure Docker, CT can optionally cleanup PIDs and act like PID 1, but it doesn't monitor the process itself (which is best monitored by the scheduler anyway).

Contributor

sethvargo commented Mar 16, 2016

Hi @steve-jansen

Vault currently does not support blocking queries, which is what Consul (and CT) use to give you that "real-time" trigger when something changes. CT will renew secrets at lease_duration/2.0, but it's not currently possible to trigger a change in Vault and have that immediately notify another process. There is an open issue in Vault for blocking queries. Please note this was an intentional design decision in Vault for performance reasons, and not a bug in CT or Vault.

CT will start the process, but it is not a supervisor or monitor. CT has the expectation that the given command will return within 30s, typically by delegating to some supervisor. If you're running in pure Docker, CT can optionally cleanup PIDs and act like PID 1, but it doesn't monitor the process itself (which is best monitored by the scheduler anyway).

@OferE

This comment has been minimized.

Show comment
Hide comment
@OferE

OferE Sep 18, 2016

+1 - any updates on this?

OferE commented Sep 18, 2016

+1 - any updates on this?

@nugend

This comment has been minimized.

Show comment
Hide comment
@nugend

nugend Oct 10, 2016

I'd also like to see this land. I'm evaluating nomad for service orchestration and it would be very nice to be able to explicitly express dependencies.

Alternatively, or as a stop gap, maybe a recipe for doing this with the existing APIs could be written up (#1065 mentions this is possible)

nugend commented Oct 10, 2016

I'd also like to see this land. I'm evaluating nomad for service orchestration and it would be very nice to be able to explicitly express dependencies.

Alternatively, or as a stop gap, maybe a recipe for doing this with the existing APIs could be written up (#1065 mentions this is possible)

@mbravorus

This comment has been minimized.

Show comment
Hide comment
@mbravorus

mbravorus Dec 18, 2016

Currently, the use cases which require dependencies and cron-like control tend to push me towards airbnb's Chronos which implies Mesos. I would be delighted if I could just use Nomad.

Recurring jobs are possible (starting from #540, ultimately described in https://www.nomadproject.io/docs/job-specification/periodic.html), but in a very common scenario where periodic tasks need interlocking dependency control, there is no way to do it with Nomad currently (or I didn't manage to find it)

mbravorus commented Dec 18, 2016

Currently, the use cases which require dependencies and cron-like control tend to push me towards airbnb's Chronos which implies Mesos. I would be delighted if I could just use Nomad.

Recurring jobs are possible (starting from #540, ultimately described in https://www.nomadproject.io/docs/job-specification/periodic.html), but in a very common scenario where periodic tasks need interlocking dependency control, there is no way to do it with Nomad currently (or I didn't manage to find it)

@nugend

This comment has been minimized.

Show comment
Hide comment
@nugend

nugend Dec 20, 2016

Given that this doesn't seem to be a high priority, could someone from the project briefly explain the suggestion about how to achieve this with existing APIs as mentioned in #1065?

nugend commented Dec 20, 2016

Given that this doesn't seem to be a high priority, could someone from the project briefly explain the suggestion about how to achieve this with existing APIs as mentioned in #1065?

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Dec 20, 2016

Contributor

@nugend You would essentially need to put a wrapper around the task that uses the allocation endpoint to determine if the task it should wait for has finished successfully and then start itself. It is non-trivial amount of work though

Contributor

dadgar commented Dec 20, 2016

@nugend You would essentially need to put a wrapper around the task that uses the allocation endpoint to determine if the task it should wait for has finished successfully and then start itself. It is non-trivial amount of work though

@RobertAtomic

This comment has been minimized.

Show comment
Hide comment
@RobertAtomic

RobertAtomic Sep 27, 2017

+1 for DAG support... It's quite important for my use case.

+1 for DAG support... It's quite important for my use case.

@tduffield

This comment has been minimized.

Show comment
Hide comment
@tduffield

tduffield Sep 28, 2017

I too would love to see this.

I too would love to see this.

@SunSparc

This comment has been minimized.

Show comment
Hide comment
@SunSparc

SunSparc Oct 21, 2017

+1 for task sequence/dependency

+1 for task sequence/dependency

@nugend

This comment has been minimized.

Show comment
Hide comment
@nugend

nugend Oct 23, 2017

For anyone really pining for this, there is a slightly easier method provided you are registering the upstream tasks as services: use a template stanza to create a run script and make that the command of your task. Then you can use a conditional dependent on the service being available to either launch the actual process or sleep forever.

Works pretty well, though it does require a bit of scripting (and I’m not sure how to do it in Windows)

nugend commented Oct 23, 2017

For anyone really pining for this, there is a slightly easier method provided you are registering the upstream tasks as services: use a template stanza to create a run script and make that the command of your task. Then you can use a conditional dependent on the service being available to either launch the actual process or sleep forever.

Works pretty well, though it does require a bit of scripting (and I’m not sure how to do it in Windows)

@vrenjith

This comment has been minimized.

Show comment
Hide comment
@vrenjith

vrenjith Nov 4, 2017

Contributor
Contributor

vrenjith commented Nov 4, 2017

@abohne

This comment has been minimized.

Show comment
Hide comment
@abohne

abohne Dec 12, 2017

Is there any update on where this might fall on the roadmap?

abohne commented Dec 12, 2017

Is there any update on where this might fall on the roadmap?

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Dec 13, 2017

Contributor

Likely in 0.9

Contributor

dadgar commented Dec 13, 2017

Likely in 0.9

@k1ng87

This comment has been minimized.

Show comment
Hide comment
@k1ng87

k1ng87 Jan 18, 2018

come on 0.9!

k1ng87 commented Jan 18, 2018

come on 0.9!

@Miserlou

This comment has been minimized.

Show comment
Hide comment
@Miserlou

Miserlou Feb 7, 2018

How likely is likely? :)

Miserlou commented Feb 7, 2018

How likely is likely? :)

@prandelicious

This comment has been minimized.

Show comment
Hide comment
@prandelicious

prandelicious Jun 5, 2018

Any news on this?

Any news on this?

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Jun 5, 2018

Contributor

As mentioned early this is either in 0.9.x or 0.10.x.

Contributor

dadgar commented Jun 5, 2018

As mentioned early this is either in 0.9.x or 0.10.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment