Add ability to launch one-off tasks

[guenter]

Many apps require the ability to launch a one-off task (for example for database migrations). Add an easy way to do this in Marathon.

[guenter]

This branch has a POC: https://github.com/mesosphere/marathon/tree/one-off
It needs cleanup.

[chendo]

+1

[chendo]

We've hacked up a super dirty nsinit solution that lets us shell into a Docker container, however we're worried about the extra memory usage that might trigger the container into being killed

[karloscodes]

+1

[mo-mughrabi]

👍

[andytinycat]

+1; any progress on this?

[defender]

+1

[gregory90]

+1

[doctapp]

+1

[j1n6]

+1

[doctapp]

For now, the only way we found to run one-off tasks is to use the suicide method:
{
"id": "myapp",
"cmd": "somecommand && curl -X DELETE marathon:8080/v2/apps/myapp"
...
}

[karloscodes]

+1

On Thu, Mar 26, 2015 at 4:15 PM Martin Tapp notifications@github.com
wrote:

For now, the only way we found to run one-off tasks is to use the suicide
method:
{
"id": "myapp",
"cmd": "somecommand && curl -X DELETE marathon:8080/v2/apps/myapp"
...
}

Reply to this email directly or view it on GitHub
#344 (comment).

[xjewer]

+1

[mikljohansson]

+1

[doctapp]

Any progress on this?

[Lawouach]

Indeed, is it something the team may look into in the future?

[meichstedt]

Hey @Lawouach @doctapp and previous commenters,
Yes, we're aware that this makes sense in the context. There is an ongoing internal discussion on integrating one off job support into Marathon, but we do not yet have a clear direction.

Feel free to share use cases: migrations, setting up prerequisites, general cron jobs – which features would you need in order to have decent benefit?

[doctapp]

One use case is running an ad hoc job such as running a batch job (e.g.
ETL) or training a machine learning model.

The principal benefit is that we can perform these tasks whenever they are
required (daily basis) without extensive planning.

We are using the 'suicide hack' that I posted for these jobs. We are using
marathon since it's API is better suited to our needs than a framework like
Chronos.

On Mon, Nov 30, 2015 at 9:19 PM, Matthias Eichstedt <
notifications@github.com> wrote:

Hey @Lawouach https://github.com/Lawouach @doctapp
https://github.com/doctapp and previous commenters,
Yes, we're aware that this makes sense in the context. There is an ongoing
internal discussion on integrating one off job support into Marathon, but
we do not yet have a clear direction.

Feel free to share use cases: migrations, setting up prerequisites,
general cron jobs – which features would you need in order to have decent
benefit?

—
Reply to this email directly or view it on GitHub
#344 (comment)
.

[Lawouach]

Another use-case, though it's more likely a workaround at this stage, is to run a task to post-configure a task. Unless I have missed something, Marathon doesn't offer me a way to run post-configuration scripts from the host where those tasks are running so I resort to one-shot tasks.

I grant you this is more a workaround than a real use-case mind you.

[kolloch]

What do you all think about having a separate service for one-off jobs with some Marathon integration, e.g. for life cycle hooks?

Or another way of asking: What would be missing compared to a solution where it all is integrated into Marathon? What does "integration" mean for you?

[doctapp]

The idea of running a Marathon job to handle one-off jobs beats the purpose of a scheduler handling such a task. Plus, we don't need another task taking cluster resources for that purpose; that's the framework's job.

We need a flag or some way of telling Marathon not to restart a job if it finishes.

[kolloch]

@doctapp I should have made it more clear: My idea was creating a separate service (= framework) that is able to run any number of one-off tasks. So you would need to talk to a different end-point to achieve this.

In Marathon itself we could include hooks for life-cycle events that would end-up creating one-off tasks using the separate service.

Would that satisfy your requirements? If not, what exactly is missing.

[doctapp]

The problem with another framework is that we would need to support two
types of APIs. From my perspective, we need Marathon's api with one-off
task launching and handling. That's why we're not using Chronos or others.

Why can't you simply extend Marathon for this purpose? The suicide hack is
the better alternative to another framework in my opinion...

On Mon, Dec 14, 2015 at 7:47 AM, Peter Kolloch notifications@github.com
wrote:

@doctapp https://github.com/doctapp I should have made it more clear:
My idea was creating a separate service (= framework) that is able to run
any number of one-off tasks. So you would need to talk to a different
end-point to achieve this.

In Marathon itself we could include hooks for life-cycle events that would
end-up creating one-off tasks using the separate service.

Would that satisfy your requirements? If not, what exactly is missing.

—
Reply to this email directly or view it on GitHub
#344 (comment)
.

[kolloch]

@doctapp "Why can't you simply extend Marathon for this purpose?" We probably could extend it. I question the "simply". I'd rather focus on making single components of software as simple as possible.

In my opinion, the API for one-off-jobs SHOULD be different. For one-off-jobs, it is common to wait for them to complete (which does not make sense for long-running services). There is more...

Also the scheduler could be optimized for this different use case. For long-running tasks, you expect to have relatively few task starts/terminations. For one-off-scheduling that is the norm.

That said it should also use the same abstractions where possible (e.g. how to specify the command to run, resource constraints, ...).

[doctapp]

Agree and totally understand this. I don't mind having a distinct marathon
API for this purpose, as long as it's aligned between both type of tasks.
One-off tasks only need a subset of what long-running tasks expect.

That's why having one common API and endpoint (being a single Marathon
endpoint from the client's perspective), but with different
frameworks/implementation would be very interesting from my perspective...
We just don't want to deal with different framework APIs to launch tasks
that share common parameters mainly ports, resources...

On Mon, Dec 14, 2015 at 9:32 AM, Peter Kolloch notifications@github.com
wrote:

@doctapp https://github.com/doctapp "Why can't you simply extend
Marathon for this purpose?" We probably could extend it. I question the
"simply". I'd rather focus on making single components of software as
simple as possible.

In my opinion, the API for one-off-jobs SHOULD be different. For
one-off-jobs, it is common to wait for them to complete (which does not
make sense for long-running services). There is more...

Also the scheduler could be optimized for this different use case. For
long-running tasks, you expect to have relatively few task
starts/terminations. For one-off-scheduling that is the norm.

That said it should also use the same abstractions where possible (e.g.
how to specify the command to run, resource constraints, ...).

—
Reply to this email directly or view it on GitHub
#344 (comment)
.

[assumednormal]

Has there been any recent progress on this?

[air]

Good news everyone - this is now actively in design phase.

[ghost]

@air This is a much needed feature. Is there any high level estimate when can this be ready in the main branch?

[ghost]

Is there a work around to have one-off docker containers using Marathon?

@doctapp Provided a nice workaround with suicide method for the commands. Is there anyway to achieve this for docker containers?

@chendo Can you please provide more ideas for the hack you tried?

[ryanolson]

+1

[junneyang]

+1
database migration
batch configuration(pre,post,etc) for multi apps

[strzelecki-maciek]

+1 much needed feature imho.
Run arbitrary command in the same "context" as the defined app (same env, same docker image etc),
a.) allow to specify if the N instances of the deployed task can be spawned in parallel with this command or shortly after it finishes.
b.) Configured timeout on the command would be a nice addition too.

Use cases - cache warming, pre-start cleanup of some distributed services, database schema migrations and much more.

[nathanjackson]

+1

[kartikjoshi07]

@doctapp Provided a nice workaround with suicide method for the commands. Is there anyway to achieve this for docker containers?

[meichstedt]

All, we have built Metronome on top of the Marathon core to provide this functionality. Please direct any requests to that project.

[yuandy]

Has there been any recent progress on this?

[grogsaxle]

+1

[rmrfself]

+1

[ozayduman]

+1

[meichstedt]

By popular request, we're happy to actually work on this. #5009 will provide more details soon.

Yes! It's true! It's happening!

[doctapp]

Excellent, finally no more suicide hack!