Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task Management #15117

Closed
imotov opened this issue Nov 30, 2015 · 23 comments

Comments

Projects
None yet
9 participants
@imotov
Copy link
Member

commented Nov 30, 2015

We have identified several potential features of elasticsearch that can spawn long running tasks and therefore require a common management mechanism for this tasks. This issue will introduce task management API that will provide a mechanism for communicating with and controlling currently running tasks. The task management API will be based on the top of existing TransportAction framework, which will allow any transport action to become a task.

The tasks will maintain parent/child relationship between tasks running on the coordinating nodes and subtasks that are spawn by the coordinating node on other nodes.

The task management will be introduced in several iterations. The first phase will be back-ported to 2.x and the second phase will be only available in 5.0.
Phase I

  • #15347 Create framework for registering and communicating with tasks.
  • #16033 Make the Task object available to the action caller
  • #15931 Establish parent/child relationship for master node, replication and broadcast actions
  • #16320 Add support for task cancellation
  • #16356 Add a task status field to the task info object
  • #16586 Add wait for task registration to testCanFetchIndexStatus
  • #16744 Combine node name and task id into single string task id
  • #16829 Store and report task start time and duration

Phase II

  • #17383 Add Task Management API Documentation
  • #17551 Add cat API version of task management APIs
  • #17928 Provide a generic mechanism for storing results of long running tasks
  • Add ability for a task to survive restart of coordinating node
@nik9000

This comment has been minimized.

Copy link
Contributor

commented Dec 2, 2015

I wonder if we need a way to store the results of a task until they are fetched? I'm thinking of something like update-by-query which would be a task because it is long running, cancelable, etc. But it wants to return counts of how many documents it updated and things like that. Maybe just write them to an index? Maybe with a ttl?

@raf64flo

This comment has been minimized.

Copy link

commented Dec 2, 2015

Nice remark of @nik9000 about long task results availability after its end, as it is already done for snapshots.
But I'd prefer a TTL or/and a dedicated query to drop the result instead of only drop on fetch, which could be problematic in my opinion.

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Dec 2, 2015

But I'd prefer a TTL or/and a dedicated query to drop the result instead of only drop on fetch, which could be problematic in my opinion.

Yeah - drop on fetch would be rough.

Not all tasks will want to do this but I think some would like it.

@imotov

This comment has been minimized.

Copy link
Member Author

commented Dec 2, 2015

@nik9000 is the goal to make results available after the task finished?

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Dec 2, 2015

@nik9000 is the goal to make results available after the task finished?

Yeah. In the case of update-by-query it'd be just to make the status available. The most "convenient" way to do it seems like write it to an index with a ttl - but I think I'm just stuck on that idea because it came to me. The point is that after the task is done you'll want to see what its results were for some period of time. You'd want some place you could fetch the results by task id, some way to clear out results when you've finished with them, some way for them to clear themselves out if you don't read them back soon enough.

I don't think it needs to come at iteration 1, but at some point it'd be nice.

Look at delete-by-query, it makes some effort to build a nice results object. Once it becomes a "task" it'll have nothing to do with the fancy result object.

Another thing that might be useful is to make an API that'd block until the task was finished and return the result of it. Or just fetch the result if it was already finished. This'd be super useful in general but kind of required for the REST tests because they don't have loops and things.

@imotov

This comment has been minimized.

Copy link
Member Author

commented Dec 2, 2015

I think traditionally we do that in two places - 1) log files for per-operation level and 2) in stats as combined metric. I can see how we might want to have a third way, but I think the biggest question here is lifecycle of this result. Persistence (even temporary persistence) of results is very unclear to me unless the result is associated with some persistent object (such as snapshot). So, I would rather make it an option to block and get result if you are interested in the result.

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Dec 2, 2015

So, I would rather make it an option to block and get result if you are interested in the result

I don't know if that'll be enough in the end though. Imagine the delete-by-query operation that takes 30 minutes too complete. Its too long for any blocking to be reliable - all kinds of http equipment will time you out after 5 minutes and something is bound to sneak in and get you a connection reset by peer.

So you'd have to build in a retry to the blocking. But if results aren't persisted, at least for a little while, then there is always the possibility that the job will finish between one request timing out and the next one starting. A low possibility but an icky one.

Something like a TTL on the result with explicit commands to read the result and delete it would work. These results wouldn't be huge documents so we could probably keep them in memory, certainly if they were serialized xContent or implemented Accountable or something.

Its complicated but I can't think of how else to report on tasks that are "do a thing" rather than "make a thing".

@nik9000 nik9000 referenced this issue Dec 2, 2015

Closed

Reindex API #15201

5 of 5 tasks complete
@niemyjski

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2015

+1

@clintongormley

This comment has been minimized.

Copy link
Member

commented Dec 5, 2015

Another user of task mgmt: the forced merge API

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2015

Another user of task mgmt: the forced merge API

I wonder if we should add a list of users to the top, like, below the requirements. I'm happy to work on retrofitting some of our long running requests to make their status more fetch-able and to make them more cancel-able but we should should make a list/tag the old issues.

imotov added a commit to imotov/elasticsearch that referenced this issue Dec 12, 2015

Task Management: Add framework for registering and communicating with…
… tasks

Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to elastic#15117
@jprante

This comment has been minimized.

Copy link
Contributor

commented Dec 21, 2015

Will it be possible to suspend/resume tasks by API? For perpetual tasks? Or to schedule tasks by a cron-like specification? It seems not, since the task design discusses TransportAction only, which means the lifetime of a task is "one-shot", corresponding to a request/response roundtrip of an action executed by a user?

@imotov

This comment has been minimized.

Copy link
Member Author

commented Dec 22, 2015

@jprante at the moment we are targeting use cases where tasks have clear start, stop and finite running time. We might extend it to perpetual tasks in the future, but this is not on the immediate road map.

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2015

Will it be possible to suspend/resume tasks by API?

I suspect that'll be opt-in in the same way that cancel will be opt-in. Reindex will probably opt in because it'll want to have API controlled throttling. So you could set the throttle to 0 and it'd just stop. The bulk request powering it would timeout pretty soon, making the whole thing fail though.

imotov added a commit to imotov/elasticsearch that referenced this issue Jan 5, 2016

Task Management: Add framework for registering and communicating with…
… tasks

Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to elastic#15117
@wuranbo

This comment has been minimized.

Copy link

commented Jan 13, 2016

We will has the internal result of a long-term-running task? For example:
The query will retrieve total 100 shards on 10 elastic node, each node has 10 shards. We can get the result immediately when the first shard is done. Then the second shard is done, we reduce the result to the former, send notify to the API, so the user can update their view. When all the shards are done, send notify that the task is done.
So we can execute this long-term-running task in background with less thread one shard by another, release the CPU resources for the high-priority task. And the user can update the views of long-term-running task frequently, got a better user experience.

@imotov

This comment has been minimized.

Copy link
Member Author

commented Feb 11, 2016

@clintongormley good idea, I have added it to the description

imotov added a commit to imotov/elasticsearch that referenced this issue Mar 4, 2016

Task Management: Add framework for registering and communicating with…
… tasks

Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to elastic#15117

karmi added a commit to elastic/elasticsearch-ruby that referenced this issue Mar 30, 2016

@clintongormley clintongormley added v5.0.0 and removed v5.0.0-beta1 labels Sep 14, 2016

@clintongormley clintongormley added v6.0.0-alpha1 and removed v5.0.0 labels Oct 11, 2016

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2017

@imotov, is this done now? I think we've decided not to do the "task can survive restart" thing, right?

@oleg-andreyev

This comment has been minimized.

Copy link

commented Jun 14, 2018

@clintongormley @imotov #15975 it was closed in favor of this ticket, so can we retrieve information about _forcemerge?

@imotov

This comment has been minimized.

Copy link
Member Author

commented Jun 15, 2018

@oleg-andreyev thanks! I have reopened #15975.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.