Show force merge/optimize progress

[markwalkom]

Are we able to expose this somehow, maybe pending tasks?
At the moment it just runs until finished, but I'm hoping there may be a way to figure out some kind of progress percentage, even a rough one.

I realise this is all a little vague, but I don't really know if it's possible, just that it'd be nice to have :p

[clintongormley]

If it can be done, it'll be via the task management API. Closing in favour of #15117

[imotov]

I think we dropped this issue on the floor at the end of the task management task. Now that the task management framework is in the place, should we revisit this issue?

[oleg-andreyev]

@imotov I'm not an expert in elasticsearch internal things, but here are my though as consumers:

• force merge action should have wait_for_completion flag as reindex action, if it is possible, this way I as consumer would send simple http request with wait_for_completion=false and won't wait for completion, because I could monitor force merge from _tasks endpoint
• also it would be great to know if force merge is cancelable? Force merge should be cancellable #17094

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@jpountz is there any way to track progress of a user driven forced merge?

[jpountz]

Not everything is exposed publicly today, but we could expose information about ongoing merges. Giving progress information is more complex due to the fact that Lucene doesn't merge document by document but data-structure by data-structure, eg. the inverted index of _id, then the inverted index of foo then doc values of seq_no, then doc values of foo, etc.

[redlus]

I understand this is a bit more complex than tracking the progress of other tasks in the cluster, but even showing the number of indices which already finished (in a multi-index force-merge) and the number of data-structures (or doc_values) already processed would be nice. Is there any plan to add this functionality?

Thanks

[cdenneen]

using the tasks api to see that a forcemerge might still be running doesn't really tell us anything as far as % completion to give estimation... when this is combined with something like curator to forcemerge a bunch of indexes that meet criteria it would be useful to know when these will end so they don't step on each other or cause other contention issues.

[SpencerLN]

The ability to at least understand what index is currently merging would be very useful. We have had users accidentally start of a force merge for an index actively being written to and there was no apparent way to identify which index was being operated on by a given task.

[pedrosk]

Since some large indices with many segments may take for a long time to force merge, it would be beneficial to:

1. see the task existing in a LIST (!) and being able to see its status (eg. in progress, failed, etc)

I feel it is essential to be able to know what is happening with forge merges that take extended amount of time.

[tanandy]

Hi, thats something we need too

[phalverson]

Even something as high level as whether the merge is running or complete would be huge. I don't need to have a play-by-play on what's going on in Lucene, I just need to know when I can start indexing documents again. Please address this soon.

[Ankithas93]

I am able to see merge: current by running below command:
curl -XGET 'http://els2002:9200/_stats?pretty'

  "merges" : {
    "current" : 0,

Is this the one to monitor to check the current merge in progress?
I am running 7.8.0

[DaveCTurner]

The task list contains a node-level task for each node holding shards that are to be force-merged. I think we could decorate this task with things like how many shards it targets, how many of those shard-level tasks are complete, the shard of the currently-running task, and how long that shard-level task has been running. We have all of that information available I think, it's more a question of exposing it in the task list.

More detailed progress from Lucene would be nice too, but that's a much larger ask, so let's rule that out of scope for now.

I'm marking this as help wanted to invite community contributions as this isn't something we expect to work on in the near future. I don't think it's an enormous job.

[ruslaniv]

But what exactly does current represent:

1. Number of segments left to merge. I.e. total 29, merged 27, left 2, currently merging segment # 2
2. Sequential number of the segment it is merging right now. I.e. total segments to merge 29, has merged segment # 1 now merging segment # 2

I really hope it is the former because the merge process been running for 7 days now and there is absolutely no way currently to check if anything is happening and how much time I still have to wait for the merge to complete.

"merges": {
                    "current": 2,
                    "current_docs": 28652980,
                    "current_size_in_bytes": 157314725265,
                    "total": 29,

I wish there was detailed explanation of index_name/_stats/merge results somewhere

[przemekwitek]

there is absolutely no way currently to check if anything is happening and how much time I still have to wait for the merge to complete

+1

I have a similar issue with a large dataset that I'm trying to use in rally.
I don't know whether the _forcemerge progresses or not.