ReindexThread always runs - never stops

[freddyucv]

This Thread is doin a Database request two times for second here:

https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotmarketing/common/reindex/ReindexThread.java#L179

I think is better is it. just is Paused when not exists reindex to proccess and start it again when a reindex is push into the queue, here could be a good place:

https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ContentletIndexAPIImpl.java#L581

Also the ReindexThread can try to procces the same reindex more than once, it keep a 'lastIdIndexed' to know witch content is already been proccesing:

https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotmarketing/common/reindex/ReindexQueueFactory.java#L274

but after a empty response from the database this index is set to 0

https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotmarketing/common/reindex/ReindexQueueFactory.java#L283

so it possible that meanwhile wait for a put reindex request the ReindexThread awake again and try to reindex the same contentlet again (it happend to me locally several times)

Describe the solution you'd like

I think the ReindexThread should pause after finish to send all the reindex request, and if a new reindex is register or a reindex fail it thread is start again.

[wezell]

Important note - a dotcms server can/should handle not just locally saved content, but content that has been imported by other servers in the cluster - indexing should be distributed across dotCMS nodes in a cluster which for the most part it is.

If we were to revisit this code, the next step would be to leverage a distributed multi-consumer queue (at least once) like kafka or redis streams to handle the queuing/de-queuing and retrying of the content to be indexed.

[wezell]

Important note - a dotcms server can/should handle not just locally saved content, but content that has been imported by other servers in the cluster - indexing should be distributed across dotCMS nodes in a cluster which for the most part it is.

If we were to revisit this code, the next step would be to leverage a distributed multi-consumer queue (at least once) like kafka or redis streams to handle the queuing/de-queuing and retrying of the content to be indexed.

Such a queue implementation will be started in this issue -
#18149

[nollymar]

PR: #19470

[nollymar]

Note to QA: Please test the creation/edition of pieces of content. Besides, it's important to run a full reindex, especially in a clustered environment

[fabrizzio-dotCMS]

I did set up a two-nodes cluster (node-1, and node-2) on my local machine.
Then I tested the following scenarios:

1. I edited single pieces of content and verified the changes became visible for both nodes. No problem. The edition of one single piece of content seems to be processed by the same node issuing the reindex operation.

2. I made modifications on content-types that already had a number of pieces of content and I can tell that the reindex operation isn't necessarily attended by the node issuing the operation. The changes in the pieces of content are picked by the two nodes smoothly.

3. I tested bulk workflow actions. Firing them from the content search portlet and I was able to verify the ReindexThread workload is distributed on both nodes (Or at least the movement on the logs give me such an impression).
   I see entries like :

13:46:16.438  INFO  reindex.ReindexThread - ---  ReindexThread total/todo/bulk: 5590/6/12
13:46:16.463  INFO  reindex.ReindexThread - ---  ReindexThread total/todo/bulk: 5600/5/10
13:46:16.479  INFO  reindex.ReindexThread - ---  ReindexThread total/todo/bulk: 5604/2/4
13:46:16.496  INFO  reindex.ReindexThread - ---  ReindexThread total/todo/bulk: 5608/2/4
13:46:16.508  INFO  reindex.ReindexThread - ---  ReindexThread total/todo/bulk: 5610/1/2

in both Node-1 and Node-2
The workload doesn't seem to be distributed evenly between the two nodes but that doesn't seem to be an issue.

4. I fired a reindex operation and again the Reindex Process seems to be distributed between the two nodes
   I see something like this going on in both nodes. The progress is visible on the UI from the two nodes.

13:38:13.631  INFO  reindex.BulkProcessorListener - -----------
13:38:13.631  INFO  reindex.BulkProcessorListener - Reindexing Server #  : 2 of 2
13:38:13.631  INFO  reindex.BulkProcessorListener - Total Indexed        : 250
13:38:13.631  INFO  reindex.BulkProcessorListener - ReindexEntries found : 300
13:38:13.632  INFO  reindex.BulkProcessorListener - BulkRequests created : 250
13:38:13.632  INFO  reindex.BulkProcessorListener - Full Reindex Elapsed : 5.632s

which again makes me think the operation is distributed and processed by the two nodes.

Therefore I am passing it to QA

As a side note, I had to increase my ES Heap memory to overcome a recurrent exception that wouldn't let me complete testing.

The exception looks more or less like this:

11:08:55.392  ERROR reindex.ReindexThread - Bulk  process failed entirely:Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [988315602/942.5mb], which is larger than the limit of [986061209/940.3mb], real usage: [988223152/942.4mb], new bytes reserved: [92450/90.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=92450/90.2kb, accounting=67407331/64.2mb]]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [<http_request>] would be [988315602/942.5mb], which is larger than the limit of [986061209/940.3mb], real usage: [988223152/942.4mb], new bytes reserved: [92450/90.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=92450/90.2kb, accounting=67407331/64.2mb]]

But I was able to make it go away by changing the value of pram "PROVIDER_ELASTICSEARCH_HEAP_SIZE=2g"
in our dot_opendistro.

[bryanboza]

After these changes we have too many lines of the same message which is causing a lot of noise in the logs, we need to take care of this.

[nollymar]

Fixed. The message will be logged every 60 minutes:
PR: #19600

[bryanboza]

Fixed, now we don't have this message in the logs.