I was able to make populate cmd 5-10 times faster.

[makasim]

The idea is split the job into smaller chunks and do it in parallel. I used enqueue bundle and RabbitMQ broker to do the job. 5x gain of performance was achieved by using 10 consumers and single ElasticSearch node. 500k entities were indexed within 3 minutes instead of 15 minutes.
It looks same as current populate command, including the progress bar.

Here's the code php-enqueue/enqueue-sandbox#1

Most important classes to look at AppBundle\Async\ ElasticaPopulateProcessor and AppBundle\Elasticsearch\ AsyncProvider.

The question is. Would it be something that we can add to fos elastica bundle as default or optional solution?

Update:

I put the code to a bundle https://github.com/php-enqueue/enqueue-elastica-bundle. It extends fos elastica bundle with messaging features.

[makasim]

I can do it as a stand alone bundle but maybe would you prefer to see it as part of the bundle?

[makasim]

https://github.com/php-enqueue/enqueue-elastica-bundle

[lsmith77]

cool .. maybe create a doc PR to reference to this Bundle?

[makasim]

@lsmith77 will do!

[Koc]

@makasim is it works without slowdowns when offset is big? Also is it possible run it without rabbitmq, for example populate command exec N commands using nohup &?

[makasim]

@Koc

for example populate command exec N commands using nohup &?

no it is not possible, the solution is built on top of messaging concept. Though you can use any transport. Currently Amqp and Stomp are supported. They cover a lot of brokers out there including RabbitMQ. In theory something simple like file transport could be added.

is it works without slowdowns when offset is big?

I did not modify any queries, so query performance should be the same as it is now.

[makasim]

messaging allows to distribute the job among different servers, exec + nohup is tied to one server.

[guillaume-perreal]

Is there any plan for a listener that delegates indexation to the workers ?

[makasim]

@guillaume-perreal what do you mean by "workers"?

[makasim]

tested the FOSElasticaBundle + EnqueueElasticaBundle with filesystem transport. php-enqueue/enqueue-dev#12

It took 5 minutes to index 400k entities (10 consumers). RabbitMQ took 3 minutes. Good results IMO.

Pros:

1. You dont need any brokers, like rabbitmq
2. Simple to use

Cons:

1. Only auto ack mode supported
2. Local. The work could not be distributed among several servers.

[XWB]

We will continue in #1192