->searchable() update everything to Algolia, even with no changes

[elfeffe]

I'm using Laravel 5.3 and MongoDB database.
If I run ->searchable() in my model, I resend everything to Algolia, even when there is no changes. Incurring in charges in Algolia.
Is that normal?
This is not supposed to update only if there is any change?
I'm trying this because in some cases my items are not being updated in Algolia when I save it in my database, I don't know if it's related to MongoDB or to Scout. In any case I think there should be a way to update the index completely but only when the document is different to your local one.

[julienbourdeau]

If you want to achieve that, you will need to keep track locally of what's in Algolia index and I guess it can become pretty tricky.

You might be able to do something based on the updated_at columns. When you do your query, maybe you can a where statement to push only recently updated entries.

[zanematthew]

By nature this is how Elasticsearch works.

documents are immutable: they cannot be changed, only replaced.

So if you run ->searchable() its not going to check if there are changes. For that, you'll want to write your own implementation following the partial updates guide.

https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#_updates_with_a_partial_document

Note the latest version of Elasticsearch 6.0 introduces breaking changes! They are removing support for _type, which means your models will no longer map to a type in Elasticsearch. You'll have to map them on your own.

[Gummibeer]

I think it would be an idea to use the isDirty() method in the models.
It should be done in the observer and handled Like the isEmpty call.
But I think that this isn't something package related cause it could bring in some problems - out of sync algolia.

But I've solved it the same - I check if there is any searchable value in dirty state, otherwise I ignore this model.
Atm it's working pretty well and I never had problems. But I also have to do all changes via eloquent or trigger manually an algolia sync.

[timramseyjr]

I might be missing something, but is there a reason the update() method of AlgoliaEngine.php couldn't first check isDirty() and toSearchableArray()? Something like:

public function update($models)
{
	$models = $models->filter(function($model){
	   return $model->isDirty() && !empty($model->toSearchableArray());
	});
	if ($models->isEmpty()) {
		return;
	}

	$index = $this->algolia->initIndex($models->first()->searchableAs());
       ...
}

and then isEmpty() would return and there would be no API calls?

Or at least have a configuration option to enable/disable the above logic?

Edit: After reading all the comments, I see where this could cause algolia to be out of sync(I personally would rather have this than all the API calls), so I will just end up doing this in a custom engine.

[elfeffe]

I have created a custom engine for it
https://gist.github.com/elfeffe/54d4125447f0b7d65620a79a0efe165e
Works good, Algolia don’t charge per query, only indexing is charged.
It is designed for MongoDB, but should work with other DB I think.

[driesvints]

Gonna close this for the same reasons as in #285 and mentioned above.