Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

->searchable() update everything to Algolia, even with no changes #226

Closed
elfeffe opened this issue Sep 28, 2017 · 6 comments
Closed

->searchable() update everything to Algolia, even with no changes #226

elfeffe opened this issue Sep 28, 2017 · 6 comments

Comments

@elfeffe
Copy link

@elfeffe elfeffe commented Sep 28, 2017

I'm using Laravel 5.3 and MongoDB database.
If I run ->searchable() in my model, I resend everything to Algolia, even when there is no changes. Incurring in charges in Algolia.
Is that normal?
This is not supposed to update only if there is any change?
I'm trying this because in some cases my items are not being updated in Algolia when I save it in my database, I don't know if it's related to MongoDB or to Scout. In any case I think there should be a way to update the index completely but only when the document is different to your local one.

@elfeffe elfeffe changed the title ->searchable() update everything to Algolia, even whith no changes ->searchable() update everything to Algolia, even with no changes Sep 28, 2017
@julienbourdeau

This comment has been minimized.

Copy link
Contributor

@julienbourdeau julienbourdeau commented Sep 28, 2017

If you want to achieve that, you will need to keep track locally of what's in Algolia index and I guess it can become pretty tricky.

You might be able to do something based on the updated_at columns. When you do your query, maybe you can a where statement to push only recently updated entries.

@zanematthew

This comment has been minimized.

Copy link

@zanematthew zanematthew commented Dec 1, 2017

By nature this is how Elasticsearch works.

documents are immutable: they cannot be changed, only replaced.

So if you run ->searchable() its not going to check if there are changes. For that, you'll want to write your own implementation following the partial updates guide.

https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#_updates_with_a_partial_document

Note the latest version of Elasticsearch 6.0 introduces breaking changes! They are removing support for _type, which means your models will no longer map to a type in Elasticsearch. You'll have to map them on your own.

@Gummibeer

This comment has been minimized.

Copy link

@Gummibeer Gummibeer commented Jan 19, 2018

I think it would be an idea to use the isDirty() method in the models.
It should be done in the observer and handled Like the isEmpty call.
But I think that this isn't something package related cause it could bring in some problems - out of sync algolia.

But I've solved it the same - I check if there is any searchable value in dirty state, otherwise I ignore this model.
Atm it's working pretty well and I never had problems. But I also have to do all changes via eloquent or trigger manually an algolia sync.

@timramseyjr

This comment has been minimized.

Copy link

@timramseyjr timramseyjr commented Mar 26, 2018

I might be missing something, but is there a reason the update() method of AlgoliaEngine.php couldn't first check isDirty() and toSearchableArray()? Something like:

public function update($models)
{
	$models = $models->filter(function($model){
	   return $model->isDirty() && !empty($model->toSearchableArray());
	});
	if ($models->isEmpty()) {
		return;
	}

	$index = $this->algolia->initIndex($models->first()->searchableAs());
       ...
}

and then isEmpty() would return and there would be no API calls?

Or at least have a configuration option to enable/disable the above logic?

Edit: After reading all the comments, I see where this could cause algolia to be out of sync(I personally would rather have this than all the API calls), so I will just end up doing this in a custom engine.

@elfeffe

This comment has been minimized.

Copy link
Author

@elfeffe elfeffe commented May 15, 2018

I have created a custom engine for it
https://gist.github.com/elfeffe/54d4125447f0b7d65620a79a0efe165e
Works good, Algolia don’t charge per query, only indexing is charged.
It is designed for MongoDB, but should work with other DB I think.

@driesvints

This comment has been minimized.

Copy link
Member

@driesvints driesvints commented Oct 5, 2018

Gonna close this for the same reasons as in #285 and mentioned above.

@driesvints driesvints closed this Oct 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.