Use queued job for "unsearching" when Scout queue is enabled

[stevebauman]

Description:

Closes #331

This PR adds a RemoveFromSearch queuable job which will be dispatched when scout.queue is enabled.

This specific queuable job has an overwritten restoreCollection() method that instantiates an Eloquent collection of the deleted models, by force-filling the model primary keys supplied by the ModelIdentifier object. This allows extended Scout engines to retrieve the ID used for indexing, so they can purge records properly.

All credit goes to @matt-allan, as this is a slightly modified copy of his work on his forked repo 🙌

---

References:

Here are references to the most popular Scout Elasticsearch packages (they only use getScoutKey() for retrieving a unique identifier for the model):

https://github.com/babenkoivan/elastic-scout-driver/blob/2e45a933dcc8b051b25cbf8010623e51ac457a35/src/Factories/DocumentFactory.php#L24

https://github.com/babenkoivan/scout-elasticsearch-driver/blob/15d349a0401e2bab6f774255dc3668fd28b6d4a0/src/Indexers/BulkIndexer.php#L67

https://github.com/babenkoivan/scout-elasticsearch-driver/blob/15d349a0401e2bab6f774255dc3668fd28b6d4a0/src/Payloads/DocumentPayload.php#L19-L29

[driesvints]

This lgtm overall. I do wonder about concurrency issues when both indexing and removing from index happens at the same time. But I guess this has been extensively tested in some production apps with the Scout Elasticsearch driver?

[driesvints]

@chamby @matt-allan would be cool if you could give this a review if you can spare the time :)

[matt-allan]

This looks good to me! Thanks for putting this together!

[driesvints]

Thanks @stevebauman!

[stevebauman]

Happy to help! Thanks guys!

[georgebohnisch]

Best day ever! Thanks @stevebauman, you're a hero to many.

[driesvints]

Btw @stevebauman did you test this with both Algolia & Meilisearch?

[stevebauman]

@driesvints I didn't 😞 , do you have any links you may have handy so I could easily whip up a test environment for these providers? If not, I can dive in head first and give it a go -- let me know!

[driesvints]

Unfortunately not I'm afraid. You'll need to set up a test account with Algolia and follow the install instructions for Meilisearch. I'll hold off with the release until you've tested them out. No rush.

[stevebauman]

Ok no worries! Challenge accepted 🙌 . I'll see if I can get this done tonight.

[shokme]

tested with meilisearch on a small set of data (1100 items)
using ->unseachable(), data are well sent to the jobs table, after running queue:work everything was removed from Meilisearch.

[driesvints]

Gonna skip this for this week's release then. Let me know if Algolia checks out alright as well.

[stevebauman]

We're good to go!

Queued.Unsearch.mp4

[driesvints]

I've just tagged v9.1.0 with this. Thanks all!

[stevebauman]

@shokme Thanks a ton for testing this with Meilisearch -- much appreciated 🙌

[GC-Mark]

@stevebauman @driesvints After a few hours of investigation, we finally figured out that this PR broke our production setup when trying to remove Models from Algolia using an Aggregator.

Not sure if the responsibility lies with Scout or Scout Extended?

The following exception is thrown...

[Algolia\ScoutExtended\Exceptions\ModelNotDefinedInAggregatorException] Model not defined in aggregator.

Here's the stack trace...

#0 /home/vendor/algolia/scout-extended/src/Searchable/ObjectIdEncrypter.php(42): Algolia\ScoutExtended\Searchable\Aggregator->getScoutKey()
#1 /home/vendor/algolia/scout-extended/src/Jobs/DeleteJob.php(58): Algolia\ScoutExtended\Searchable\ObjectIdEncrypter::encrypt(Object(App\Search\KeywordSearch))
#2 [internal function]: Algolia\ScoutExtended\Jobs\DeleteJob->Algolia\ScoutExtended\Jobs\{closure}(Object(App\Search\KeywordSearch), 0)
#3 /home/vendor/laravel/framework/src/Illuminate/Collections/Collection.php(642): array_map(Object(Closure), Array, Array)
#4 /home/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Collection.php(348): Illuminate\Support\Collection->map(Object(Closure))
#5 /home/vendor/algolia/scout-extended/src/Jobs/DeleteJob.php(59): Illuminate\Database\Eloquent\Collection->map(Object(Closure))
#6 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Algolia\ScoutExtended\Jobs\DeleteJob->handle(Object(Algolia\AlgoliaSearch\SearchClient))
#7 /home/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#8 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#9 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#10 /home/vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#11 /home/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call(Array)
#12 /home/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(Algolia\ScoutExtended\Jobs\DeleteJob))
#13 /home/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Algolia\ScoutExtended\Jobs\DeleteJob))
#14 /home/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#15 /home/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php(421): Illuminate\Bus\Dispatcher->dispatchNow(Object(Algolia\ScoutExtended\Jobs\DeleteJob), false)
#16 /home/vendor/algolia/scout-extended/src/Engines/AlgoliaEngine.php(78): dispatch_now(Object(Algolia\ScoutExtended\Jobs\DeleteJob))
#17 /home/vendor/laravel/scout/src/Jobs/RemoveFromSearch.php(41): Algolia\ScoutExtended\Engines\AlgoliaEngine->delete(Object(Illuminate\Database\Eloquent\Collection))
#18 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\Scout\Jobs\RemoveFromSearch->handle()
#19 /home/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#20 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#21 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#22 /home/vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#23 /home/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call(Array)
#24 /home/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(Laravel\Scout\Jobs\RemoveFromSearch))
#25 /home/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Laravel\Scout\Jobs\RemoveFromSearch))
#26 /home/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#27 /home/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(120): Illuminate\Bus\Dispatcher->dispatchNow(Object(Laravel\Scout\Jobs\RemoveFromSearch), false)
#28 /home/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue\{closure}(Object(Laravel\Scout\Jobs\RemoveFromSearch))
#29 /home/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Laravel\Scout\Jobs\RemoveFromSearch))
#30 /home/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(122): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#31 /home/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\Queue\Jobs\RedisJob), Object(Laravel\Scout\Jobs\RemoveFromSearch))
#32 /home/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(98): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\RedisJob), Array)
#33 /home/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(428): Illuminate\Queue\Jobs\Job->fire()
#34 /home/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(378): Illuminate\Queue\Worker->process('redis', Object(Illuminate\Queue\Jobs\RedisJob), Object(Illuminate\Queue\WorkerOptions))
#35 /home/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(172): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\RedisJob), 'redis', Object(Illuminate\Queue\WorkerOptions))
#36 /home/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(117): Illuminate\Queue\Worker->daemon('redis', 'redis-low', Object(Illuminate\Queue\WorkerOptions))
#37 /home/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\Queue\Console\WorkCommand->runWorker('redis', 'redis-low')
#38 /home/vendor/laravel/horizon/src/Console/WorkCommand.php(51): Illuminate\Queue\Console\WorkCommand->handle()
#39 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\Horizon\Console\WorkCommand->handle()
#40 /home/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#41 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#42 /home/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#43 /home/vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#44 /home/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\Container\Container->call(Array)
#45 /home/vendor/symfony/console/Command/Command.php(288): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#46 /home/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#47 /home/vendor/symfony/console/Application.php(974): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#48 /home/vendor/symfony/console/Application.php(291): Symfony\Component\Console\Application->doRunCommand(Object(Laravel\Horizon\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#49 /home/vendor/symfony/console/Application.php(167): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#50 /home/vendor/laravel/framework/src/Illuminate/Console/Application.php(92): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#51 /home/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#52 /home/artisan(37): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#53 {main}

If you need any more information to figure this out, let me know.

Thanks

[driesvints]

@GC-Mark since that's an Algolia Extended concept it's best to report that to their issue tracker.

[GC-Mark]

Added the issue over at https://github.com/algolia/scout-extended incase anyone else stumbles across this - algolia/scout-extended#282