I can't index my documents (they are ok at Algolia)

[elfeffe]

I'm using Laravel 7, MS 0.13
I'm trying to index my model using:
php artisan scout:import "App\Models\Product"
Then I do (local_product is my index):
curl -X GET 'http://localhost:7700/indexes/local_product/documents'
And I get []

If I index in Algolia, my documents are ok there.
Also if I index movies doing:
curl -X POST 'http://127.0.0.1:7700/indexes/movies/documents' --data @movies.json
The movies are also indexed.

[shokme]

Hello @elfeffe,

This is hard to help you without more information.

if you log your meilisearch instance did you get:

"POST /indexes/local_product/documents?primaryKey=id HTTP/1.1" 202 14 

If you have use of toSearchableArray() in your Product model, can you show me the content, so I might be able to reproduce the same error.

[elfeffe]

I’m using MongoDb. I think the key is _id
Can be that?
I use toSearchableArray to filter some fields, but they index on Algolia. And I dump the final array and it’s ok.
I don’t have the computer now, what do you want me to try?

[elfeffe]

I can't understand what you mean with

if you log your meilisearch instance did you get:

Let me know what do you want me to try

[elfeffe]

If I try

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 287947,"title": "Shazam","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

I get
{"updateId":31}

But then I do
curl -X GET 'http://localhost:7700/indexes/local_product/documents'
And I get

[]
Again

[eskombro]

Hello @elfeffe

You can always check an update status, most of the times MeiliSearch will tell you what the error was, in case the update failed.

You can check: Get an update status

And for the last example you can try

curl -X GET 'http://localhost:7700/indexes/local_product/updates/31'

which should give you more information on what is happening.

[shokme]

I’m using MongoDb. I think the key is _id
Can be that?

When you have created your index local_product did you specify the key parameter ?
php artisan scout:index local_product --key _id

I use toSearchableArray to filter some fields, but they index on Algolia. And I dump the final array and it’s ok.

Can you also try by comment the toSearchableArray(), I don't think this is the problem here, but it cost nothing to try.

[elfeffe]

I have removed toSearchableArray()
Then I have tried doing:
php artisan scout:index local_product --key _id
Doesn't work.

Then doing
php artisan scout:index local_product --key id
Together with

/**
     * Get the value used to index the model.
     *
     * @return mixed
     */
    public function getScoutKey()
    {
        return $this->id;
    }

    /**
     * Get the key name used to index the model.
     *
     * @return mixed
     */
    public function getScoutKeyName()
    {
        return 'id';
    }

Nothing, when I do
curl -X GET 'http://localhost:7700/indexes/local_product/documents'

I just get one document, the one indexed using

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 287947,"title": "Shazam","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

Is there any log that I can check?

[shokme]

When you import your data, did you get something like this:

shok@pop-os:~/web/blog$ php artisan scout:import "App\Models\Post"
Imported [App\Models\Post] models up to ID: 100
All [App\Models\Post] records have been imported.

logs are located storage/logs/laravel.log

Did you set queue => true in scout.php or SCOUT_QUEUE=true in your .env ?

[elfeffe]

I have tried with both true and false for queues. Always I get

Imported [App\Models\Product] models up to ID: 5eea4b314c80095faf5be593
Imported [App\Models\Product] models up to ID: 5eea4bd94c80095faf5be787
Imported [App\Models\Product] models up to ID: 5eea4cc34c80095faf5be97b
Imported [App\Models\Product] models up to ID: 5eea4dfa4c80095faf5beb6f
Imported [App\Models\Product] models up to ID: 5eea4f6b4c80095faf5bed63
Imported [App\Models\Product] models up to ID: 5eea51164c80095faf5bef57
Imported [App\Models\Product] models up to ID: 5eea524e4c80095faf5bf099

But there are no ducuments.
I know where laravel's logs are located, but I have no errors there, what I want to see are the MS logs.
Is there any way to see what is happening?

With all the same settings, Algolia is indexing my documents.

[shokme]

Ok, Meilisearch laravel scout is only an engine like a copy paste of the Algolia one. So it shouldn't be a problem but also I have never tried to run it with mongodb...

@eskombro @curquiza can answer to were are the MS logs.

[elfeffe]

This is related to MongoDB, because Users model is MySQL and works.
I mean, this is related to MongoDB / Meilisearch, because Algolia also work with MongoDB.

It will be great if i can ee some logs from MS, it is sad if Algolia supports it but not Meilisearch.

[shokme]

I can try to dig the problem, but I need your help on how I can install mongoDB with laravel

[eskombro]

As for now, MeiliSearch logs are directly printed to stdout, so it depends on where you are running your MeiliSearch instance. Where is it running?

I opened an issue for discussing persistent logs in MeiliSearch here: meilisearch/meilisearch#898

@elfeffe did you check the update status as I suggested in previous comments? If the documents were successfully sent to MeilISearch and you got a status ID, you will very probably find inside that update status the information about what is the problem when MeilISearch tries to index. Try this: #36 (comment)

[elfeffe]

Look at this

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 287947,"title": "Shazam","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

That works, but

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 5eea4bd94c80095faf5be787,"title": "Shazam 2","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

"5eea4bd94c80095faf5be787" is a MongoDB _id
That returns this beauty

{"message":"Invalid JSON: invalid number at line 1 column 11","errorCode":"bad_request","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#bad_request"}

I think that JSON is valid, there is some problem with MS and custom Key.
I don't know if it's the length, the letters, or what.

[elfeffe]

After some tries I think the letters are the problem, MS doesn't accept letters on the id. But Laravel let you put letters there.
That is the problem I think

[eskombro]

Yes, you just need to send that as a string, surrounded by double quotes

[curquiza]

It will work with "id": "5eea4bd94c80095faf5be787" instead of "id": 5eea4bd94c80095faf5be787 because 5eea4bd94c80095faf5be787 is not a valid number 🙂

[elfeffe]

But I don't control how your Scout extension send it 🙂

[elfeffe]

Even doing this

/**
     * Get the value used to index the model.
     *
     * @return mixed
     */
    public function getScoutKey()
    {
        return (string)$this->_id;
    }

It doesn't work, I think this engine must handle string ids

[shokme]

do the same thing in toSearchableArray()

return [ 'id' => (string) $this->_id]
or
return [ '_id' => (string) $this->_id]

I don't know what you have set as key

[elfeffe]

 public function toSearchableArray()
    {
        $array = $this->toArray();

        $array['id'] = (string)$array['_id'];

        return $array;
    }

That works

[elfeffe]

One last thing.
Algolia handle's that automatically, together with the index creation.
I mean, Algolia creates the index if doesn't exist, and I can index IDs with letters without any conversion.
I think MS should include both features.

[curquiza]

I mean, Algolia creates the index if doesn't exist

Indeed, the MeiliSearch API does not handle this feature yet.
As an alternative solution, most of our SDKs provide a get_or_create_index method (as the PHP SDK does) 🙂

and I can index IDs with letters without any conversion.

The problem you had comes from the JSON validity. Even Algolia API does not accept invalid JSON: sending a number with letters without quoting it (") is considered as an invalid JSON and would fail.

See the Algolia dashboard:

Maybe the Algolia's Laravel SDK transforms your number with letters into a string and that's why you didn't have any failure 🙂

I think MS should include both features.

So maybe meilisearch-laravel-scout could include the "quoting feature", but MeiliSearch could not since it expects a valid JSON 😉

[elfeffe]

Yes, in both cases I'm speaking about this Scout Engine, we are in the Scout integration here.
The SDK can provide a method to create the index, but Scout fails if the index doesn't exist.

The same with Id with letters, I understand that JSOn is malformed, but the Scout engine shpuld control that before it fails.
As the Algolia one does, because that one doesn't fail with letters.

[curquiza]

Ok ok wasn't sure to understand 😉

They could be nice features to implement then! 🙂

Edit: we should think if we implement that in the meilisearch-php or in the meilisearch-laravel-scout package.

[shokme]

@curquiza
It depends, both are good.

If we implement it in scout I guess the symfony(and others framework implementation) should also implement it (may be it's already the case). So it might be better to have it in the SDK instead ?

I'm not sure scout implement it for Algolia, I think this is handle by the Algolia PHP SDK, I will check this when I have a bit of free time.

[curquiza]

@shokme, you're right, it might be smarter to do it in the PHP SDK rather in the laravel-scout integration.
Let us know about your investigation, I will move the issue (or recreate one) accordingly! 🙂

[shokme]

So, after investigating the whole thing.
I have installed mongodb in laravel using jenssegers/laravel-mongodb.

I have started by creating my index.
php artisan scout:index cars -k _id

create a model

<?php

namespace App;

use Jenssegers\Mongodb\Eloquent\Model;
use Laravel\Scout\Searchable;

class Car extends Model
{
    use Searchable;

    public function getScoutKey() // not required if this is already your primary key
    {
        return $this->_id;
    }

    public function getScoutKeyName() // not required if this is already your primary key
    {
        return '_id';
    }
}

creating a new car

$car = new App\Car();
$car->name = 'mustang';
$car->constructor = 'ford';
$car->type = 'muscle';
$car->horses = 280;
$car->save();

The model was send to meilisearch without any problem !

{
  "status": "processed",
  "updateId": 1,
  "type": {
    "name": "DocumentsAddition",
    "number": 1
  },
  "duration": 0.005275826,
  "enqueuedAt": "2020-08-29T17:43:55.104917708Z",
  "processedAt": "2020-08-29T17:43:55.121602037Z"
}

When doing a search I get the correct result

App\Car::search('mustang')->get();

[
  {
    "_id": "5f4a93dbd9668370086c7752",
    "name": "mustang",
    "constructor": "ford",
    "type": "muscle",
    "horses": 280,
    "updated_at": "2020-08-29T17:43:55.102000Z",
    "created_at": "2020-08-29T17:43:55.102000Z"
  }
]

The only problem I have encounter while trying to reproduce the bug was by forgot to set _id has primaryKey
and the update status was returning:

{
  "status": "failed",
  "updateId": 3,
  "type": {
    "name": "DocumentsAddition",
    "number": 1
  },
  "error": "serializer error; Primary key is missing.",
  "duration": 9.817e-6,
  "enqueuedAt": "2020-08-29T17:21:00.098580201Z",
  "processedAt": "2020-08-29T17:21:00.100495474Z"
}

So, to be honest I don't understand why you don't get a string back on $this->_id @elfeffe

[elfeffe]

@shokme I don’t know. I will resume my work with search in a few days and I will let you know if I find something new.
The SDK should create the index automatically as Algolia does, with the correct I’d, to avoid mistakes.

[curquiza]

The SDK should create the index automatically as Algolia does, with the correct I’d, to avoid mistakes.

It is planned in the MeiliSearch core-engine! 🙂
meilisearch/meilisearch#918