Index: primaryKey as an index setting

[eskombro]

Every index needs a primaryKey. It can be set at index creation or when you start indexing documents. It can also be infered from your first document. The primaryKey can only be set once, and can't be updated afterwards.

But that is the only specific behavior that makes it any different to any other index settings. Is there a reason to handle the Index primaryKey in a different way than all the other index settings? Also, shouldn't we add it to the GET /indexes/:index_uid/settings?

There are three weird behaviours for this choice:

• There is no dedicated route for getting or setting the Index primaryKey
• It is used as a parameter when creating or updating an index (and ignored if the primaryKey has already been set). It doesn't belong here, and the user is not notified about the fact that this parameter is being ignored.
• it is documented at two different places: here and here. Which makes me think that there is a complex choice behind it

[bidoubiwa]

It is documented at two different places mostly because it has connexion with both documents and indexes. Every document has the same primary key field that is stored in the index. The index has no primaryKey on creation, but once one has been set, it cannot be changed. So I made a connection between those two guides, but the one in the index guide links to documents.

One of the reason why it is was not added in the settings is because primaryKey is not updatable. Which, if we had added it in the setting would have made the use of the settings route also confusing.

The use of the POST /settings route wont work if you add primaryKey: "something" in the body and that a primary Key already exists.

Every subroute of the settings have a GET/POST/DELETE method. With the exception of the primaryKey if it was in the settings.

Because MeiliSearch is schemaless, we do not explicitly ask for a primaryKey on creation. But if no primaryKey can be infered during document addition, MeiliSearch wont be able to index the document. Which make it a required configuration in some specific situations.

Should a required configuration be in the settings? The settings are not a required knowledge to use and experience MeiliSearch. I do not realize how much friction this causes, but is it easier to ask a user to update the settings or to update the index?

The index and documents on the other hand must be understood by the user to be able to set up a MeiliSearch. So I think it is easier to introduce the primaryKey in the index, or during documentation addition, than in the settings.

Now, one thing that is possible, is that the primaryKey are badly explained in the documentation.
Maybe it just need a rewrite. I'm open for suggestions.

Do we know other users for whom this was an issue to understand?

[eskombro]

@bidoubiwa thanks for your answer!

When I mention that the primaryKey is documented in two different places, I don't mean that this is a documentation issue, but more like a way to show that the choice of what the primaryKey is and how it should be handled isn't necessarily as clear as for the other index settings. I am not against the fact that you either specify a primaryKey at index creation or not and let the schemaless architecture infer it. But I don't really agree with the fact that it becomes an optional parameter when you create/update documents, and specially if it can be ignored sometimes without notifying the user.

For example. Let's imagine I send a request to update documents: I can specify a primaryKey as a parameter. Let's say I send an invalid primaryKey (any key different to the one that was set already). MeiliSearch will apply the modifications to the documents anyway, send me a HTTP status of 200. How can I know as a user if the primaryKey was modified or not? I will discover that later, and will be lost trying to understand what happened. I don't think this is a transparent way of handling such a request. In my opinion, those two things should be separate. Of course if I have read the documentation, I can understand the problem, but I don't think this is intuitive, and it should be.

Which makes me think... I should have a specific route to get/set a primaryKey (even if it is not updatable after the first time I set it, I will just get an error if I try to update). And I shouldn't be specifying the primary key in a POST or PUT request to /indexes/:index_uid/documents

Adding it to the general settings route can indeed create some conflicts, which I still can't reflect on as clearly as I would like. But I'm glad this can start a discussion about it! :)

But if you try to update your settings by modifying the primaryKey, it seems normal to me that you get an error. And if you don't modify it while you try to update your settings, that shouldn't be an issue to handle it in MeiliSearch core (if the primaryKey you specified is the same than the one already set, you can ignore it)

[bidoubiwa]

But I don't really agree with the fact that it becomes an optional parameter when you create/update documents, and specially if it can be ignored sometimes without notifying the user.

This is a valid point. Having so many possibilities is more confusing than helping. Also the fact that it will not raise an error if it is ignored during document addition but will raise an error if you try to update the primary key in the index when it already exists make the behavior not consistent.

I would suggest also to remove the query parameter from the document addition route. If this is what you're suggesting.

However, I still prefer the update of the primary key being in the index routes and not the settings route.

For a reminder, this is the behavior on the index route:

On index creation

$ curl \
  -X POST 'http://localhost:7700/indexes' \
  --data '{
    "uid": "movies",
    "primaryKey": "movie_id"
  }'

primaryKey is optional in this route.

Index update (which will return an error if a primary key already exists)

$ curl \
  -X PUT 'http://localhost:7700/indexes/movie_review' \
  --data '{
      "primaryKey" : "movie_review_id"
  }'

Returns:

• SUCCESS: if the primary key was at null before
• ERROR: a primary key already existed (even if it was a different one)

This behavior already describes what you're are suggesting in the settings route. I'm not sure I understand what you dislike about this?

[curquiza]

(sorry for the delay)

I completely agree with @bidoubiwa on both points:

1. the query parameter in addition routes (POST and PUT) does not bring any useful usage since the primary-key is updatable with PUT /indexes/:uid. Plus, it leads to confusion because if the primary-key is already set, the params is ignored without any warning.

2. I would rather the update of the primary key be in the index routes and not the settings route. The primary-key in settable during index creation. Why this setting would be settable during index creation and not the others? Not really consistent from my POV.
   But anyway, let's supposed we considered the primaryKey as a setting (with GET and POST routes) and not as an index property anymore, so we remove the possibility to set it during index creation/update:

• it would be the only setting that you can only set once, which is a little bit confusing, but not a big deal either from my POV.
• we want to stay as simple as possible and we want the user to avoid the settings for their first tries with MS. But to set the primaryKey is really often needed with the user's datasets. And testing with their own datasets is what the users do really quickly. That would lead most of the users to deal with the settings for their first steps with MS. Letting the possibility to do it during index creation is more convenient.
• because you can only set it once, it would need that MeiliSearch supports warnings (this might be available one day 🚀). If you try to update all your settings (with the general route GET /indexes/:uid/settings) and you let the primaryKey in the settings you don't want your update of settings to be stopped by a MS error because you let the primaryKey. You need a warning.
  This situation can happen when you are coding with an SDK, using the get_or_create_index method, and then you update the settings in your script. If the index was already created your script would fail. I use this kind of script all the time to populate my local MS with different datasets depending on what I'm working on.

[simar7]

Slightly tangential to this discussion here, but I'm curious why can the primaryKey not have any spaces in it?

[curquiza]

Slightly tangential to this discussion here, but I'm curious why can the primaryKey not have any spaces in it?

Hello @simar7

If you are talking about the primaryKey value (ex: "id": "foo and bar", "foo and bar" is the value), this value is used in URL routes, so MeiliSearch only allows "simple" characters to avoid encoding issues

[VororRich]

This however prevents data where the primary key is an IRI for example however (e.g. OWL Entities). I also note that if you DO put in an IRI as a primary key, no error is produced, a "202 Accepted" is received (with a valid updateId) but the document isnt saved.

POST http://3.10.169.133/indexes
X-Meili-API-Key: xxx
Content-Type: application/json

{
  "uid": "concepts",
  "primaryKey": "iri"
}

POST http://3.10.169.133/indexes/concepts/documents
X-Meili-API-Key: xxx
Content-Type: application/json

[
  {
    "iri": "http://endhealth.info/im#894281000252100",
    "name": "Scoliosis",
    "code": "123456",
    "Scheme": "READ2"
  }
]

[gmourier]

Hi @ErgoSoftUK!

It's basically because the validation is done asynchronously. You can find the error within the update generated by this call via the endpoint indexes/concepts/updates/:updateIdGivenInThe202

Concerning having a document identifier that cannot contain chars different than a-z A-Z 0-9 - _, one possible workaround is to store iri as it is in a field that is not marked as the primary key and compute some sort of a hash representing that fully qualified iri for the document identifier value.

However, I am aware that this is clearly not the most straightforward solution and we may want to seek a better solution.