Allow getting many documents by id

[ghost]

After reading around I think it's only possible to get one document by it's unique id. It would be nice if we could give an array of ids and get the result back in a single request. Right now I am making a request for each id.

Thanks.

[paligiannis]

I need to address this issue too. Any ideas of implementing that?

[gmourier]

Hello @san-serif and @paligiannis! Thank you for using MeiliSearch and taking the time to contribute to this project. Currently, we are putting a lot of effort into the release of the new engine but it may be something that we want to address in the future. Can you please tell me more about your needs regarding this feature? Thanks!

[paligiannis]

@gmourier It seems like we need something to fetch specific records from the database. Usually, the use case is something like getting ids from database like products[ids]=100,300,400,500. So it's something like filtering specific results and them use them with facets of any other implementation of Meilisearch.

[ghost]

Hey @gmourier! This is kind of low priority for me as MeiliSearch is already working great for me. I want this feature because I want to get some data without directly querying my main database (mostly for backoffice stuff).

[paligiannis]

@san-serif How have you implemented such a thing?

[ghost]

@paligiannis

I'm using javascript and I essentially did it like this:

(async () => {
  const client = new MeiliSearch({ host: "https://endpoint" });
  const index = client.index("products");

  const ids = [100,300,400,500];

  const promises = ids.map(id => index.getDocument(id));

  const documents = await Promise.all(promises);

})();

[gmourier]

Hi! Thank you for your answers.

I mark the issue as a feature request to track it in our backlog. You can also propose this feature as an idea here. It allows people to vote for it and make it more visible in terms of importance.

[ghost]

@MarinPostma Just to clarify I think this is low priority because @gmourier mentioned you guys are working on a new engine, I did not want to give the impression that I think this is super important.

I don't know how to explain my use case better. I want to query data directly from MeiliSearch because I think it would be faster than my database.

[jiminy-billy-bob]

We solved this by declaring the id field as an attribute for faceting. Then we simply do something like this:

await meilisearch.getIndex('myIndex').search(undefined, {
  facetFilters: [documents.map(d => `id:${d.id}`)],
  limit: 999999
});

This should return documents with id foo and bar.

[tty2]

I see that this discussion is outdated but I'm curious if there is a convenient way to get documents by ids array using current engine?
I'm using golang api and can't find common way to do that.

[mikerogerz]

I was unable to find any optimal way via the API, so I did something like this (in PHP) to pull and update existing documents with new data:

$offset = 0;
$moreDocs = true;
do {
	$docs = collection($index->getDocuments([
		'limit' => 10000,
		'offset' => $offset,
		'attributesToRetrieve' => 'objectID,id,another_attr'
	]));
	if (!$docs->isEmpty()) {
		// match docs to update (based on id, etc), update attribute, and push updates to queue via API updateDocuments()
		
		$offset += 10000;
	} else {
		$moreDocs = false;
	}
} while ($moreDocs);

Not the most elegant solution, but it is/was the only reliable way I was able to match all documents, especially when indexing long-form content (multiple docs sharing the same id, but with different chunks of data to match on), so pulling individual documents wouldn't work either. Being able to pull by array would be extremely useful, in my opinion.. to grab a few hundred/thousand at a time.

[tacman]

Although this is an older post, I'm surprised it hasn't been solved yet. Getting a single document by primary key is simple, it shouldn't be hard to get a set of documents by an array of primary keys. Am I missing something?

[derekperkins]

This has been solved now

• Get Documents by Batch #644 (comment)

[tacman]

#644 is a solution that works for small sets, by duplicating the ID as a facet. I'm hoping, though, that the ability to fetch a batch of documents by ID without needing to setup a filter is offered in some future version of meilisearch. Like others who have posted in #644, the ability to delete a batch by id already exists, so fetching by id seems natural.

[irevoire]

Thanks, do you know what kind of API you want to see?

[tacman]

I think expanding on /documents (GET) and /fetch (POST) would be seamless.

curl -X GET 'http://localhost:7700/indexes/dogs/documents?limit=1&filter=doggo=bernese'

curl -X GET 'http://localhost:7700/indexes/dogs/documents?ids=cody,finn,brandy,gambit'

curl -X POST http://localhost:7700/indexes/dogs/documents/fetch
-H 'Content-Type: application/json'
--data-binary '{ "limit": 1, "ids": ["cody","finn"] }'

That way, it could be combined with filter and search, which would rock.

// psuedo-code

$ids = $this->getStudentIds(['class' => 'civics101']); // some database query
$index->getDocuments(['ids'=>$ids, 'filter'=> ['gender'=>'nonbinary']);

[irevoire]

Oh, interesting I was wondering if the filter and ids should be excluding each other or could be used together.
I’ll send your proposition to the team; thanks for your time 🔥

[tacman]

basically, everywhere you can send in a filter, you should be able to send in a list of ids, since really the primary key is a property but because it's unique, it is handled differently than a filter. But it's fundamentally just a way of accessing the data.

[tacman]

Here's an example of why this is important. I have two indexes, movies and actors. Both have a million rows (which meili, because it's awesome, handles without a sweat).

When I look up a movie, I want to get the actors and some data about them, but that data depends on the application. One application may want birthday, another nationality or languages. I don't want to store all the actor data in the movie database, I simply want an array of actors by code or imdb id or whatever.

So I get my movie results back and aggregate actor list so I have a single set of all the actors referenced from the movie result. I now want to fetch the documents in the actor index, by id, get back all the data (or the data my application specifies), merge it with my result set and send it off to the next step. Obviously fetching the actor data by id may need to be done in batches.

I don't want to iterate through the list of hundreds of actors one at a time (loading the document by id), I want to load all the actor data as efficiently as possible.

Facets are a terrible approach. They're not designed for millions of individual ids -- that's the role of the ID!

In a relational database, this would be a simple inner join between the movie and actor tables. In meili, it's a simple 2-index lookup, but is impossible without the ability to lookup a batch of documents by id.

The meili developers have created a blazingly fast lookup for searching, I imagine it would be very simple for them to bypass the "hard" searching and simple return a collection of documents based on the id.

My actual application is more complicated than this, but I hope this example is clear enough to warrant prioritizing batch lookup by id in the next release. Thank you for your consideration.

[tacman]

Here's an example of why this is important. I have two indexes, movies and actors. Both have a million rows (which meili, because it's awesome, handles without a sweat).

When I look up a movie, I want to get the actors and some data about them, but that data depends on the application. One application may want birthday, another nationality or languages. I don't want to store all the actor data in the movie database, I simply want an array of actors by code or imdb id or whatever.

So I get my movie results back and aggregate actor list so I have a single set of all the actors referenced from the movie result. I now want to fetch the documents in the actor index, by id, get back all the data (or the data my application specifies), merge it with my result set and send it off to the next step. Obviously fetching the actor data by id may need to be done in batches.

I don't want to iterate through the list of hundreds of actors one at a time (loading the document by id), I want to load all the actor data as efficiently as possible.

Facets are a terrible approach. They're not designed for millions of individual ids -- that's the role of the ID!

In a relational database, this would be a simple inner join between the movie and actor tables. In meili, it's a simple 2-index lookup, but is impossible without the ability to lookup a batch of documents by id.

The meili developers have created a blazingly fast lookup for searching, I imagine it would be very simple for them to bypass the "hard" searching and simple return a collection of documents based on the id.

My actual application is more complicated than this, but I hope this example is clear enough to warrant prioritizing batch lookup by id in the next release. Thank you for your consideration.

[macraig]

Hi @tacman , apologies for the late reply. We currently rely on facets as our approach, though we recognize it might not be the ideal solution for every use case. Other than the preference to not use facets from a theoretical standpoint, are there any examples where this blocking your use case?

[tacman]

Suppose there were a special exemption for searching by the primary key? That is, if the primary key is imdb_id, and the /fetch endpoint requests imdb_id=1,2,3, instead of throwing an error and saying "hey, you need to set imdb_id as filterable" it say "this is the primary key, I already have the index, this doesn't need to be set as filterable".

That would be natural for developers and wouldn't require any additional tweaks to the endpoint.

And those of us using the "set the id to filterable" technique would simply need to remove that.

There's no value to setting a facet / filterable property to something that has millions of unique ids. But we need a way to get batch documents by id. Again, thanks for your consideration.

[tacman]

It's a terrible solution. Facets don't make sense for unique values. ANYTHING where you need to access the data in meili by id. In my case, I have related data as a list of keys, so I fetch the data first, then I need to fetch all the records with those ids. Meili is a search engine, but also a datastore, and it's shocking that I can't get multiple records by id. Facets are designed for millions of unique records, but meili is.

…

On Wed, Jan 24, 2024 at 11:09 AM Maria Craig ***@***.***> wrote: Hi @tacman <https://github.com/tacman> , apologies for the late reply. We currently rely on facets as our approach, though we recognize it might not be the ideal solution for every use case. Other than the preference to not use facets from a theoretical standpoint, are there any examples where this blocking your use case? — Reply to this email directly, view it on GitHub <#99 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEXIQIHRDW5QDMMDVFJIUDYQFMANAVCNFSM5E734W52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQMRTGY3TIMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[macraig]

Thank you for sharing your candid feedback with us. I understand this approach is not the most efficient for scenarios like yours, and we know we always have room for optimization and improvement.
While we'd love to address all the requests we get, we only have a limited amount of people and time to work on them. The use of facets, as it stands, is our best available solution for the time being. We value your feedback, and I assure you that we are continuously working to enhance our capabilities. I appreciate your patience and understanding as we strive to improve Meilisearch.