New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Paging support? #23
Comments
I'm going to admit that I'm still wrapping my head around how the Because of that, I'm not completely sure that Do you have an example of what you're trying to do? |
What I'm doing right now (with the v3 SDK) is this:
This returns an |
For a bit more explanation, See: https://theburningmonk.com/2011/09/fsharp-yield-vs-yield/ |
Looking into the v3 SDK source and comparing it to v4, it looks like it works a bit differently. In v4 there isn't the Now, digging through this a bit further it turns out that this is really a wrapper over So, I probably would have to have |
Could you not just use OFFSET and LIMIT in your query "SQL" and remember what page index you're on? Or have I missed something here? Edit: I believe that's the official CosmosDB approach for paging. |
@seankearon I don't think these two types of paging are the same thing, though I may be wrong. The type I'm referring to is more akin to batching. The Cosmos SDK client won't return everything all at once. You have to continually call it to get the next batch of data. I'll take a look at what's in V4 when I get a chance. |
Yeah, I'm wondering where the difference is between those two. If an async stream is like using a drinking straw to drink from a lake. Then paging/batching is like using a bucket to drink from a lake. If I'm passing you the bucket to drink from, do you care whether I fill it all at once or in little steps using my drinking straw? Probably not - you just want the next bucket of water. I'm thinking that the way to fill up bucket n using the straw would be something like this:
But then, it's been a loooong day! :) |
I've finally had some time to come back and revisit this issue and work out if it's possible/viable to do pagination support. TL;DR: Use the OFFSET and LIMIT as @seankearon has suggested, I don't think I can put anything into the API to do it for you. Best I can do it have a way to get batched results per iteration of the We're going to dive through a rabbit hole now, so choose if you want to read on as I'm partially writing this down for my own sake. I'm going to trace through a bunch of the Azure.Cosmos code as it currently stands. When you execute a This type is then wrapped in
Our Now, if we trace through here, So, what's the difference between iterating over the I tested this against a large Cosmos store I have with the following code: async Task Main()
{
var client = new CosmosClient("...");
var container = client.GetDatabase("...").GetContainer("...");
var qd = new QueryDefinition("SELECT c.id FROM c");
var nonCount = 0;
"Non-paged query".Dump();
await foreach (var response in container.GetItemQueryIterator<Dictionary<string, string>>(qd))
{
nonCount++;
}
"Paged query".Dump();
var pageCount = 0;
await foreach (var response in container.GetItemQueryIterator<Dictionary<string, string>>(qd).AsPages())
{
pageCount++;
}
$"Non-Paged ran {nonCount} times to Paged {pageCount}".Dump();
} And here's the response:
The iteration count dropped and I ran a network trace on it, which saw the same number of network requests happening. I might look at putting in a |
Thanks for the detailed write-up. I'm waiting to use this on my project until the v4 Cosmos API SDK is fully released, but this looks great. |
I've added a "pagination" option in a new branch: https://github.com/aaronpowell/FSharp.CosmosDb/tree/pagination Basically all it does is adds a new method |
This will be coming in the next release. |
If anyone wants to test, grab the |
Available on NuGet now. |
This looks like a great new library that I'd love to use in place of my own hacked together wrapper. Are you expecting the caller to do the paging, or should that be included when returning a query result? I'm not sure if
ofAsyncEnum
does that or not, but the code doesn't seem to check.HasMoreResults
.The text was updated successfully, but these errors were encountered: