`filter` helper when used in conjunction with pagination and a simple text filter misses results

[willemclarke]

In this discord thread (https://discord.com/channels/1019350475847499849/1385358143437017178) I surfaced a problem I was having whereby using the filter helper on a query thats paginated will miss results when doing a simple string filter.

Given a query like this:

export const paginate = query({
  args: {
    paginationOpts: paginationOptsValidator,
    searchTerm: v.optional(v.string()),
  },
  handler: async (ctx, args) => {
    const term = args.searchTerm?.toLowerCase();

    return filter(ctx.db.query("users"), (user) => {
      return term ? user.name.toLowerCase().includes(term) : true;
    })
      .order("asc")
      .paginate(args.paginationOpts);
  },
});

And a callsite like this:

 export default function Home() {
  const [searchTerm, setSearchTerm] = useState("");

  const term = searchTerm === "" ? undefined : searchTerm;

  const { results: users, isLoading } = usePaginatedQuery(
    api.queries.paginate,
    {
      searchTerm: term,
    },
    { initialNumItems: 10 }
  );

...
}

Versions:

 "dependencies": {
    "convex": "^1.29.3",
    "convex-helpers": "^0.1.106"
  }

When I search for a user thats not on the first page, it never seems to find a result. I have seeded 50 users, showing 10 for the initial page size. If I search for anyone thats not on the first page, but is definitely in the data, I can never find them.

Reproduction repo:
https://github.com/willemclarke/convex-filters-pagination-reproduction

Steps:

• git clone
• bun install or npm install
• npx convex dev
• npx convex import --table users sampleUsers.jsonl
• bun dev or npm run dev
• Search for "Kevin White" or anyone below "Julia Jackson" in the sampleUsers.jsonl file -> it should not find them.

Now I really want to apologise if I'm blatantly mis-configuring anything here, but I essentially took this code from https://stack.convex.dev/complex-filters-in-convex#escape-hatch-pagination. Any help is appreciated.

[ianmacartney]

Thanks for the writeup!
What you're seeing is that it's not finding any results in the first page. But if you call loadMore it does show up.

diff --git i/app/page.tsx w/app/page.tsx
index cfa047c..dd94e19 100644
--- i/app/page.tsx
+++ w/app/page.tsx
@@ -9,7 +9,11 @@ export default function Home() {
 
   const term = searchTerm === "" ? undefined : searchTerm;
 
-  const { results: users, isLoading } = usePaginatedQuery(
+  const {
+    results: users,
+    isLoading,
+    loadMore,
+  } = usePaginatedQuery(
     api.queries.paginate,
     {
       searchTerm: term,
@@ -61,6 +65,8 @@ export default function Home() {
             )}
           </div>
         )}
+
+        <button onClick={() => loadMore(10)}>Load More</button>
       </div>
     </div>
   );

Ultimately the query won't do an unbounded amount of reading, so if you have a selective filter you need to fetch more pages until you have enough results.

The post you reference calls this out:

This can return small or empty pages to the client, which can slow down load times and cost database bandwidth as it traverses the entire table across many pages.

So the solution here is to keep calling loadMore on the client until you have enough or the status is Exhausted - or to load it all on the client and do the filtering there.

[willemclarke]

Thank you for the answer. I'd argue that most people using this in a paginated context aren't expecting to load all results or load up until the record you want to filter for is visible though. Why paginate at all then? If I had a user I wanted to search for on the 100th page, I would have to scroll down 100 pages and then do the filter. I think most people reaching for pagination are by default not wanting to do sorting/filtering on the client.

I know what I'm talking about is doable with a search index and such, which is what I'm actually doing, but this behaviour was still a surprise to me and makes me wonder why you would ever use the filter helper in a paginated scenario.

[ianmacartney]

Each query should only read so many rows. If the filter took 100 pages to find the first result, it would exceed the per-transaction database limits. I agree you shouldn't require the user to scroll. You can call the function from the client programmatically until you have enough rows. We could make a more ergonomic paginated query hook for these sorts of situations, but ultimately if you can't filter how many rows you're scanning with an index, you'll need more queries and will consume more bandwidth as a result. You could use text search which is already case insensitive, then filter from there perhaps. And there's also options to pass in limits on how many rows to scan/ bandwidth to use, but those might not be plumbed through to the filter helper yet. PR welcome for that!

…

On Tue, Nov 25, 2025 at 1:37 PM Willem Clarke ***@***.***> wrote: *willemclarke* left a comment (get-convex/convex-helpers#864) <#864 (comment)> Thank you for the answer. I'd argue that most people using this in a paginated context aren't expecting to load all results or load up until the record you want to filter for is visible though. Why paginate at all then? If I had a user I wanted to search for on the 100th page? I would have to scroll down 100 pages and then do the filter. — Reply to this email directly, view it on GitHub <#864 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACZQW7J6ERHB3CYU55F43L36TDYLAVCNFSM6AAAAACNDKHSTWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKNZXGY4DGNZXHE> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>