fix: convert data tables to cursor-based pagination

[larryro]

Summary

• Add infiniteScroll prop to DataTable component for cursor-based pagination with "Load more" buttons
• Create cursor-based pagination APIs for documents, products, executions, and automations in Convex
• Migrate all data tables (customers, vendors, websites, documents, products, executions, automations) from offset-based to cursor-based pagination
• Remove deprecated offset-based pagination APIs that are no longer used

This prevents Convex's "Too many bytes read" (16MB limit) error for organizations with large data volumes.

Test plan

• Verify all tables load initial data correctly
• Click "Load more" button and verify additional rows appear
• Test with filters applied to ensure pagination works with filtered data
• Verify no console errors related to pagination
• Test tables with large datasets to confirm no "Too many bytes read" errors

Closes #124

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added infinite scroll "Load more" functionality across data tables for customers, documents, products, vendors, websites, automations, and executions.

• UI Changes

  • Replaced traditional page-based pagination with cursor-based infinite scrolling navigation.
  • Added "Load more" button instead of page number controls.
  • Removed sorting controls from select data tables.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request migrates multiple dashboard tables from offset-based pagination to cursor-based pagination. The changes include backend API refactoring to use early termination (avoiding full dataset reads), frontend component updates to use the useCursorPaginatedQuery hook with infinite scroll UI, and removal of sorting capabilities from paginated views. Affected modules include customers, documents, products, vendors, websites, automations, and workflow executions. New cursor-based query implementations are added to Convex backend models, along with updates to preloaded data types and component props.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

• optimize chat agent #5 — Modifies customers API surface and updates customers-table preloaded types alignment with cursor-based getCustomers endpoint.
• support pptx,docx and reduce envs #8 — Updates Convex documents API and server-side document utilities with cursor-based pagination implementation.
• tale-project/poc2#400 — Extends executions-table component and refines the Execution interface used in cursor-based execution listing.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 18

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

services/platform/app/(app)/dashboard/[id]/(knowledge)/websites/components/websites-table.tsx (1)

83-109: Consider providing type parameter to the hook to avoid the cast.

The cast websites as Doc<'websites'>[] on line 86 works but could potentially be avoided if useCursorPaginatedQuery accepts a type parameter for the item type. This would provide end-to-end type safety.

♻️ Suggested improvement

If the hook supports generic typing:

-  const { data: websites, isLoadingMore, hasMore, loadMore } = useCursorPaginatedQuery({
+  const { data: websites, isLoadingMore, hasMore, loadMore } = useCursorPaginatedQuery<
+    typeof api.websites.getWebsites,
+    Doc<'websites'>
+  >({
     query: api.websites.getWebsites,
     ...
   });

Then you could use websites directly without casting.

services/platform/app/(app)/dashboard/[id]/automations/[amId]/executions/components/executions-table.tsx (1)

41-54: Consider whether this interface export is necessary.

The Execution interface is exported but may only be used internally within this component. If it's not consumed by other modules, prefer keeping it non-exported to reduce the public API surface.

♻️ Make interface private if unused externally

-export interface Execution {
+interface Execution {

🤖 Fix all issues with AI agents

In
@services/platform/app/(app)/dashboard/[id]/(knowledge)/customers/components/customers-table.tsx:
- Around line 95-110: The status cast in queryArgs is unsafe because it strips
the "lost" option; remove the hard cast and either (A) widen the outgoing type
to include "lost" (update the API/types to accept 'lost') or (B) perform
runtime-narrowing before assigning status: derive a new status array from
filterValues.status by filtering only the allowed values (e.g.,
['active','churned','potential']) and pass that filtered array (or undefined if
empty) into queryArgs; update references to queryArgs and filterValues
accordingly.

In
@services/platform/app/(app)/dashboard/[id]/(knowledge)/documents/components/document-table.tsx:
- Around line 275-278: The component currently returns null when isLoading is
true which causes a flash; remove the early "return null" and always render the
DataTable (or table markup) so previous data stays visible, and indicate loading
via a visual overlay or reduced opacity; specifically, keep rendering the
DataTable component instead of returning null, pass an isLoading/loading prop to
DataTable if available, or wrap the table body with a semi-transparent style +
spinner when isLoading is true, and only rely on the Suspense boundary for the
initial load (detect via presence of previousData/rows or an isInitialLoading
flag) so filter changes show the existing table with a loading indicator rather
than blank space.

In @services/platform/app/(app)/dashboard/[id]/(knowledge)/products/page.tsx:
- Around line 47-68: The call to parseSearchParams passing { defaultSort:
'lastUpdated', defaultDesc: true } is misleading because the cursor-based API
(used by preloadQuery/api.products.getProductsCursor) doesn't support sorting;
remove the defaultSort and defaultDesc properties from the parseSearchParams
invocation (the call that destructures { filters, pagination } from
parseSearchParams(rawSearchParams, productFilterDefinitions, ...)) so the
function is invoked only with rawSearchParams and productFilterDefinitions (and
any remaining valid options), leaving filter/pagination behavior intact.
- Around line 62-65: The status filter UI allows multi-select but the backend
only accepts one value; in both page.tsx and product-table.tsx replace the
current check that sets status only when filters.status.length === 1 with logic
that, when filters.status.length > 0, sends filters.status[0] (first selected
value) to the backend and, when filters.status.length > 1, emit a warning for
the user (or at minimum console.warn) indicating multiple selections are not
supported; update the status assignment that references filters.status to use
the first element and add a user-facing or dev-facing warning so the behavior is
explicit.

In
@services/platform/app/(app)/dashboard/[id]/(knowledge)/vendors/components/vendors-table.tsx:
- Around line 41-50: The useMemo that builds queryArgs currently depends on the
whole filterValues object which can change identity and trigger unnecessary
pagination resets; update the dependency array of the useMemo for queryArgs to
depend only on the specific fields used (organizationId, filterValues.query,
filterValues.source, and filterValues.locale) so the memo only recalculates when
those values change; keep the same shape for queryArgs (including the
conditional conversion of empty arrays/strings to undefined) but reference the
individual properties in the deps instead of filterValues.

In @services/platform/app/(app)/dashboard/[id]/(knowledge)/vendors/page.tsx:
- Around line 56-59: Extract the hardcoded page size from the paginationOpts
(numItems: 20) into a shared constant (e.g., DEFAULT_CURSOR_PAGE_SIZE) and
import it where needed; update the paginationOpts object in the page component
(paginationOpts) to use the constant instead of the literal 20 so other pages
(documents, products, customers) can reuse the same value for consistent
pagination.

In @services/platform/app/(app)/dashboard/[id]/(knowledge)/websites/page.tsx:
- Around line 53-65: The getProductsCursor function in
services/platform/convex/products.ts currently takes flat args (numItems,
cursor); refactor it to accept a single paginationOpts object { numItems, cursor
} to match the paginationOpts pattern used by
getWebsites/getThreads/queryProducts and use the existing
paginationOptsValidator inside the Convex function; then update all callers of
getProductsCursor to pass paginationOpts instead of separate args (and adjust
any type/signature imports) so the API signature is consistent across pagination
endpoints.

In
@services/platform/app/(app)/dashboard/[id]/automations/[amId]/executions/components/executions-table.tsx:
- Line 396: The explicit cast `executions as Execution[]` indicates a type
mismatch between the data returned by useCursorPaginatedQuery and the local
Execution interface; fix by aligning types: import the Execution (or equivalent)
type from the API layer and use it as the generic for useCursorPaginatedQuery
(or replace the local Execution interface with the imported type) so
`executions` is already typed as the correct array, or if the local interface
must differ, update the query's mapping to produce the local shape and add a
short code comment documenting why a cast is necessary; refer to the symbols
`executions`, `Execution`, and `useCursorPaginatedQuery` when making the change.

In
@services/platform/app/(app)/dashboard/[id]/automations/components/automations-table.tsx:
- Around line 62-65: The inline transformArgs arrow creates a new function each
render and defeats memoization in useCursorPaginatedQuery; wrap transformArgs in
React.useCallback to return the same function reference across renders (e.g.,
const transformArgs = useCallback((baseArgs, cursor, numItems) => ({
...baseArgs, paginationOpts: { cursor, numItems } }), [])) and ensure
React.useCallback is imported where transformArgs is defined so
useCursorPaginatedQuery receives a stable callback.

In @services/platform/components/ui/data-table/data-table.tsx:
- Around line 414-428: The Button usage currently renders a manual Loader2 +
conditional text; replace that by passing
isLoading={infiniteScroll.isLoadingMore} and keep
onClick={infiniteScroll.onLoadMore} and aria-label={t('pagination.loadMore')}
(remove the manual conditional JSX and the disabled prop since isLoading handles
it), and then remove the now-unused Loader2 import if it isn’t used elsewhere in
this file.

In @services/platform/convex/documents.ts:
- Around line 593-609: The args for getDocumentsCursor inline numItems and
cursor instead of using the existing cursorPaginationOptsValidator; change the
args to include/compose cursorPaginationOptsValidator (use it in place of the
inline numItems/cursor definitions) and ensure the handler passes the expected
shape to DocumentsModel.getDocumentsCursor—either update
DocumentsModel.getDocumentsCursor signature to accept the pagination object or
add a small wrapper to map from the validator-shaped args to the model's flat
params; reference getDocumentsCursor, cursorPaginationOptsValidator, and
DocumentsModel.getDocumentsCursor when making the change.

In @services/platform/convex/model/documents/get_documents_cursor.ts:
- Around line 44-67: In the filter function (filter: (doc: Doc<'documents'>) =>
boolean) avoid repeating casts by extracting const metadata = doc.metadata as
DocumentMetadata | undefined at the top of the function and then use
metadata?.storagePath and metadata?.name in the folderPath and searchQuery
checks; update title/name checks to use metadata where appropriate and remove
duplicate casts to simplify and clarify the logic.

In @services/platform/convex/model/products/types.ts:
- Around line 41-47: The current file selectively re-exports
GetProductsCursorArgs, ProductCursorItem, and CursorPaginatedProductsResult from
get_products_cursor; to follow the project's guideline preferring wildcard
re-exports, replace the selective export statement with a single wildcard type
re-export (export type * from './get_products_cursor') only if
get_products_cursor does not export extra internal types you don’t want
exposed—otherwise keep the explicit names.

In @services/platform/convex/model/wf_definitions/get_automations_cursor.ts:
- Around line 52-53: The code is performing an unnecessary type assertion by
doing "const workflow = wf as WorkflowDefinition" inside the for-await loop even
though the query already yields Doc<'wfDefinitions'> (equivalent to
WorkflowDefinition); remove the cast and use the original iterator variable
(e.g., "const workflow = wf" or rename the loop variable to "workflow") so the
code relies on the existing type from "query" (referencing the for-await loop
variable "wf", the "query" iterator, and the
"WorkflowDefinition/Doc<'wfDefinitions'>" types).

In @services/platform/convex/model/wf_executions/validators.ts:
- Around line 84-93: The validator listExecutionsCursorArgsValidator is using
flat pagination fields (numItems, cursor) which is inconsistent with the
documented nested pattern; replace those two fields with paginationOpts:
paginationOptsValidator while keeping the rest of the properties
(wfDefinitionId, searchTerm, status, triggeredBy, dateFrom, dateTo), and then
update all call sites that consume this payload (notably listExecutionsCursor
and getProductsCursor) to read paginationOpts.numItems and paginationOpts.cursor
(or accept the nested object) so behavior matches getCustomers/getWebsites and
existing paginationOptsValidator expectations; also update any tests or type
annotations that expect the flat shape.

In @services/platform/hooks/use-cursor-paginated-query.ts:
- Around line 94-103: The memo for queryArgs includes transformArgs in its
dependency list which will cause recomputation when callers pass an unstable
inline function; add a brief JSDoc above the useCursorPaginatedQuery hook (or
the transformArgs parameter) advising callers to wrap transformArgs with
useCallback for reference stability, and optionally add an internal note in the
file near the queryArgs useMemo suggesting callers provide stable callbacks;
keep the existing dependency array as-is (do not change logic here).