useLiveInfiniteQuery

[KyleAMathews]

We want an API that's nearly identical to TanStack Query's useInfiniteQuery except powered by DB's live queries. It'll take advantage of #612 (if the collection is using that) to load pages on demand.

[KyleAMathews]

Implementation Plan for useLiveInfiniteQuery

We've designed the implementation approach for useLiveInfiniteQuery. The hook will provide a TanStack Query-style infinite scroll API powered by DB's live queries, with a phased rollout that aligns with #612.

Implementation Approach

Single Dynamic Query Design

We'll use a single live query that adjusts its limit as pages are fetched:

function useLiveInfiniteQuery(queryFn, config) {
  const [loadedPages, setLoadedPages] = useState(1)
  const pageSize = config.pageSize || 20

  // Single live query with increasing limit
  const { data, collection, ...rest } = useLiveQuery(
    (q) => {
      const query = queryFn(q)
      return query.limit(loadedPages * pageSize)
    },
    [loadedPages]
  )

  // Split data into pages - rebuild this array each time
  const pages = useMemo(() => {
    const result = []
    for (let i = 0; i < loadedPages; i++) {
      result.push(data.slice(i * pageSize, (i + 1) * pageSize))
    }
    return result
  }, [data, loadedPages, pageSize])

  const fetchNextPage = () => {
    const lastPage = pages[pages.length - 1]
    const nextParam = config.getNextPageParam(lastPage, data)

    if (nextParam !== undefined) {
      setLoadedPages(prev => prev + 1)
    }
  }

  return {
    data,           // Flattened array of all items
    pages,          // Array of page arrays
    fetchNextPage,
    hasNextPage: pages[pages.length - 1]?.length === pageSize,
    ...rest
  }
}

Why this approach:

• Single subscription (efficient)
• Simple state management
• All data in one live collection with automatic updates
• Natural fit for offset-based pagination

Phased Implementation

Phase 1: Basic Infinite Query (Initial Release)

Scope: Paginate through collection's current dataset

• Implement single-query approach with increasing limit
• Support fetchNextPage() and hasNextPage
• Split data into pages array
• Works entirely within collection's loaded data
• Query recreates when limit changes (acceptable for now)

Limitation: When you reach the end of the collection's data, there's no more to fetch.

Phase 2: Query Optimization (Future)

Depends on: Query builder enhancements

• Add dynamic limit adjustment without query recreation
• Incremental "fetch N more items" from collection
• Much better performance for frequent page loads

Phase 3: Backend Integration (Future)

Depends on: #612 (Query-driven partitioned sync)

Once #612 is implemented, collections will be able to accept query predicates and load data on demand:

• useLiveInfiniteQuery emits "need more data" events to collections
• Each collection type handles these events differently:
  • Query Collection: Uses its query function to fetch next page (cursor or offset-based pagination)
  • Electric Collection: Requests additional snapshots
  • Custom Collections: Implements custom fetching logic
• The hook is agnostic - it just communicates "I need more data" and collections decide how to get it

This transforms the hook from "paginating through local data" to "infinite scroll with backend fetching."

API Design

Proposed Signature

function useLiveInfiniteQuery<TContext extends Context>(
  config: {
    query: (q: InitialQueryBuilder) => QueryBuilder<TContext>
    initialPageParam?: number
    getNextPageParam: (lastPage: InferResultType<TContext>, allData: InferResultType<TContext>) => number | undefined
    pageSize?: number
    gcTime?: number
  },
  deps?: Array<unknown>
): UseLiveInfiniteQueryReturn<TContext>

// Also support function form like useLiveQuery
function useLiveInfiniteQuery<TContext extends Context>(
  queryFn: (q: InitialQueryBuilder) => QueryBuilder<TContext>,
  config: {
    initialPageParam?: number
    getNextPageParam: (lastPage: InferResultType<TContext>, allData: InferResultType<TContext>) => number | undefined
    pageSize?: number
  },
  deps?: Array<unknown>
): UseLiveInfiniteQueryReturn<TContext>

Return Type

type UseLiveInfiniteQueryReturn<TContext> = {
  // Data
  data: InferResultType<TContext>           // Flattened array of all items
  pages: Array<InferResultType<TContext>>   // Array of pages
  pageParams: Array<unknown>                 // Array of page params used

  // Pagination methods
  fetchNextPage: () => void
  fetchPreviousPage: () => void  // Phase 4

  // Pagination state
  hasNextPage: boolean
  hasPreviousPage: boolean
  isFetchingNextPage: boolean
  isFetchingPreviousPage: boolean

  // From useLiveQuery
  collection: Collection
  state: Map<string | number, GetResult<TContext>>
  status: CollectionStatus
  isLoading: boolean
  isReady: boolean
  isIdle: boolean
  isError: boolean
  isCleanedUp: boolean
}

Example Usage

// Simple offset-based infinite query
const { data, pages, fetchNextPage, hasNextPage } = useLiveInfiniteQuery(
  (q) => q
    .from({ posts: postsCollection })
    .orderBy(({ posts }) => posts.createdAt, 'desc')
    .select(({ posts }) => ({
      id: posts.id,
      title: posts.title,
      createdAt: posts.createdAt
    })),
  {
    pageSize: 20,
    getNextPageParam: (lastPage, allData) =>
      lastPage.length === 20 ? allData.length : undefined
  }
)

// Render
<div>
  {data.map(post => (
    <PostItem key={post.id} post={post} />
  ))}

  {hasNextPage && (
    <button onClick={fetchNextPage}>
      Load More
    </button>
  )}
</div>

Key Differences from TanStack Query

1. Live Updates: Data automatically updates when underlying collection changes (inserts/updates/deletes)
2. Data Boundary (Phase 1): Operates within collection's current dataset only
3. Backend Fetching (Phase 3): Will integrate with Query-driven sync #612 for on-demand loading
4. Query Recreation (Phase 1→2): Initially recreates query on limit change, will be optimized

Next Steps

1. Implement Phase 1 basic infinite query
2. Add tests for pagination, live updates, edge cases
3. Document usage patterns
4. Wait for Query-driven sync #612 to enable Phase 3 (backend integration)