Persistence implementation for pagination in some requests #1838

dimas-b · 2025-06-09T18:27:02Z

Following up on #1555

Refactor pagination code to delineate API-level page tokens and
internal "pointers to data"
Requests deal with the "previous" token, user-provided page size
(optional) and the previous request's page size.
Concentrate the logic of combining page size requests
and previous tokens in PageTokenUtil
PageToken subclasses are no longer necessary. EntityIdPaging handles
pagination over ordered result sets with static helper methods.

snazy

Overall the approach LGTM.

Some places however could be simplified and clarified though.

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java

snazy · 2025-06-10T11:04:46Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java

+   * @param requestedPageSize optional page size for the next page. If not set, the page size of the
+   *     previus page (encoded in the previous page token) will be reused.
+   */
+  public static PageRequest nextPage(


The function name and phrase 'from the previous API-level...are a bit confusing: what's "next"? What's "previous"? Why's there no "current"? All it does is constructing aPageRequest` from the two pagination-parameters.

Why not call this function constructFromParameters?

refactored to PageToken.decode()... Is it better?

snazy · 2025-06-10T11:06:34Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java

+  }
+
+  /** Represents a request to start paginating with a particular page size. */
+  public static PageRequest firstPage(int limit) {


What's the general use case of this one? It's only used in some drop*() implementation.

This is now PageToken.fromLimit() (same name as before).

The use case is for handling initial paged requests where a size is provided, but a token in not. For API-based calls that is taken care of by the decode() method. This method is a convenience entry point for internal calls.

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java

...ris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java

polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageTokenUtil.java

dimas-b · 2025-06-10T21:49:55Z

Squashed, rebased, resolved conflicts and addressed some feedback. Please review again.

Following up on apache#1555 * Refactor pagination code to delineate API-level page tokens and internal "pointers to data" * Requests deal with the "previous" token, user-provided page size (optional) and the previous request's page size. * Concentrate the logic of combining page size requests and previous tokens in PageTokenUtil * PageToken subclasses are no longer necessary. EntityIdPaging handles pagination over ordered result sets with static helper methods. Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>

eric-maynard · 2025-06-11T03:27:07Z

.../java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkStore.java

+    if (pageToken.paginationRequested()) {
+      hql += " order by m.id asc";
+
+      if (pageToken.hasDataReference()) {


This is still polymorphism, we're just not taking advantage of the fact that this is built into the language already:

if (pageToken isinstanceof EntityIdPageToken e) { tokenId = e.getEntityId(); }

Polymorphism at the java type level is not necessary in this case. Users of the "data pointer" know how to parse it based on the API method called. It is not a matter of providing a sub-type, but about understanding the format.

Currently parsing is achieved via static methods in EntityIdPaging.

…nceImpl

github-actions · 2025-07-14T02:13:15Z

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Based on apache#1838, following up on apache#1555 * Allows multiple implementations of `Token` referencing the "next page", encapsulated in `PageToken`. No changes to `polaris-core` needed to add custom `Token` implementations. * Extensible to (later) support (cryptographic) signatures to prevent tampered page-token * Refactor pagination code to delineate API-level page tokens and internal "pointers to data" * Requests deal with the "previous" token, user-provided page size (optional) and the previous request's page size. * Concentrate the logic of combining page size requests and previous tokens in `PageTokenUtil` * `PageToken` subclasses are no longer necessary. * Serialzation of `PageToken` uses Jackson serialization (smile format) Since no (metastore level) implementation handling pagination existed before, no backwards compatibility is needed. Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>

Based on #1838, following up on #1555 * Allows multiple implementations of `Token` referencing the "next page", encapsulated in `PageToken`. No changes to `polaris-core` needed to add custom `Token` implementations. * Extensible to (later) support (cryptographic) signatures to prevent tampered page-token * Refactor pagination code to delineate API-level page tokens and internal "pointers to data" * Requests deal with the "previous" token, user-provided page size (optional) and the previous request's page size. * Concentrate the logic of combining page size requests and previous tokens in `PageTokenUtil` * `PageToken` subclasses are no longer necessary. * Serialzation of `PageToken` uses Jackson serialization (smile format) Since no (metastore level) implementation handling pagination existed before, no backwards compatibility is needed. Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>

eric-maynard · 2025-07-22T17:50:46Z

IIUC #1938 supersedes this

dimas-b requested review from adutra, ashvina, dennishuo, eric-maynard, jackye1995, jbonofre, vvcephei, collado-mike, snazy and RussellSpitzer as code owners June 9, 2025 18:27

github-project-automation bot added this to Basic Kanban Board Jun 9, 2025

dimas-b requested review from takidau, MonkeyCanCode, flyrain, ebyhr, ajantha-bhat, HonahX, singhpk234 and pingtimeout as code owners June 9, 2025 18:27

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jun 9, 2025

dimas-b changed the title ~~Pagination alt~~ Delineate pagination requests and tokens Jun 9, 2025

snazy reviewed Jun 10, 2025

View reviewed changes

dimas-b force-pushed the pagination-alt branch from b0b0543 to 22dacc4 Compare June 10, 2025 21:47

dimas-b changed the title ~~Delineate pagination requests and tokens~~ Persistence implementation for pagination in some requests Jun 10, 2025

dimas-b force-pushed the pagination-alt branch from 22dacc4 to 9b284f6 Compare June 10, 2025 21:49

dimas-b force-pushed the pagination-alt branch from 9b284f6 to c2fcf3b Compare June 10, 2025 21:53

dimas-b force-pushed the pagination-alt branch from c2fcf3b to 636ba44 Compare June 10, 2025 21:58

eric-maynard reviewed Jun 11, 2025

View reviewed changes

dimas-b force-pushed the pagination-alt branch from dfd679d to 14e562e Compare June 11, 2025 15:53

dimas-b added 4 commits June 11, 2025 12:36

review: Rename PageToken.decode() back to .build()

ed9b517

review: avoid magic numbers in PolarisEclipseLinkStore

ad1b0a6

review: restore List return type of ListEntitiesResult.getEntities()

45369dc

review: locally immutable tokenFilter in TreeMapTransactionalPersiste…

cf88a5c

…nceImpl

snazy mentioned this pull request Jun 25, 2025

Extensible pagination token implementation #1938

Merged

github-actions bot added the Stale label Jul 14, 2025

eric-maynard closed this Jul 22, 2025

github-project-automation bot moved this from PRs In Progress to Done in Basic Kanban Board Jul 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Persistence implementation for pagination in some requests #1838

Persistence implementation for pagination in some requests #1838

Uh oh!

dimas-b commented Jun 9, 2025 •

edited

Loading

Uh oh!

snazy left a comment

Uh oh!

Uh oh!

Uh oh!

snazy Jun 10, 2025

Uh oh!

dimas-b Jun 11, 2025

Uh oh!

snazy Jun 10, 2025

Uh oh!

dimas-b Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b commented Jun 10, 2025

Uh oh!

eric-maynard Jun 11, 2025

Uh oh!

dimas-b Jun 11, 2025

Uh oh!

github-actions bot commented Jul 14, 2025

Uh oh!

eric-maynard commented Jul 22, 2025

Uh oh!

Uh oh!

Persistence implementation for pagination in some requests #1838

Persistence implementation for pagination in some requests #1838

Uh oh!

Conversation

dimas-b commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

snazy Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

snazy Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b commented Jun 10, 2025

Uh oh!

eric-maynard Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

dimas-b Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 14, 2025

Uh oh!

eric-maynard commented Jul 22, 2025

Uh oh!

Uh oh!

dimas-b commented Jun 9, 2025 •

edited

Loading