Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add a query paginator #1530

Merged
merged 9 commits into from Nov 29, 2022
Merged

feat: add a query paginator #1530

merged 9 commits into from Nov 29, 2022

Conversation

mutianf
Copy link
Contributor

@mutianf mutianf commented Nov 22, 2022

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕️

If you write sample code, please follow the samples format.

@mutianf mutianf requested a review from a team as a code owner November 22, 2022 18:36
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigtable Issues related to the googleapis/java-bigtable API. labels Nov 22, 2022
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Nov 23, 2022
this.query = query;
this.chunkSize = chunkSize;
this.prevSplitPoint = ByteString.EMPTY;
this.firstRun = true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need to set the limit on the query here? otherwise gtNextQuery will return the wrong thing

Copy link
Contributor Author

@mutianf mutianf Nov 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm this should be handled by advnace and getNextQuery needs to be called after advance:

Query query = Query.create(...).range("a", "z");
Query.QueryPaginator paginator = query.createQueryPaginator(100);
ByteString lastSeenRowKey = ByteString.EMPTY;
while (paginator.advance(lastSeenRowKey)) {
    List<Row> rows = client.readRowsCallable().all().call(paginator.getNextQuery());
      for (Row row : rows) {
      // do some processing
     lastSeenRow = row;
    }
 }

private ByteString prevSplitPoint;

QueryPaginator(@Nonnull Query query, int pageSize) {
this.hasOverallLimit = query.builder.getRowsLimit() == 0 ? false : true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this.hasOverallLimit = query.builder.getRowsLimit() > 0;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦‍♀️

Copy link
Contributor

@igorbernstein2 igorbernstein2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm after nit

@mutianf mutianf added the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 29, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 29, 2022
@gcf-owl-bot gcf-owl-bot bot requested a review from a team as a code owner November 29, 2022 21:20
@mutianf mutianf merged commit 5c8e1f6 into googleapis:main Nov 29, 2022
@mutianf mutianf deleted the paginator branch November 29, 2022 22:21
@mutianf mutianf restored the paginator branch November 29, 2022 22:21
@mutianf mutianf deleted the paginator branch November 29, 2022 22:21
@mutianf mutianf restored the paginator branch November 29, 2022 22:21
gcf-merge-on-green bot pushed a commit that referenced this pull request Dec 7, 2022
🤖 I have created a release *beep* *boop*
---


## [2.17.0](https://togithub.com/googleapis/java-bigtable/compare/v2.16.0...v2.17.0) (2022-12-07)


### Features

* Add a query paginator ([#1530](https://togithub.com/googleapis/java-bigtable/issues/1530)) ([5c8e1f6](https://togithub.com/googleapis/java-bigtable/commit/5c8e1f656b15177ecef4709b9e502cef58cca479))
* Add test proxy to java-bigtable client ([#1498](https://togithub.com/googleapis/java-bigtable/issues/1498)) ([132b4e4](https://togithub.com/googleapis/java-bigtable/commit/132b4e4fe340ca316de8eef2e2133b6dabd9dec3))
* Implement cross-project table restore ([#1536](https://togithub.com/googleapis/java-bigtable/issues/1536)) ([335977c](https://togithub.com/googleapis/java-bigtable/commit/335977c499b1776c8fa861d13195ffc49e468e0a))
* Next release from main branch is 2.17.0 ([#1514](https://togithub.com/googleapis/java-bigtable/issues/1514)) ([4fc6a93](https://togithub.com/googleapis/java-bigtable/commit/4fc6a93a25362df7bc107d48f94e1c00b6bd608d))


### Dependencies

* Update dependency com.google.cloud:google-cloud-monitoring-bom … ([#1531](https://togithub.com/googleapis/java-bigtable/issues/1531)) ([ee98338](https://togithub.com/googleapis/java-bigtable/commit/ee9833835a84cee202b142950b28704db682ac0c))
* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.0 ([#1539](https://togithub.com/googleapis/java-bigtable/issues/1539)) ([3244cef](https://togithub.com/googleapis/java-bigtable/commit/3244cefd4d77a44bab6ccaa8f5c03e3d31575625))
* Update dependency kr.motd.maven:os-maven-plugin to v1.7.1 ([#1518](https://togithub.com/googleapis/java-bigtable/issues/1518)) ([8309681](https://togithub.com/googleapis/java-bigtable/commit/830968109a3754a12bd0bc92674fe42ae529b924))
* Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.19 ([#1537](https://togithub.com/googleapis/java-bigtable/issues/1537)) ([7f7c478](https://togithub.com/googleapis/java-bigtable/commit/7f7c478a2163c46d10ed39ea3c1b046f971d4569))
* Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.19 ([#1538](https://togithub.com/googleapis/java-bigtable/issues/1538)) ([9d3df57](https://togithub.com/googleapis/java-bigtable/commit/9d3df57d7bfeafd9cbcf56ecd58b52cc1b14ba7b))

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigtable Issues related to the googleapis/java-bigtable API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants