Skip to content

Refactoring pagination config in list objects #963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Nov 5, 2021
Merged

Refactoring pagination config in list objects #963

merged 10 commits into from
Nov 5, 2021

Conversation

@jaidisido
Copy link
Contributor

@jaidisido jaidisido commented Oct 19, 2021

Issue #, if available:
#955

Description of changes:
Refactored pagination config logic to accept being overridden by the user

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@jaidisido jaidisido added the bug Something isn't working label Oct 19, 2021
@jaidisido jaidisido added this to the 2.13.0 milestone Oct 19, 2021
@jaidisido jaidisido requested a review from kukushking October 19, 2021 14:59
@jaidisido jaidisido self-assigned this Oct 19, 2021
@jaidisido
Copy link
Contributor Author

jaidisido commented Oct 19, 2021

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
  • Commit ID: 8db837e
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido jaidisido linked an issue Oct 19, 2021 that may be closed by this pull request
wr.s3.list_objects not able to limit results by setting s3_additional_kwargs={"MaxKeys": num} #955
Closed
@jaidisido
Copy link
Contributor Author

jaidisido commented Oct 19, 2021

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
  • Commit ID: d80a6be
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking
kukushking reviewed Oct 19, 2021
View reviewed changes
Copy link
Contributor

@kukushking kukushking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows to pass pagination config, but then when the iterator is created, it still loops through all pages and collects the results into an array. Does it make sense to also refactor the code to yield the results page by page if pagination config is provided?

@jaidisido
Copy link
Contributor Author

jaidisido commented Nov 3, 2021

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
  • Commit ID: f37b8d4
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido
Copy link
Contributor Author

jaidisido commented Nov 3, 2021

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
  • Commit ID: 450dd4d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking
kukushking reviewed Nov 4, 2021
View reviewed changes
awswrangler/timestream.py
Returns
-------
pd.DataFrame
Union[pd.DataFrame, Iterator[pd.DataFrame]]
Copy link
Contributor

@kukushking kukushking Nov 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks :)

@jaidisido
Copy link
Contributor Author

jaidisido commented Nov 5, 2021

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
  • Commit ID: ff7b3a9
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido
Copy link
Contributor Author

jaidisido commented Nov 5, 2021

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
  • Commit ID: f3201fe
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@kukushking kukushking merged commit 10fbf10 into main Nov 5, 2021
@kukushking kukushking deleted the bug-955-s3-list-pagination branch November 5, 2021 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kukushking kukushking kukushking left review comments

Assignees

@jaidisido jaidisido

Labels

bug Something isn't working

Projects

None yet

Milestone

2.13.0

Development

Successfully merging this pull request may close these issues.

wr.s3.list_objects not able to limit results by setting s3_additional_kwargs={"MaxKeys": num}

2 participants

@jaidisido @kukushking