Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prefetch functionality for paginator #5310

Merged
merged 3 commits into from
May 9, 2024

Conversation

zychen5186
Copy link
Contributor

TL;DR

The get function takes more time to fetch data from the remote server as the amount increases, hence we only fetch 100 rows initially, and prefetch 100 rows of data at once while the user scrolls through pages to enhance the user experience.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

For testing run:

make compile
./bin/flytectl get execution -p my-project -d development -i

We initially fetch the first 100 rows when the paginator is called and start to prefetch the next batch (next 100 rows).
Prefetch will be triggered when the current page exceeds a prefetch threshold.
The first or last batch will be discarded when it exceeds the cache limit (currently set to 1000 rows).
We added a loading animation to indicate the fetching process is ongoing.
Apr-27-2024 17-32-22

Users can use --filter.page flag to specify their starting page, if not specified, page will start from 1.
Apr-28-2024 16-21-42

Tracking Issue

#4440

Follow-up issue

Follow-up of flyteorg/flytectl#473

Big thanks to @troychiu for discussing and debugging with me!! 🙌 ❤️

Copy link

welcome bot commented May 1, 2024

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
  • Sign off your commits (Reference: DCO Guide).

Copy link

codecov bot commented May 1, 2024

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 61.10%. Comparing base (cb57beb) to head (228b346).

Files Patch % Lines
flytectl/cmd/get/execution.go 0.00% 5 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5310   +/-   ##
=======================================
  Coverage   61.10%   61.10%           
=======================================
  Files         794      794           
  Lines       51213    51213           
=======================================
  Hits        31295    31295           
  Misses      17037    17037           
  Partials     2881     2881           
Flag Coverage Δ
unittests-datacatalog 69.31% <ø> (ø)
unittests-flyteadmin 58.90% <ø> (ø)
unittests-flytecopilot 17.79% <ø> (ø)
unittests-flytectl 68.30% <0.00%> (ø)
unittests-flyteidl 79.30% <ø> (ø)
unittests-flyteplugins 61.94% <ø> (ø)
unittests-flytepropeller 57.32% <ø> (ø)
unittests-flytestdlib 65.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@troychiu troychiu marked this pull request as ready for review May 1, 2024 18:09
pingsutw
pingsutw previously approved these changes May 2, 2024
@pingsutw
Copy link
Member

pingsutw commented May 7, 2024

@zychen5186 One of the flytectl tests is failing. Mind taking a look?

@zychen5186
Copy link
Contributor Author

@zychen5186 One of the flytectl tests is failing. Mind taking a look?

No problem

Signed-off-by: zychen5186 <brianchen5197@gmail.com>
Signed-off-by: zychen5186 <brianchen5197@gmail.com>
Signed-off-by: zychen5186 <brianchen5197@gmail.com>
@pingsutw pingsutw merged commit aa1f211 into flyteorg:master May 9, 2024
47 of 48 checks passed
Copy link

welcome bot commented May 9, 2024

Congrats on merging your first pull request! 🎉

robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this pull request Jul 2, 2024
Signed-off-by: zychen5186 <brianchen5197@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants