-
Notifications
You must be signed in to change notification settings - Fork 124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
File load & pagination issues for user with many files #1362
Comments
Done some initial investigation on this, there's a few ways to approach this task and am keen to get some feedback before tackling it! Look like right now 1000 results are requested by the frontend, Pagination, searching and sorting are completed on the Frontend. If we want to keep all "sort by" and "search" functionality, we would have to replicate this in the API. Pagination
Sorting
Searching
I could fix this all in a full fat solution in order keep parity with existing functionality, or I could implement an MVP hotfix for pagination using mainly existing API features but resulting in losing the sorting for file size and name. Would love to hear your thoughts on this @dchoi27, @alanshaw & @orvn! |
Thanks for this @joshJarr! Yes, I agree, this is far from ideal and was really just implemented as a stopgap. It definitely needs to be replaced because:
Since there's a split table design (not yet in production), where we show files that have been uploaded via the Pinning Services API separately, I think these features (
Agree as well, +1 to using
Most of the time, if my file is named correctly/semantically, I'll search by file name. But I'll also search by a fragment of a CID (to reliably find a specific file in a really large list). |
Thanks @orvn! I've got a WIP PR for these changes: #1408
That's a great shout. I think we can perhaps make a new ticket for applying this logic to the
Cool, I've implemented this in my WIP PR, currently weighing up using an
Cool - this is great to know. I may move adding a search endpoint/filtering by query to a new ticket. This could mean temporarily removing this functionality unless we block this ticket from being released until the search is implemented. I also may move sorting by size to a new ticket. Currently the file size is not stored on the upload entity, so we may need to add this column via migration as it is currently denormalised on request, so I cannot sort the uploads by this efficiently. Keen to know the importance of sort by size and if this too can temporarily be removed to keep this fix small. |
hey @joshJarr Thanks for going through these issues. The historical reason for no pages appearing is that this was not possible with FaunaDB (DB we were using before). Now we can add that if it is important. The offset here will map to use https://supabase.com/docs/reference/dart/range#examples with the page, is it correct? I would call it size and page, given we already name like this in other APIs and prefer to keep things consistent. |
Hey @vasco-santos, thanks for the context. Yep have it currently mapping to a |
@joshJarr @mbommerez Just let me or @drewdelano know once the pagination updates to the api lands so we can update the frontend to use it. |
@JeffLowe Thanks, we are reviewing this internally and get a review from PL next and then get this ready for the next API release. |
To clarify, this ticket's scope should be on pagination for the upload api. The utilization of the api by the frontend is ticketed in #1501 |
Thanks, will do @JeffLowe. @vasco-santos I've requested your review on the PR, would be great to get your input and ensure it's consistent with existing functionality before merging. |
Hey @JeffLowe @drewdelano Pagination I've added pagination metadata to the API response headers that you should be able to use to calculate page numbers and such. These are:
Sorting When running the Web documentation locally, there should be more info there. Let me know if you have any questions or feedback! |
Thank you so much for this into @joshJarr !! |
User observations
Acceptance criteria
Notes
There are 2 parts to this:
https://images.zenhubusercontent.com/378893240/704a835d-8695-4f22-853d-85ad199a1058/screen_recording_2022_03_21_at_10_38_42_720p.mp4
The text was updated successfully, but these errors were encountered: