Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pikpak: improve upload reliability and resolve potential file conflicts #7852

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

pikpak: improve upload reliability and resolve potential file conflicts #7852

wants to merge 3 commits into from

Conversation

wiserain
Copy link
Contributor

What is the purpose of this change?

This PR includes several attempts to improve upload reliability and to resolve potential file conflicts:

  • Checking upload status: Implemented getTask() to verify successful completion on the server-side for each upload.
  • Cancelling uploads on errors: Uploads encountering errors are cancelled to remove residual files.
  • Forced & Minimum Sleep:
    • Based on experiments, a forced delay(500ms) is introduced after uploads to ensure server-side updates register (minimal impact on total execution time).
    • Minimum sleep for pacer increased from 10ms to 100ms to resolve server errors.
  • Removed uploadByForm(): This method is less reliable for a large number of small files. We now solely use uploadByResumable().

Was the change discussed in an issue or in the forum before?

#7787

Checklist

  • I have read the contribution guidelines.
  • I have added tests for all changes in this PR if appropriate.
  • I have added documentation for the changes if appropriate.
  • All commit messages are in house style.
  • I'm done, this Pull Request is ready for review :-)

@wiserain
pikpak: fix upload conflict
2e582d7 
This attempts to resolve upload conflicts by implementing cancel/cleanup on failed
uploads

Fixes #7787
@wiserain
fix panic error on defer cancel upload
cbdd879
@wiserain
many tweaks
223ba86 
* increase pacer min sleep from 10 to 100 ms
* stop using uploadByForm()
* introduce force sleep before and after async tasks
* use pacer's retry scheme instead of manual implementation
@wiserain wiserain requested a review from ncw May 16, 2024 22:45
@wiserain wiserain mentioned this pull request May 16, 2024
Open
@wiserain
Copy link
Contributor Author

@ncw If this looks ok, I will merge it. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ncw ncw Awaiting requested review from ncw

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@wiserain