-
Notifications
You must be signed in to change notification settings - Fork 513
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CLI to upload arbitrary huge folder #2254
Merged
Merged
Changes from all commits
Commits
Show all changes
97 commits
Select commit
Hold shift + click to select a range
1004434
still an early draft
Wauplin 5a8605e
this is better
Wauplin 68a6cf1
fix
Wauplin 9b25f38
Merge branch 'main' intto 1738-revampt-download-local-dir
Wauplin 8e903f8
revampt/refactor download process
Wauplin 5f610ee
resume download by default + do not upload .huggingface folder
Wauplin 5a9762f
compute sha256 if necessary
Wauplin 283977d
fix hash
Wauplin e909022
add tests + fix some stuff
Wauplin 39cfef4
fix snapshot download tests
Wauplin dbece97
fix test
Wauplin 0206964
lots of docs
Wauplin 82b46b3
add secu
Wauplin 3300b28
as constant
Wauplin c606a94
dix
Wauplin 95171ef
fix tests
Wauplin 7180746
remove unused code
Wauplin 4e664d4
don't use jsons
Wauplin 7bb263e
style
Wauplin 3595042
Apply suggestions from code review
Wauplin 3401880
Apply suggestions from code review
Wauplin 9210648
Warn more about resume_download
Wauplin fb477e5
fix test
Wauplin 0eacbc9
Add tests specific to .huggingface folder
Wauplin 1a4320a
remove advice to use hf_transfer when downloading from cli
Wauplin 8c9dc8b
fix torhc test
Wauplin 6260a17
more test fix
Wauplin c788e2d
Merge branch 'main' into 1738-revampt-download-local-dir
Wauplin 3a45f4b
feedback
Wauplin 30abe07
First draft for large upload CLI
Wauplin 4317f5a
Fixes + CLI
Wauplin 94951e4
verbose by default
Wauplin 0061401
ask for report
Wauplin 8d31fc0
line
Wauplin 4f6f531
suggested changes
Wauplin 41c8ae3
more robust
Wauplin 84f55ff
Apply suggestions from code review
Wauplin f5d1faa
comment
Wauplin edc9790
commen
Wauplin a768426
Merge branch 'main' into 1738-revampt-download-local-dir
Wauplin d0ea3ea
robust tests
Wauplin dffa539
fix CI
Wauplin d414825
ez
Wauplin 0a5605c
rules update
Wauplin 9e6d569
more ribust?
Wauplin e6fe766
allow for 1s diff
Wauplin 28991c9
don't raise on unlink
Wauplin a0b61a1
style
Wauplin fccabe0
robustenss
Wauplin c089477
Merge branch '1738-revampt-download-local-dir' into large-upload-cli
Wauplin 6295ead
Merge branch 'main' into large-upload-cli
Wauplin 06c9dec
tqdm while recovering
Wauplin d6e4163
Merge branch 'main' into large-upload-cli
Wauplin 33be5f0
Merge branch 'main' into large-upload-cli
Wauplin e48d525
Merge branch 'main' into large-upload-cli
Wauplin 64352de
make sure upload paths are correct on windows
Wauplin d936caf
test get_local_upload_paths
Wauplin 8dae1ac
only 1 preupload LFS at a time if hf_transfer enabled
Wauplin 8326ea2
upload one at a time if hf_transfer
Wauplin e747f9b
Add waiting workers in report
Wauplin 3d7d5a3
better reporting
Wauplin 07650cd
Merge branch 'main' into large-upload-cli
Wauplin 095aa90
raise on KeyboardInterrupt + can disable bars
Wauplin 5119e74
Merge branch 'main' into large-upload-cli
Wauplin 76af338
Merge branch 'main' into lt parge-upload-cli
Wauplin bf0f967
fix type annotation on Python3.8
Wauplin ba7f248
make repo_type required
Wauplin dd3c31c
Merge branch 'main' into large-upload-cli
Wauplin b134e17
dcostring
Wauplin b725f94
style
Wauplin cd8bac2
fix circular import
Wauplin 4ccfec0
docs
Wauplin d8f6846
docstring
Wauplin 5f16da2
init
Wauplin f03b55d
guide
Wauplin da0bb42
dedup
Wauplin 1b04201
instructions
Wauplin 84c65a5
add test
Wauplin f31dfee
styl
Wauplin d0ae3a0
Merge branch 'main' into large-upload-cli
Wauplin 602f299
tips
Wauplin dabd594
Apply suggestions from code review
Wauplin 5583f0c
typo
Wauplin 69dd31d
remove comment
Wauplin 86edb11
comments
Wauplin bc7129f
move determine_task to its own method
Wauplin 0dc048a
rename to upload_large_folder
Wauplin 8181193
fix md
Wauplin b568a54
update
Wauplin 3bbdc2e
dont wait on exit
Wauplin ea765be
Fix typo in docs
Wauplin 46ce7a8
Apply suggestions from code review
Wauplin ba6e4a8
add PR tips
Wauplin 3e2db8e
Merge branch 'large-upload-cli' of github.com:huggingface/huggingface…
Wauplin cac2b50
Merge branch 'main' into large-upload-cli
Wauplin cdfb27f
add comment
Wauplin 6d8017a
add comment about --no-bars
Wauplin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -138,3 +138,5 @@ dmypy.json | |
|
||
# Spell checker config | ||
cspell.json | ||
|
||
tmp* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For a follow-up PR: I think we could support this. The PR title could be the commit message (and then just add "Part 1" or something like that)