Upload large folders with symlinks

[neverix]

Is your feature request related to a problem? Please describe.
I have a folder with symlinks. upload_large_folder currently doesn't work with that.

Describe the solution you'd like

huggingface_hub/src/huggingface_hub/_upload_large_folder.py

Line 99 in 290aa26

(path.relative_to(folder_path).as_posix() for path in folder_path.glob("**/*") if path.is_file()),

Set recurse_symlinks to True here or add an option to enable that
Describe alternatives you've considered
Make it possible to upload_large_folder into paths other than the root. There's no other way to upload multiple large folders to one repo.

[hanouticelina]

Hello @neverix, thanks for the feature request!

Set recurse_symlinks to True here or add an option to enable that

Adding symlink support seems feasible by adding a parameter to control Path.glob() behavior. We'll investigate if there are any limitations or additional maintenance burden, so let's keep this issue open 😄

Make it possible to upload_large_folder into paths other than the root. There's no other way to upload multiple large folders to one repo.

Regarding uploading to subfolders support: This limitation is intentional. The caching system that enables resuming interrupted uploads relies on a direct 1:1 relationship between local files and their destination in the repo. For now, we recommend structuring your local folders to match your desired repo structure.

In any case, upload_large_folder is primarily designed to be a reliable, resumable, and "resilient to errors" upload tool without having to compromise its robustness.