-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More advanced pipelining for hf_transfer to increase download speed even further #32
Comments
A simpler alternative solution might be to launch the next hf_transfer instance when the previous file is about 75% done rather than waiting for 100%. But reusing connections from a pool, ideally with at least http/2 so they can launch the second download onwards at full speed immediately, would be better. |
Hi @aikitoria, thanks for the suggestion! Do you have a repo example where such an improvement could have a significant impact? At the moment I'm also cc-ing @Narsil who's implemented |
Sure, here is an example where it would make a difference: https://huggingface.co/databricks/dbrx-instruct/tree/main When downloading this on a server with 10gbit/s download speed, restarting the connections and growing the sliding window again takes up a significant portion of the time. We end up with something like this (exaggerated): |
@Wauplin here is another model where this would be very beneficial: |
The reconnect only occurs because of this: https://github.com/huggingface/hf_transfer/blob/main/src/lib.rs#L162 The reconnects probably only occur because large chunks take more that 20s to download. Edit: Reflecting on this, since we're cloning the client across task, it's very well possible that connection reuse doesn't happen, http2 or not. |
I dunno? Huggingface are hosting the server themselves. Wasn't even aware you can use hf_transfer for something other than downloading models from Huggingface. |
It's just a tool that multiplexes the download over multiple byte range (bypassing some server side rate liimits and using all your cores). Everything I said is a bit wrong, your whole argument is about multiple files, and that still holds, I was all thinking about thread multiplexing. |
Is your feature request related to a problem? Please describe.
hf_transfer is very fast for individual files, but for models with many split files, it's not quite as fast as it could be.
Describe the solution you'd like
![image](https://private-user-images.githubusercontent.com/151776613/317890960-120c4e8a-fc07-47fd-a012-d7dbb43d1d09.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxNzQ0MjIsIm5iZiI6MTcyNDE3NDEyMiwicGF0aCI6Ii8xNTE3NzY2MTMvMzE3ODkwOTYwLTEyMGM0ZThhLWZjMDctNDdmZC1hMDEyLWQ3ZGJiNDNkMWQwOS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyMFQxNzE1MjJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05NGIzNGFkNmFiNGMxMjExOGNlYmIyZDQ4MDhhYjc3NmZlMTZkOGQ1MzMwMDVmNmI1NGRkMTViZGU3NTYxMzkxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.x468w710w3NBY3yp626AlcHNP8PcCy__yLBq_SAYkOw)
Currently, it seems like hf_transfer is invoked serially for each file, starting over from scratch with a new set of connections. That results in a timeline similar to this:
To fully maximize the download speed, what it should do instead is use a shared pool of connections that is reused between small files and chunks of multiple larger files in parallel:
![image](https://private-user-images.githubusercontent.com/151776613/317891243-e53ad700-c72d-46cd-a217-3472873c11cc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxNzQ0MjIsIm5iZiI6MTcyNDE3NDEyMiwicGF0aCI6Ii8xNTE3NzY2MTMvMzE3ODkxMjQzLWU1M2FkNzAwLWM3MmQtNDZjZC1hMjE3LTM0NzI4NzNjMTFjYy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyMFQxNzE1MjJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lZjZkMGE0ZTdmY2Y5ZDRhNTU1MWNlNDllY2E3M2QxOTQyOTViYTY3OTk0ZGIyZGJkYzQ3Mjg5Njg3ZTVhYmQ2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.VRjTFhWk5iZm5vEYIFLvOFDVXd6BWYIjL3t5G2krwNk)
The text was updated successfully, but these errors were encountered: