Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple connections / multipart download #86

Open
daveuk100 opened this issue Jan 4, 2021 · 4 comments
Open

Support multiple connections / multipart download #86

daveuk100 opened this issue Jan 4, 2021 · 4 comments

Comments

@daveuk100
Copy link

grab works very well and I see that you are already making use of Accept-Ranges response to support resuming of downloads, but do you have any supports or plans to support multipart downloading? (i.e. allowing us to download a file in chunks using several connections).

Scenario: I have a large file to download (several GB) and I find that when I use other go packages that support multipart I am able to download the file in a fraction of the time when compared to grab's single connection.

@MGThePro
Copy link

I am also looking to use a library for downloading files and multiple connection seems to be the only major piece that is missing for my needs

@daveuk100
Copy link
Author

daveuk100 commented Jan 24, 2021

I am also looking to use a library for downloading files and multiple connection seems to be the only major piece that is missing for my needs

As there was not a reply here, I went looking and adopted another project “GetParty” which has been fantastic:

https://github.com/vbauerster/getparty

It supports multi-part chunked downloading and is amazing at retrying the parts if they fail. Absolutely perfect for my needs and any issues I raise are resolved within 24 hours as the maintainer is very active.

I think that if Grab is still being maintained (not sure) it would be great for them to adopt the same approach and/or work with GetParty to create the best overall combination of the two projects. One go download library to rule them all!

@justinfx
Copy link

justinfx commented Jun 23, 2023

I'm using Grab pretty deeply integrated into my current solution, with the callback hooks, and now I also want to support Range chunk downloading. I was looking at the current implementation to see if there is a decent hook in to apply it in my own code. The only spot I see is a custom HTTPClient interface to intercept and handle the http Request. But it doesn't seem like it plays well with the rest of the design of the state machine.
GetParty, mentioned in the earlier comment, doesn't appear to be a library; only a CLI.
I've had a play with this option, which is both a CLI and library: https://github.com/pgollangi/fastget
I do need this support, so I may end up trying to implement the fastget logic into Grab and see where I get. If it works out, I can submit a PR.

@justinfx
Copy link

I've got a PR up with support for parallel "Range" requests, if anyone wants to have a look?
#102

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants