Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download of large files blows memory usage and new download type issues #5

Open
affinityv opened this issue Jan 1, 2019 · 2 comments

Comments

@affinityv
Copy link
Contributor

I've done work to fix this, will do pull request when I have the time.
Essentially the changes I made this time allow for download directly to file system files, rather than a string in memory. The sha1 and md5 checks are also done against the files only.

There were some issues with a new bundle [1] , very large video zip files for starters and "supplement" files -- both of which are just zip files, so a bit of fudging for the file naming.

Attached is a quick "diff -u" patch file .... FYI (before the pull request), feel free to incorporate early if you don't want to wait for the pull request. I may make some other minor adjustments as well, but I think it is okay as it is now.

affinityv.patch.txt

[1] https://www.humblebundle.com/books/python-packt-2019-books

@tlc
Copy link

tlc commented Aug 18, 2020

Thanks for the downloader, especially the file renaming! This will improve my workflow a lot.

But a large comics download is crawling and using all of my 12G memory. The files so far range from 79M to 795M. I assume the program is trying to download all at once?

If the affinityv pull isn't going to be taken, perhaps a way to limit concurrent downloads?

BTW, at least once I got a comic from HB > 2G.

@affinityv
Copy link
Contributor Author

affinityv commented Aug 18, 2020

I am happy for my code to be fully implemented in to the original project, but I'm not personally able, at this time, to do what needs to be done to get it sorted. People are welcome to diff my code changes and implement it back to the project properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants