Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aria2 eats all the RAM #902

Open
Dingo64 opened this issue May 5, 2017 · 4 comments
Open

Aria2 eats all the RAM #902

Dingo64 opened this issue May 5, 2017 · 4 comments
Labels

Comments

@Dingo64
Copy link

@Dingo64 Dingo64 commented May 5, 2017

I run aria2 1.31.0 for Win64 with parameters:
aria2c -i list.txt -c false -j 18
File list.txt is 126 MB in size and uses this format:
http://website.com/file1.txt
out=dir/file1.txt
for each of 1 950 000 files (yes, almost 2 million).

It quickly eats all the available RAM (over 15 GB) and doesn't even start downloading.
So it's not really "lightweight".

Since it is going to download only 18 files in parallel perhaps it could just load the file list bit by bit? Or store links in memory in a more efficient way? Again the whole list in TXT format is just 126 MB so perhaps it could load it as is into memory and then copy from it 1 MB chunks into an array, download everything from the array, empty it and copy next 1 MB part?
Or maybe I can already configure it somehow to eat less RAM?

Compiler: mingw-w64 5.0 (alpha) / gcc 6.2.1 20161118
built by x86_64-pc-linux-gnu
targetting x86_64-w64-mingw32
on Jan 16 2017 22:39:36
System: Windows 7 (Service Pack 1) (x86_64) (6.1)
aria2

@Dingo64

This comment has been minimized.

Copy link
Author

@Dingo64 Dingo64 commented May 5, 2017

I regenerated by list.txt with only 500 000 links. Now aria2 started and uses almost 6 GB of RAM.
So I think it needs about 24 GB of RAM for 2 000 000 links.

@BrutalSimplicity

This comment has been minimized.

Copy link

@BrutalSimplicity BrutalSimplicity commented May 6, 2017

Aria2 saves the status of each download in memory, which is likely why you are having this issue. You may want to try using the --deferred-input option.

--deferred-input[=true|false]
If true is given, aria2 does not read all URIs and options from file specified by --input-file option at startup, but it reads one by one when it needs later. This may reduce memory usage if input file contains a lot of URIs to download. If false is given, aria2 reads all URIs and options at startup. Default: false

@Dingo64

This comment has been minimized.

Copy link
Author

@Dingo64 Dingo64 commented May 6, 2017

Thank you, it helped.
One thing was quite bothering. It started with 8 MB and was eating 1 MB more every 7 seconds. But it stopped at 18 MB so it's great. About 1 million links loaded.

@tatsuhiro-t

This comment has been minimized.

Copy link
Collaborator

@tatsuhiro-t tatsuhiro-t commented May 7, 2017

If you are concerned with the memory usage, also try --disk-cache=0.

@tatsuhiro-t tatsuhiro-t added the question label May 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.