Could mmap supports max file size?

[luokar]

mmap make downloading faster and generating less disk IO which is really important for a low end computer. However, if the file length is bigger than the total number of RAM in machine, aria2 will just crash and slow down the system. Could it be possible to add a setting like mmap-max-file-size and turn off mmap for file bigger than certain size to workaround this problem?

[tatsuhiro-t]

As far as I know, if mmaped memory is written to the disk and purged from physical RAM in the case of low memory situation. mmap is not faster than ordinary read/write system call. If mmap causes an issue, could you try --disk-cache option? It uses 16MB memory per aria2 instance for disk cache by default.

[luokar]

@tatsuhiro-t

Disk-cache works, but it performance much worse than mmap on a fast line.

Downloading http://ubuntu.hkets.org/14.04.3/ubuntu-14.04.3-desktop-amd64.iso
on window with 1000m broadband and file-allocation=falloc

with mmap : 15 second without hang
without mmap & disk-cache=1000M: 25 second and hang

I guess there are two reason for the difference in performance between cache and mmap

1. 16M cache is too small for >50MBps
   HDD might work like this:
   cache full, wait for flushing 16MB, cache full ...
   with mmap: keep downloading, flush 1G file into disk sequently

2)file-allocation works better with mmap on windows
aria2 always hang when file-allocation. I didn't observe hanging anymore after turning on mmap.

As I know, people are getting decent broadband for a affordable price. People living including me in Hong Kong pay 23USD for 1000m 20USD for 500m per month.
To prepare for the future(high network throughput but slow disk), I believe aria2 should have a larger cache for single instance or make mmap less dangerous. Anyway, thanks for your quick response.

[tatsuhiro-t]

The disk cache size is configurable. Try --disk-cache option.

[luokar]

@tatsuhiro-t Sorry, I might misunderstood what you mean by per aria2 instance as per task, but i have set disk-cache=1000M. Aria2 still hang during file-allocation and it spending more time at downloading a 1G file.

[tatsuhiro-t]

file-allocation=falloc is independent from mmap or disk cache. If it blocks with disk-cache, then it also blocks with mmap. BTW, what file system do you use?

[tatsuhiro-t]

And are you using aria2 on windows?

[luokar]

@tatsuhiro-t I use NTFS on Windows 7. falloc should take only a few milliseconds for a large file, but instead it works like zero filling on Windows 7 with NTFS

[tatsuhiro-t]

I have also experienced similar problem with NTFS with falloc before. aria2 blocked after file allocation completed, somehow operation system was doing something funny.
My knowledge of Windows is limited and still not sure how to fix this properly.
My comment about mmap is for Linux. Windows mmap counterpart may not as smart as Linux's, and might blow up if memory is short.
Probably the safe bet is use --file-allocation=prealloc. If you use --file-allocation=prealloc, do you have the same problem? NTFS supports sparse files, so you may try --file-allocation=trunc, which is faster, but could be resulted in a fragmented file.

[tatsuhiro-t]

We use SetEndOfFile to allocate file space, but it seems it performs zero-fills.
Instead, documentation suggests that SetFileValidData avoids zero-fills, but it also suggests security implications:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365544%28v=vs.85%29.aspx

[luokar]

@tatsuhiro-t I have tested file-allocation=trunc for a while. The outcome is depressing. It yields even lower performance(maybe problem related to fragmentation) and hanging still appears sometime. I think I better keep using mmap with caution. If this won't be fixed, please tell me and I will close this issue.

[tatsuhiro-t]

fc95a91 will make --file-allocation=falloc faster by eliminating zero-fill.
Crash does not sound good, so I'm now considering to add option to set maximum file size for mmap.

[tatsuhiro-t]

--max-mmap-limit options was added by commit 8f51793

[luokar]

Excellent, I know closed source stuffs like windows is somehow weird and broken(being forced to use it by my job). I am so glad you are willing to fix these problems for people in pain. Nicely done. I will compile and test it tomorrow.

[luokar]

The patch does its job.
The performance gain of using mmap is probably from tricking NTFS not to do zero fills by writing sequentially. But since the root cause have been fixed, I am closing this issue.

[tatsuhiro-t]

Great to hear that. Thank you.

[zizhengwu]

Hi, I am wondering what will happen if mmap runs out of memory. Will the memory be flushed to the disk or the process be killed by kernel? I am using aria2 on Windows with Cygwin.

[tatsuhiro-t]

I'm not sure about Windows Cygwin internals. As for Linux, when memory is required for another processes, kernel will flush data into disk as needed.