Compile unrar for Linux with fallocate

[itouch5000]

Thank you for the great new features in 19.0-testing-r1991. But I have a small problem with DirectUnpack in Unix in combination with Kodi as video player.

Unrar for Windows preallocs the drive space for extracted files to reduce fragmentation. In Unix the appropriate code is present in unrar source, but not enabled yet, because fallocate call was not
available on older Linux kernels. So in Unix the file size increase with each extracted part. VLC can play these videos but Kodi for example stops playback after a few minutes (after the length of the video at the time of started the playback).
How to fix that: itouch5000/unrarstream@2faa08e

[hugbug]

NZBGet doesn't require any unrar patches and works fine with stock unrar. However you are free to use whatever version of unrar you prefer.

I'm not sure what do you mean with this issue. Are you suggesting to activate fallocate in the unrar shipped with nzbget? I don't know if I can do that as it may make nzbget installer incompatible with older Linux versions. Currently installer works with kernel 2.6.something whereas I don't know how low "something" can be.

Wouldn't fallocating kill unpack perfomance as it needs to fill the whole file with nulls?
Have you measured unrar with and without fallocate?

I wonder why do you modify unrar source instead of using proper compiler flags as intended by unrar author.

BTW if Kodi can't play partially downloaded files isn't that something to report to Kodi team? In my tests Kodi also behaves bad on Windows in combination with direct unpack as it blocks files from being moved/renamed. Windows media player works very well in this regard.

[itouch5000]

Yes I am suggesting to activate fallocate in the unrar shipped with nzbget. But if it drops the compatibility I will try to report this issue to Kodi team. I thought that it's the easiest method to solve this problem and support more video player.

Wouldn't fallocating kill unpack perfomance as it needs to fill the whole file with nulls?
Have you measured unrar with and without fallocate?

I have no idea how to measure that. Which program should I use for this test?

I wonder why do you modify unrar source instead of using proper compiler flags as intended by unrar author.

Sorry I'm a noob and I have never used a compiler flag. I have written with the unrar author via email some time ago and he recommended me to change his line in source before compile unrar. Nothing about a compiler flag.
Is it possible to compile nzbget with this unrar compiler flag to use it as long as Kodi is not able to play these files?

[hugbug]

Is it possible to compile nzbget with this unrar compiler flag

Unrar and nzbget are two different binaries. You can compile unrar yourself and use it with nzbget.

Nothing about a compiler flag.
The unrar code from your commit checks symbol USE_FALLOCATE. If you define the symbol in compiler flags then you don't need to change sources of unrar.

You can pass it via CLFAGS (I haven't tried this but it should work):

CFLAGS=-DUSE_FALLOCATE make

Or you can put extra flags into unrar's makefile, which is supposed to be changed by user to adjust to system (added -DUSE_FALLOCATE):

DEFINES=-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DRAR_SMP -DUSE_FALLOCATE

I have no idea how to measure that. Which program should I use for this test?

Since you compile unrar yourself: make two versions with and without fallocate. Then try unpacking the same archive with both unrars, at best do the test multiple times and unpack into convenient HDD. You can use command time to measure run time:

time unrar x archive.rar

[itouch5000]

Thank you very much for the detailed explanation! I will make some tests and report the results.

[itouch5000]

CFLAGS=-DUSE_FALLOCATE make does not work for me. But append the flag to unrars makefile works great.
I have made 12 tests (6 for each executable). In the order "1. unrar without fallocate", "1. unrar with fallocate", "2. unrar without fallocate", "2. unrar with fallocate"... So you can compare them directly.
The result is that unrar with fallocate is even faster than unrar without fallocate.

unrar without fallocate:

1. real 0m50.048s
   user 0m9.616s
   sys 0m10.280s
2. real 0m50.435s
   user 0m10.280s
   sys 0m9.684s
3. real 1m3.102s
   user 0m9.660s
   sys 0m11.580s
4. real 0m59.830s
   user 0m9.752s
   sys 0m11.232s
5. real 0m54.594s
   user 0m9.608s
   sys 0m10.400s
6. real 0m55.073s
   user 0m9.224s
   sys 0m11.044s

unrar with fallocate:

1. real 0m49.018s
   user 0m10.764s
   sys 0m7.900s
2. real 0m49.448s
   user 0m10.916s
   sys 0m7.720s
3. real 0m52.695s
   user 0m11.068s
   sys 0m7.588s
4. real 0m53.896s
   user 0m11.960s
   sys 0m6.540s
5. real 0m53.670s
   user 0m11.776s
   sys 0m6.580s
6. real 0m54.016s
   user 0m12.396s
   sys 0m5.872s

[hugbug]

Nice tests!

HDD or SSD?
What is the filesystem?
How big the test achive?

[itouch5000]

First I have made some tests on my Desktop PC (Linux Mint 18.1) with an SSD. But the measured times were too different. Still I have seen, that fallocate is faster than without. (If you are interested in the tests I can post them too)

The tests above I have made with my little homeserver (CPU while unpack ~30%) and extracted the archive to an HDD because I have no HDD in my Desktop PC.
Ubuntu Server 16.04.2 LTS (GNU/Linux 4.4.0-75-generic x86_64)
Filesystem is ext4
Size of archive.rar is 4,7 GB (4.667.572.901 Bytes)

[hugbug]

The docs for fallocate say it fills the file with nulls. That must take time, right? I can imagine ext4 may have special support for that like marking blocks nullified without actual writing to disk.

Do you have a chance to test with other file systems? Ext3 on local drive and ext3/ext4 via samba (NAS) or maybe even ntfs via samba.

[itouch5000]

Yes, theoretically it has to cost time. But I'm not an expert :D
I have some HDDs laying around and can make these tests the next days.

• ext4 on local hdd
• ext3 on local hdd
• ext4 via samba
• ext3 via samba
• ntfs via samba

[hugbug]

There is fallocate system call and fallocate console utility. The man page for the utility documents which file systems it supports (only a few actually, including ext4 but not ext3). The man page for system call doesn't mention file systems.

I wonder what will happen on a file system which has no special support: will the call fail or will it write the nulls. In the first case the file size will not be set to desired size (you should be able to control that). In the second case it should take long time writing all these nulls.

Also intersring is if the ext4 accessed via network can still take advantage or the network protocol will prevent optimizations.

In our case I would prefer fallocate quickly to fail on unsupported file systems instead of writing nulls and doubling the unpack time.

Thanks for testing this.

[itouch5000]

local ext3:

fallocate does not work, file is not filled with nulls

unrar original:

1. real 0m54.634s
   user 0m9.192s
   sys 0m14.708s

2. real 0m51.775s
   user 0m9.140s
   sys 0m14.704s

3. real 0m53.301s
   user 0m9.248s
   sys 0m14.820s

4. real 0m52.774s
   user 0m8.984s
   sys 0m14.924s

5. real 0m53.480s
   user 0m9.168s
   sys 0m15.192s

6. real 0m52.349s
   user 0m9.160s
   sys 0m14.948s

unrar fallocate:

1. real 0m51.729s
   user 0m9.216s
   sys 0m14.632s

2. real 0m49.967s
   user 0m9.216s
   sys 0m14.880s

3. real 0m52.664s
   user 0m9.176s
   sys 0m15.028s

4. real 0m53.940s
   user 0m9.060s
   sys 0m15.116s

5. real 0m54.939s
   user 0m9.116s
   sys 0m15.016s

6. real 0m53.952s
   user 0m9.268s
   sys 0m14.720s

local ext4:

fallocate works, file is filled with nulls

unrar original:

1. real 0m41.577s
   user 0m9.516s
   sys 0m9.584s

2. real 1m36.869s
   user 0m11.280s
   sys 0m14.492s

3. real 0m43.929s
   user 0m9.244s
   sys 0m9.884s

4. real 0m42.626s
   user 0m9.320s
   sys 0m10.020s

5. real 0m41.864s
   user 0m9.640s
   sys 0m9.644s

6. real 0m41.444s
   user 0m9.616s
   sys 0m9.556s

unrar fallocate:

1. real 0m41.614s
   user 0m9.600s
   sys 0m8.312s

2. real 1m0.009s
   user 0m10.084s
   sys 0m9.884s

3. real 0m42.756s
   user 0m9.460s
   sys 0m8.140s

4. real 0m41.972s
   user 0m9.444s
   sys 0m8.564s

5. real 0m41.505s
   user 0m9.952s
   sys 0m7.764s

6. real 0m41.495s
   user 0m10.296s
   sys 0m7.488s

local ntfs:

fallocate does not work, file is not filled with nulls

unrar original:

1. real 1m25.394s
   user 0m10.308s
   sys 0m15.088s

2. real 1m25.013s
   user 0m10.156s
   sys 0m15.136s

3. real 1m24.999s
   user 0m10.052s
   sys 0m15.328s

4. real 1m25.005s
   user 0m10.448s
   sys 0m15.000s

5. real 1m25.256s
   user 0m10.492s
   sys 0m14.856s

6. real 1m25.328s
   user 0m10.252s
   sys 0m15.116s

unrar fallocate:

1. real 1m28.696s
   user 0m9.976s
   sys 0m15.336s

2. real 1m28.053s
   user 0m9.840s
   sys 0m15.524s

3. real 1m28.161s
   user 0m10.236s
   sys 0m15.108s

4. real 1m28.118s
   user 0m9.864s
   sys 0m15.420s

5. real 1m28.221s
   user 0m9.824s
   sys 0m15.432s

6. real 1m28.367s
   user 0m9.780s
   sys 0m15.640s

samba ext3:

fallocate does not work, file is not filled with nulls

unrar original:

1. real 1m5.491s
   user 0m2.288s
   sys 0m4.300s

2. real 1m5.003s
   user 0m2.556s
   sys 0m3.788s

3. real 1m1.285s
   user 0m2.476s
   sys 0m3.892s

4. real 1m2.796s
   user 0m2.204s
   sys 0m3.956s

5. real 1m4.359s
   user 0m2.368s
   sys 0m4.196s

6. real 1m1.211s
   user 0m2.444s
   sys 0m3.536s

unrar fallocate:

1. real 1m4.110s
   user 0m2.296s
   sys 0m4.140s

2. real 1m4.851s
   user 0m2.388s
   sys 0m4.140s

3. real 1m3.523s
   user 0m2.376s
   sys 0m4.048s

4. real 1m3.358s
   user 0m2.544s
   sys 0m3.924s

5. real 1m5.581s
   user 0m2.308s
   sys 0m3.760s

6. real 1m1.893s
   user 0m2.288s
   sys 0m3.896s

samba ext4:

fallocate does not work, file is not filled with nulls

unrar original:

1. real 1m0.321s
   user 0m3.264s
   sys 0m3.088s

2. real 0m50.765s
   user 0m2.072s
   sys 0m4.104s

3. real 0m49.819s
   user 0m2.736s
   sys 0m3.828s

4. real 0m50.443s
   user 0m2.488s
   sys 0m4.188s

5. real 0m49.070s
   user 0m2.196s
   sys 0m4.324s

6. real 0m49.121s
   user 0m2.164s
   sys 0m3.852s

unrar fallocate:

1. real 0m59.154s
   user 0m4.080s
   sys 0m3.344s

2. real 0m49.171s
   user 0m2.144s
   sys 0m4.232s

3. real 0m48.418s
   user 0m2.292s
   sys 0m3.804s

4. real 0m50.506s
   user 0m2.196s
   sys 0m3.848s

5. real 0m48.666s
   user 0m2.440s
   sys 0m3.952s

6. real 0m48.819s
   user 0m2.380s
   sys 0m3.920s

samba ntfs:

fallocate does not work, file is not filled with nulls

unrar original:

1. real 1m47.614s
   user 0m4.420s
   sys 0m3.872s

2. real 1m46.052s
   user 0m4.532s
   sys 0m3.856s

3. real 1m29.882s
   user 0m2.088s
   sys 0m4.932s

4. real 1m30.453s
   user 0m2.184s
   sys 0m5.408s

5. real 1m29.219s
   user 0m2.072s
   sys 0m5.784s

6. real 1m30.863s
   user 0m2.284s
   sys 0m5.204s

unrar fallocate:

1. real 1m29.152s
   user 0m2.244s
   sys 0m5.380s

2. real 1m28.854s
   user 0m2.372s
   sys 0m5.176s

3. real 1m28.314s
   user 0m2.176s
   sys 0m5.492s

4. real 1m26.468s
   user 0m2.024s
   sys 0m4.968s

5. real 1m26.793s
   user 0m1.912s
   sys 0m6.008s

6. real 1m27.311s
   user 0m2.836s
   sys 0m4.412s

BTW who use ntfs in Unix? As said in first comment unrar for Windows preallocs the drive space by default.

[hugbug]

Great testing. Thank you. I'll try to activate fallocate for my unrar build.

BTW who use ntfs in Unix?

The use case is accessing a Windows machine via samba.

[itouch5000]

All right, thank you too.

[hugbug]

The next nzbget version will include unrar with fallocate.

[hugbug]

Would you like to try (test) the current version before I release it?

If you do - please send me a note to nzbget@gmail.com and I'll send you current installer package. Let me also know your CPU architecture.

[itouch5000]

Looks good. Fallocate works and I can play videos while downloading in Kodi now.

In my tests Kodi also behaves bad on Windows in combination with direct unpack as it blocks files from being moved/renamed.

After this test I can't agree that Kodi blocks files from being moved/renamed. After the half of video nzbget has moved the file and I was able to watch the video completely (Kodi hasn't already buffered the video).

[hugbug]

Thanks for testing.

That's nice Kodi on Linux doesn't prevent the file from being renamed/moved. On Windows it's a different case however. First it opens files in a share mode preventing renaming/moving. Second if file is opened via Kodi's file browser the directory (_unpack) can not be deleted. Windows media player doesn't have these problems though.

As for fallocate. For some reason on my Linux box it doesn't work. A test with console command fallocate reports:

fallocate: fallocate failed: Operation not supported

Although it's ext4 partition and kernel 4.10.

[itouch5000]

I have tested with Kodi 17 on Linux and Kodi 16 on Windows both worked great here. But Kodi 17 on Windows stopped video playback after nzbget moved the file, strange...
I have no idea why fallocate does not work for you. The kernel is even newer than mine (4.4).
Maybe an custom unpack path in the nzbget options is a way to prevent move the file after unpacking.

[hugbug]

When you talk about Kodi on Windows do you playback files downloaded by nzbget on Windows?

Fallocate started to work for me after I reformatted the drive to ext4. It turned out the drive was in ext3 although mount reported ext4. I'm doing more tests now.

I have a report from nzbget user who tried unpack with fallocate and had a system lockup. After googling I've found a few references where fallocate caused system lockups.

I'm looking further into this to decide whether I should revert to default unrar.

[itouch5000]

When you talk about Kodi on Windows do you playback files downloaded by nzbget on Windows?

No, in all tests nzbget downloads the file in Linux. Kodi on Windows connects via samba to this Linux machine.

[tc60045]

Hugbug, any testing done (or needed) for ZFS?

[hugbug]

@toddcurry, I believe no one has tested it yet.
That would be nice if you can.