Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile unrar for Linux with fallocate #379

Closed
itouch5000 opened this issue May 16, 2017 · 22 comments

Comments

Projects
None yet
3 participants
@itouch5000
Copy link

commented May 16, 2017

Thank you for the great new features in 19.0-testing-r1991. But I have a small problem with DirectUnpack in Unix in combination with Kodi as video player.

Unrar for Windows preallocs the drive space for extracted files to reduce fragmentation. In Unix the appropriate code is present in unrar source, but not enabled yet, because fallocate call was not
available on older Linux kernels. So in Unix the file size increase with each extracted part. VLC can play these videos but Kodi for example stops playback after a few minutes (after the length of the video at the time of started the playback).
How to fix that: itouch5000/unrarstream@2faa08e

@hugbug

This comment has been minimized.

Copy link
Member

commented May 16, 2017

NZBGet doesn't require any unrar patches and works fine with stock unrar. However you are free to use whatever version of unrar you prefer.

I'm not sure what do you mean with this issue. Are you suggesting to activate fallocate in the unrar shipped with nzbget? I don't know if I can do that as it may make nzbget installer incompatible with older Linux versions. Currently installer works with kernel 2.6.something whereas I don't know how low "something" can be.

Wouldn't fallocating kill unpack perfomance as it needs to fill the whole file with nulls?
Have you measured unrar with and without fallocate?

I wonder why do you modify unrar source instead of using proper compiler flags as intended by unrar author.

BTW if Kodi can't play partially downloaded files isn't that something to report to Kodi team? In my tests Kodi also behaves bad on Windows in combination with direct unpack as it blocks files from being moved/renamed. Windows media player works very well in this regard.

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 17, 2017

Yes I am suggesting to activate fallocate in the unrar shipped with nzbget. But if it drops the compatibility I will try to report this issue to Kodi team. I thought that it's the easiest method to solve this problem and support more video player.

Wouldn't fallocating kill unpack perfomance as it needs to fill the whole file with nulls?
Have you measured unrar with and without fallocate?

I have no idea how to measure that. Which program should I use for this test?

I wonder why do you modify unrar source instead of using proper compiler flags as intended by unrar author.

Sorry I'm a noob and I have never used a compiler flag. I have written with the unrar author via email some time ago and he recommended me to change his line in source before compile unrar. Nothing about a compiler flag.
Is it possible to compile nzbget with this unrar compiler flag to use it as long as Kodi is not able to play these files?

@hugbug

This comment has been minimized.

Copy link
Member

commented May 17, 2017

Is it possible to compile nzbget with this unrar compiler flag

Unrar and nzbget are two different binaries. You can compile unrar yourself and use it with nzbget.

Nothing about a compiler flag.
The unrar code from your commit checks symbol USE_FALLOCATE. If you define the symbol in compiler flags then you don't need to change sources of unrar.

You can pass it via CLFAGS (I haven't tried this but it should work):

CFLAGS=-DUSE_FALLOCATE make

Or you can put extra flags into unrar's makefile, which is supposed to be changed by user to adjust to system (added -DUSE_FALLOCATE):

DEFINES=-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DRAR_SMP -DUSE_FALLOCATE

I have no idea how to measure that. Which program should I use for this test?

Since you compile unrar yourself: make two versions with and without fallocate. Then try unpacking the same archive with both unrars, at best do the test multiple times and unpack into convenient HDD. You can use command time to measure run time:

time unrar x archive.rar
@itouch5000

This comment has been minimized.

Copy link
Author

commented May 17, 2017

Thank you very much for the detailed explanation! I will make some tests and report the results.

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 18, 2017

CFLAGS=-DUSE_FALLOCATE make does not work for me. But append the flag to unrars makefile works great.
I have made 12 tests (6 for each executable). In the order "1. unrar without fallocate", "1. unrar with fallocate", "2. unrar without fallocate", "2. unrar with fallocate"... So you can compare them directly.
The result is that unrar with fallocate is even faster than unrar without fallocate.

unrar without fallocate:

  1. real 0m50.048s
    user 0m9.616s
    sys 0m10.280s
  2. real 0m50.435s
    user 0m10.280s
    sys 0m9.684s
  3. real 1m3.102s
    user 0m9.660s
    sys 0m11.580s
  4. real 0m59.830s
    user 0m9.752s
    sys 0m11.232s
  5. real 0m54.594s
    user 0m9.608s
    sys 0m10.400s
  6. real 0m55.073s
    user 0m9.224s
    sys 0m11.044s

unrar with fallocate:

  1. real 0m49.018s
    user 0m10.764s
    sys 0m7.900s
  2. real 0m49.448s
    user 0m10.916s
    sys 0m7.720s
  3. real 0m52.695s
    user 0m11.068s
    sys 0m7.588s
  4. real 0m53.896s
    user 0m11.960s
    sys 0m6.540s
  5. real 0m53.670s
    user 0m11.776s
    sys 0m6.580s
  6. real 0m54.016s
    user 0m12.396s
    sys 0m5.872s
@hugbug

This comment has been minimized.

Copy link
Member

commented May 18, 2017

Nice tests!

HDD or SSD?
What is the filesystem?
How big the test achive?

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 18, 2017

First I have made some tests on my Desktop PC (Linux Mint 18.1) with an SSD. But the measured times were too different. Still I have seen, that fallocate is faster than without. (If you are interested in the tests I can post them too)

The tests above I have made with my little homeserver (CPU while unpack ~30%) and extracted the archive to an HDD because I have no HDD in my Desktop PC.
Ubuntu Server 16.04.2 LTS (GNU/Linux 4.4.0-75-generic x86_64)
Filesystem is ext4
Size of archive.rar is 4,7 GB (4.667.572.901 Bytes)

@hugbug

This comment has been minimized.

Copy link
Member

commented May 18, 2017

The docs for fallocate say it fills the file with nulls. That must take time, right? I can imagine ext4 may have special support for that like marking blocks nullified without actual writing to disk.

Do you have a chance to test with other file systems? Ext3 on local drive and ext3/ext4 via samba (NAS) or maybe even ntfs via samba.

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 18, 2017

Yes, theoretically it has to cost time. But I'm not an expert :D
I have some HDDs laying around and can make these tests the next days.

  • ext4 on local hdd
  • ext3 on local hdd
  • ext4 via samba
  • ext3 via samba
  • ntfs via samba
@hugbug

This comment has been minimized.

Copy link
Member

commented May 18, 2017

There is fallocate system call and fallocate console utility. The man page for the utility documents which file systems it supports (only a few actually, including ext4 but not ext3). The man page for system call doesn't mention file systems.

I wonder what will happen on a file system which has no special support: will the call fail or will it write the nulls. In the first case the file size will not be set to desired size (you should be able to control that). In the second case it should take long time writing all these nulls.

Also intersring is if the ext4 accessed via network can still take advantage or the network protocol will prevent optimizations.

In our case I would prefer fallocate quickly to fail on unsupported file systems instead of writing nulls and doubling the unpack time.

Thanks for testing this.

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 19, 2017

local ext3:

fallocate does not work, file is not filled with nulls

unrar original:

  1. real 0m54.634s
    user 0m9.192s
    sys 0m14.708s

  2. real 0m51.775s
    user 0m9.140s
    sys 0m14.704s

  3. real 0m53.301s
    user 0m9.248s
    sys 0m14.820s

  4. real 0m52.774s
    user 0m8.984s
    sys 0m14.924s

  5. real 0m53.480s
    user 0m9.168s
    sys 0m15.192s

  6. real 0m52.349s
    user 0m9.160s
    sys 0m14.948s

unrar fallocate:

  1. real 0m51.729s
    user 0m9.216s
    sys 0m14.632s

  2. real 0m49.967s
    user 0m9.216s
    sys 0m14.880s

  3. real 0m52.664s
    user 0m9.176s
    sys 0m15.028s

  4. real 0m53.940s
    user 0m9.060s
    sys 0m15.116s

  5. real 0m54.939s
    user 0m9.116s
    sys 0m15.016s

  6. real 0m53.952s
    user 0m9.268s
    sys 0m14.720s

local ext4:

fallocate works, file is filled with nulls

unrar original:

  1. real 0m41.577s
    user 0m9.516s
    sys 0m9.584s

  2. real 1m36.869s
    user 0m11.280s
    sys 0m14.492s

  3. real 0m43.929s
    user 0m9.244s
    sys 0m9.884s

  4. real 0m42.626s
    user 0m9.320s
    sys 0m10.020s

  5. real 0m41.864s
    user 0m9.640s
    sys 0m9.644s

  6. real 0m41.444s
    user 0m9.616s
    sys 0m9.556s

unrar fallocate:

  1. real 0m41.614s
    user 0m9.600s
    sys 0m8.312s

  2. real 1m0.009s
    user 0m10.084s
    sys 0m9.884s

  3. real 0m42.756s
    user 0m9.460s
    sys 0m8.140s

  4. real 0m41.972s
    user 0m9.444s
    sys 0m8.564s

  5. real 0m41.505s
    user 0m9.952s
    sys 0m7.764s

  6. real 0m41.495s
    user 0m10.296s
    sys 0m7.488s

local ntfs:

fallocate does not work, file is not filled with nulls

unrar original:

  1. real 1m25.394s
    user 0m10.308s
    sys 0m15.088s

  2. real 1m25.013s
    user 0m10.156s
    sys 0m15.136s

  3. real 1m24.999s
    user 0m10.052s
    sys 0m15.328s

  4. real 1m25.005s
    user 0m10.448s
    sys 0m15.000s

  5. real 1m25.256s
    user 0m10.492s
    sys 0m14.856s

  6. real 1m25.328s
    user 0m10.252s
    sys 0m15.116s

unrar fallocate:

  1. real 1m28.696s
    user 0m9.976s
    sys 0m15.336s

  2. real 1m28.053s
    user 0m9.840s
    sys 0m15.524s

  3. real 1m28.161s
    user 0m10.236s
    sys 0m15.108s

  4. real 1m28.118s
    user 0m9.864s
    sys 0m15.420s

  5. real 1m28.221s
    user 0m9.824s
    sys 0m15.432s

  6. real 1m28.367s
    user 0m9.780s
    sys 0m15.640s

samba ext3:

fallocate does not work, file is not filled with nulls

unrar original:

  1. real 1m5.491s
    user 0m2.288s
    sys 0m4.300s

  2. real 1m5.003s
    user 0m2.556s
    sys 0m3.788s

  3. real 1m1.285s
    user 0m2.476s
    sys 0m3.892s

  4. real 1m2.796s
    user 0m2.204s
    sys 0m3.956s

  5. real 1m4.359s
    user 0m2.368s
    sys 0m4.196s

  6. real 1m1.211s
    user 0m2.444s
    sys 0m3.536s

unrar fallocate:

  1. real 1m4.110s
    user 0m2.296s
    sys 0m4.140s

  2. real 1m4.851s
    user 0m2.388s
    sys 0m4.140s

  3. real 1m3.523s
    user 0m2.376s
    sys 0m4.048s

  4. real 1m3.358s
    user 0m2.544s
    sys 0m3.924s

  5. real 1m5.581s
    user 0m2.308s
    sys 0m3.760s

  6. real 1m1.893s
    user 0m2.288s
    sys 0m3.896s

samba ext4:

fallocate does not work, file is not filled with nulls

unrar original:

  1. real 1m0.321s
    user 0m3.264s
    sys 0m3.088s

  2. real 0m50.765s
    user 0m2.072s
    sys 0m4.104s

  3. real 0m49.819s
    user 0m2.736s
    sys 0m3.828s

  4. real 0m50.443s
    user 0m2.488s
    sys 0m4.188s

  5. real 0m49.070s
    user 0m2.196s
    sys 0m4.324s

  6. real 0m49.121s
    user 0m2.164s
    sys 0m3.852s

unrar fallocate:

  1. real 0m59.154s
    user 0m4.080s
    sys 0m3.344s

  2. real 0m49.171s
    user 0m2.144s
    sys 0m4.232s

  3. real 0m48.418s
    user 0m2.292s
    sys 0m3.804s

  4. real 0m50.506s
    user 0m2.196s
    sys 0m3.848s

  5. real 0m48.666s
    user 0m2.440s
    sys 0m3.952s

  6. real 0m48.819s
    user 0m2.380s
    sys 0m3.920s

samba ntfs:

fallocate does not work, file is not filled with nulls

unrar original:

  1. real 1m47.614s
    user 0m4.420s
    sys 0m3.872s

  2. real 1m46.052s
    user 0m4.532s
    sys 0m3.856s

  3. real 1m29.882s
    user 0m2.088s
    sys 0m4.932s

  4. real 1m30.453s
    user 0m2.184s
    sys 0m5.408s

  5. real 1m29.219s
    user 0m2.072s
    sys 0m5.784s

  6. real 1m30.863s
    user 0m2.284s
    sys 0m5.204s

unrar fallocate:

  1. real 1m29.152s
    user 0m2.244s
    sys 0m5.380s

  2. real 1m28.854s
    user 0m2.372s
    sys 0m5.176s

  3. real 1m28.314s
    user 0m2.176s
    sys 0m5.492s

  4. real 1m26.468s
    user 0m2.024s
    sys 0m4.968s

  5. real 1m26.793s
    user 0m1.912s
    sys 0m6.008s

  6. real 1m27.311s
    user 0m2.836s
    sys 0m4.412s

BTW who use ntfs in Unix? As said in first comment unrar for Windows preallocs the drive space by default.

@hugbug

This comment has been minimized.

Copy link
Member

commented May 20, 2017

Great testing. Thank you. I'll try to activate fallocate for my unrar build.

BTW who use ntfs in Unix?

The use case is accessing a Windows machine via samba.

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 20, 2017

All right, thank you too.

@hugbug hugbug changed the title DirectUnpack improvement Compile unrar for Linux with fallocate May 20, 2017

hugbug added a commit that referenced this issue May 20, 2017

@hugbug

This comment has been minimized.

Copy link
Member

commented May 20, 2017

The next nzbget version will include unrar with fallocate.

@hugbug hugbug closed this May 20, 2017

@hugbug

This comment has been minimized.

Copy link
Member

commented May 20, 2017

Would you like to try (test) the current version before I release it?

If you do - please send me a note to nzbget@gmail.com and I'll send you current installer package. Let me also know your CPU architecture.

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 20, 2017

Looks good. Fallocate works and I can play videos while downloading in Kodi now.

In my tests Kodi also behaves bad on Windows in combination with direct unpack as it blocks files from being moved/renamed.

After this test I can't agree that Kodi blocks files from being moved/renamed. After the half of video nzbget has moved the file and I was able to watch the video completely (Kodi hasn't already buffered the video).

@hugbug

This comment has been minimized.

Copy link
Member

commented May 20, 2017

Thanks for testing.

That's nice Kodi on Linux doesn't prevent the file from being renamed/moved. On Windows it's a different case however. First it opens files in a share mode preventing renaming/moving. Second if file is opened via Kodi's file browser the directory (_unpack) can not be deleted. Windows media player doesn't have these problems though.

As for fallocate. For some reason on my Linux box it doesn't work. A test with console command fallocate reports:

fallocate: fallocate failed: Operation not supported

Although it's ext4 partition and kernel 4.10.

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 20, 2017

I have tested with Kodi 17 on Linux and Kodi 16 on Windows both worked great here. But Kodi 17 on Windows stopped video playback after nzbget moved the file, strange...
I have no idea why fallocate does not work for you. The kernel is even newer than mine (4.4).
Maybe an custom unpack path in the nzbget options is a way to prevent move the file after unpacking.

@hugbug

This comment has been minimized.

Copy link
Member

commented May 21, 2017

When you talk about Kodi on Windows do you playback files downloaded by nzbget on Windows?

Fallocate started to work for me after I reformatted the drive to ext4. It turned out the drive was in ext3 although mount reported ext4. I'm doing more tests now.

I have a report from nzbget user who tried unpack with fallocate and had a system lockup. After googling I've found a few references where fallocate caused system lockups.

I'm looking further into this to decide whether I should revert to default unrar.

@hugbug hugbug reopened this May 21, 2017

@itouch5000

This comment has been minimized.

Copy link
Author

commented May 21, 2017

When you talk about Kodi on Windows do you playback files downloaded by nzbget on Windows?

No, in all tests nzbget downloads the file in Linux. Kodi on Windows connects via samba to this Linux machine.

@tc60045

This comment has been minimized.

Copy link

commented May 23, 2017

Hugbug, any testing done (or needed) for ZFS?

@hugbug

This comment has been minimized.

Copy link
Member

commented May 23, 2017

@toddcurry, I believe no one has tested it yet.
That would be nice if you can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.