Can't read file (over 4G?)

[hasse69]

What steps will reproduce the problem?

1. Create 16GiB blank file with truncate.
    # truncate -s 16G test

2. Compress this file with WinRAR 5.00. I used following options:
    -cfg- -ep1 -htb -m5 -ma5 -mcD -md256m -ms- -mt1 -oi:0 -qo- -scu -s-

3. Mount archive with rar2fs.

4. Try to read compressed file with dd.
    # dd if=(mounted path)/test of=/dev/null bs=8k count=32

What is the expected output? What do you see instead?

     - It must be read data correctly. But, it hangs. dd won't kill even sends SIGKILL. rar2fs process can be kill only with SIGKILL. dd reports 0 bytes readed.

What version of the product are you using? On what operating system?

    fuse 2.9.3 (openSUSE 13.1 RC1, x86_64), unrar 5.0.12, rar2fs 1.18.0

Please provide any additional information below.

     - It seems occur with archive which has very large archived file offset. I'll attach example file which is explained above.

    p.s. I'm sorry about my poor english.

Original issue reported on code.google.com by jyhpsycho on 2013-11-04

[hasse69]

Thanks for the attached file. I will try to reproduce this locally.
In general rar2fs should have no issues with large file support.
Something behaves strange also if you try 'unrar t test.rar'. Testing almost immediately
progress to 99% but then get stuck for a very long time. It completes but it takes
a lot longer than expected.

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 13:46:26

[hasse69]

I am having problems reproducing this issue (on SUSE11) :(

>rar2fs issue19 d
>cd d
>ls -l
ls -l
total 16777220
-rw-rw-r-- 1 foobar foobar 17179869184 Nov  4 13:22 test.blank
>dd if=test.blank of=/dev/null bs=8k count=32
32+0 records in
32+0 records out
262144 bytes (262 kB) copied, 0.00018429 s, 1.4 GB/s

Can you try eg. this instead:
>hexdump -C -n 64 test.blank

Just want to see if it makes a difference or not.

I might need some more information about exactly how you mounted it (full set of arguments
etc.) I do not have a system with fuse 2.9 so currently I am unable to test if this
could be a fuse issue or not. Do you have some other machine with a different configuration?

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 14:04:28

[hasse69]

Can you also please provide the output from

>rar2fs -V

And a copy of your config.log.

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 14:12:06

[hasse69]

> Something behaves strange also if you try 'unrar t test.rar'. Testing almost immediately
progress to 99% but then get stuck for a very long time. It completes but it takes
a lot longer than expected.

It's normal. Because compression ratio(compressed size / original size) is almost 0.
Criteria of WinRAR's percentage indicator is archive's file pointer(including recovery
record), not the uncompressed data size. Testing archive that have 10% RR, you can
see extraction process is done at 90%.



Explanation of whole steps:

Working directory : /dev/shm/test

1. Prepare unrar library. I used manually-made static version of libunrar to prevent
shared library path problems.
# tar -xf unrarsrc-5.0.12.tar.gz
# cd unrar
# make lib
# ar -r libunrar.a rar.o strlist.o strfn.o pathfn.o smallfn.o global.o file.o filefn.o
filcreat.o archive.o arcread.o unicode.o system.o isnt.o crypt.o crc.o rawread.o encname.o
resource.o match.o timefn.o rdwrfn.o consio.o options.o errhnd.o rarvm.o secpassword.o
rijndael.o getbits.o sha1.o sha256.o blake2s.o hash.o extinfo.o extract.o volume.o
list.o find.o unpack.o headers.o threadpool.o rs16.o cmddata.o filestr.o scantree.o
dll.o qopen.o
# rm libunrar.so
# cd ..

 - I copied command line that create shared object printed by make, and just modified
command to create static library.

2. Compile rar2fs.
# tar -xf rar2fs-1.18.0.tar.gz
# cd rar2fs-1.18.0
# LIBS='-lstdc++ -lpthread' ./configure --with-unrar=../unrar --with-unrar-lib=../unrar
# make
# cd ..

 - LIBS variable is required because currently used unrar library is static. Without
this, configure script failed detection of unrar. With shared version of libunrar,
it doesn't occur.

3. Mount filesystem and read data

# mkdir a
# rar2fs-1.18.0/rar2fs test.rar a
# dd if=a/test.blank of=/dev/null bs=8k count=32

It hangs. For terminating this process, I tried following:

# ps aux | grep rar
root     26945  0.0  0.0 287612  6604 ?        Ssl  01:23   0:00 rar2fs-1.18.0/rar2fs
test.rar a
root     26951  0.0  0.5 557612 276292 ?       S    01:24   0:00 rar2fs-1.18.0/rar2fs
test.rar a
root     27012  0.0  0.0  14432  1028 pts/5    S+   01:24   0:00 grep --color=auto
rar
# kill -9 26951

After sending SIGKILL, dd outputs:

0+0 records in
0+0 records out
0 bytes (0 B) copied, 36.4623 s, 0.0 kB/s

# hexdump -C -n 64 test.blank

It hangs, too. After kill rar2fs, it doesn't print anything.


# uname -a
Linux localhost 3.11.3-1-desktop #1 SMP PREEMPT Wed Oct 2 05:54:02 UTC 2013 (485f0d1)
x86_64 x86_64 x86_64 GNU/Linux

# rar2fs-1.18.0/rar2fs -V
rar2fs v1.18.0 (DLL version 6)    Copyright (C) 2009-2013 Hans Beckerus
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under
certain conditions; see <http://www.gnu.org/licenses/> for details.
FUSE library version: 2.9.3
fusermount version: 2.9.3
using FUSE kernel interface version 7.19



I'll attach my config.log. I'm not have another environment that can be test it yet,
but it will be possible soon via VM...

Original issue reported on code.google.com by jyhpsycho on 2013-11-04 16:36:51

---

- _Attachment: [config.log](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-19/comment-4/config.log)_

[hasse69]

I used shared version of libunrar, too. Only difference is below:

# LD_LIBRARY_PATH=unrar rar2fs-1.18.0/rar2fs test.rar a

Results are same. It hangs. It causes hang all process which accesses mounted path,
Even bash and xfce-terminal when press Tab key to use auto-complete!



I'll attach config.log with shared library, too.

Original issue reported on code.google.com by jyhpsycho on 2013-11-04 16:51:54

---

- _Attachment: [config.log](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-19/comment-5/config.log)_

[hasse69]

First of all, why would you need to build statically and why would not doing so result
in failures in detecting libunrar.so? If you already have an old version of libunrar.so
installed, all you have to do to override it at compile/configure time is by using
--with-unrar-lib=<path>. Then, of course, you need to be careful when running it since
rar2fs will pick up the library as provided by the system unless you override it using
eg. LD_LIBRARY_PATH.
Secondly, can you please try to not mount the file but rather the folder that contains
it? I think I managed to reproduce the problem when mounting the file only.
So try moving test.rar into eg. a folder named s and try:

>rar2fs s a

I am almost certain that it will work. I need to look into why mounting the file rather
than the folder does not work in this particular case.

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 17:05:15

[hasse69]

1. Yes, I tested case of shared library, too. It mentioned in #5. And, it should be
work regardless unrar library is shared or static if it compiled sucessfully. Both
cases --with-unrar and --with-unrar-lib option is required.

2. What's difference between my mount command and yours?
# rar2fs-1.18.0/rar2fs test.rar a
> rar2fs s a

Only difference is executive binary path, not archive.

Original issue reported on code.google.com by jyhpsycho on 2013-11-04 17:29:28

[hasse69]

l. Yes, --with-unrar tell where to find the header files (source), --with-unrar-lib
specifies where to search for libunrar.so besides the default system paths.

2. It is very different. rar2fs can mount in two different ways, as a file or as a
directory. What you are doing is mounting one (1) single .rar file. What I was asking
you to do is to mount a directory (containing test.rar) instead.
To be more precise, try making two folders; a1 and a2. Then copy test.rar to a1 and
mount a1 on a2.

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 17:53:41

[hasse69]

I have done some basic trouble shooting and this looks like a tricky problem :( Not
something I will manage to resolve quickly I am afraid. For some reason the communication
between the process extracting data and the thread waiting for it does not work when
using file mount. But only for this file? I have lots of other files for which file
mount work without problem. 

Can you create one more *small* file (e.g. 1GB) using the same method and attach it
here. I need to tell if it is the size of the file or something else that is causing
this. 

Also, please try the directory mount instead to see if it can be used as a temporary
workaround.

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 18:28:26

[hasse69]

I know a bit more about the problem now, but still I do not have a solution.
It seems to be related to RAR5 archives only. I can reproduce this problem easily using
any RAR5 archive. For non-RAR5 archives it works just fine!? So you do not need to
create another file. It is really weird since mounting a folder containing the file
instead of mounting the file itself also works without problems!?

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 20:36:21

[hasse69]

Got it (I think)! It was a missing close of a file handle before freeing some resources
that resulted in some major breakage in libunrar. Only seems to affect RAR5 archives
though which is most likely the reason why this error has passed unnoticed. I will
commit a new version to trunk. Can you test latest version in SVN to verify if the
problem is resolved or not in your environment?

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 22:05:59

[hasse69]

Yes, but I can test svn build 12 hours later, due to work... :-(

I'm sorry about misunderstanding your suggestion. I don't know it is possible to mount
directory containing RAR file... ;-)

Original issue reported on code.google.com by jyhpsycho on 2013-11-04 23:26:23

[hasse69]

No problem. No need to rush it ;) I am confident that it will work for you this time.
And yes, it is possible to mount a directory containing not only one (1) RAR file,
but many, in addition to normal files as well, completely transparent.

Original issue reported on code.google.com by hans.beckerus on 2013-11-04 23:44:24

[hasse69]

Good, svn version works! Thanks.

Original issue reported on code.google.com by jyhpsycho on 2013-11-05 11:52:29

[hasse69]

Great! Thanks for reporting the issue and taking your time testing the patch!

Case closed.

Original issue reported on code.google.com by hans.beckerus on 2013-11-05 12:18:20