Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ls failed on device-stored archive. #28

Closed
hasse69 opened this issue Mar 13, 2015 · 57 comments
Closed

ls failed on device-stored archive. #28

hasse69 opened this issue Mar 13, 2015 · 57 comments
Labels
Defect Priority-Medium

Comments

@hasse69
Copy link
Owner

hasse69 commented Mar 13, 2015

What steps will reproduce the problem?

Just execute ls. Testing condition below.

What is the expected output? What do you see instead?

That should be print archive content. But it says :

ls: reading directory TEST/: Software caused connection abort

What version of the product are you using? On what operating system?

openSUSE 13.1 x86-64, unrar 5.0.14

# rar2fs -V
rar2fs v1.19.8 (DLL version 6)    Copyright (C) 2009-2014 Hans Beckerus
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under
certain conditions; see <http://www.gnu.org/licenses/> for details.
FUSE library version: 2.9.3
fusermount version: 2.9.3
using FUSE kernel interface version 7.19

Please provide any additional information below.

I created some hard disk partition to store archive file itself. Its size is about 130GiB, contains 6 files which sizes are 20 ~ 23 GiB and some RR. Linux treats partition device almost same as regular file, thus this should be work.

Instruction step :

  1. mount partition.
# rar2fs -o ro,allow_other /dev/sdb2 TEST/
  1. run "ls -l TEST/".

Debug messages:

    mounting file system on /dev/shm/WORK/TEST
    unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
    INIT: 7.22
    flags=0x0000f7fb
    max_readahead=0x00020000
    rar2_init()   
       INIT: 7.19
       flags=0x00000010
       max_readahead=0x00020000
       max_write=0x00020000
       max_background=0
       congestion_threshold=0
       unique: 1, success, outsize: 40
    unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 17161
    getattr /
    rar2_getattr2()   /
    MISS    /
    STAT retrieved for /dev/
       unique: 2, success, outsize: 120
    unique: 3, opcode: GETXATTR (22), nodeid: 1, insize: 65, pid: 17161
    getxattr / security.selinux 255
    rar2_getxattr()   /
       unique: 3, error: -61 (No data available), outsize: 16
    unique: 4, opcode: GETXATTR (22), nodeid: 1, insize: 72, pid: 17161
    getxattr / system.posix_acl_access 0
    rar2_getxattr()   /
       unique: 4, error: -61 (No data available), outsize: 16
    unique: 5, opcode: GETXATTR (22), nodeid: 1, insize: 73, pid: 17161
    getxattr / system.posix_acl_default 0
    rar2_getxattr()   /
       unique: 5, error: -61 (No data available), outsize: 16
    unique: 6, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 17161
    opendir flags: 0x18800 /
    rar2_opendir()   /
       opendir[140122905549680] flags: 0x18800 /
       unique: 6, success, outsize: 32
    unique: 7, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 17161
    readdir[140122905549680] from 0
    rar2_readdir2()   
    listrar()   /   arch=/dev/sdb2
    Looking up /Vol.1.rar in cache
    Adding /Vol.1.rar to cache
    23501538393 byte RAR file /Vol.1.rar found in archive /dev/sdb2
    Looking up /Vol.2.rar in cache
    Adding /Vol.2.rar to cache
    22110405466 byte RAR file /Vol.2.rar found in archive /dev/sdb2
    Got signal 11, faulty address is 0x0x7f766826b750, from 0x0x4188c0
    ../rar2fs-1.19.8/rar2fs() [0x4188c0]
    /lib64/libpthread.so.0(+0xf9f0) [0x7f70ee84d9f0]
    /lib64/libc.so.6(strnlen+0x44) [0x7f70ee5120c4]
    /lib64/libc.so.6(fmemopen+0x6c) [0x7f70ee50335c]
    ../rar2fs-1.19.8/rar2fs() [0x41d086]
    ../rar2fs-1.19.8/rar2fs() [0x41ea48]
    ../rar2fs-1.19.8/rar2fs() [0x420c59]
    ../rar2fs-1.19.8/rar2fs(fuse_fs_readdir+0x82) [0x42bc42]
    ../rar2fs-1.19.8/rar2fs() [0x42bdcc]
    ../rar2fs-1.19.8/rar2fs() [0x4326f6]
    ../rar2fs-1.19.8/rar2fs() [0x4338a6]
    ../rar2fs-1.19.8/rar2fs(fuse_session_loop+0xac) [0x4302dc]
    ../rar2fs-1.19.8/rar2fs(fuse_loop+0x1e8) [0x428928]
    ../rar2fs-1.19.8/rar2fs() [0x424f21]
    /lib64/libpthread.so.0(+0x80db) [0x7f70ee8460db]
    /lib64/libc.so.6(clone+0x6d) [0x7f70ee57690d]
    [*] Process terminated at this point.

I think it causes some memory crash...

There's no problem when mounting file in filesystem - I mean mount with "rar2fs -o ro,allow_other Vols.rar TEST/".

Original issue reported on code.google.com by jyhpsycho on 2014-02-21 11:39:03
@hasse69 hasse69 closed this as completed Mar 13, 2015
@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I just copied debug messages. But, there's some typo on debug message I think... Can
you fix also that messages?

 - Got signal 11, faulty address is "0x0x"7f766826b750, from "0x0x"4188c0

Original issue reported on code.google.com by jyhpsycho on 2014-02-21 12:10:26

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I will fix the debug message. But for the actual problem, I have never ever tested mounting
a RAR archive like a device file. Why would you want to do that? If I can detect this
in some easy way I think I will add some code to throw an error in this case and refuse
the mount from happening.

Original issue reported on code.google.com by hans.beckerus on 2014-02-21 15:21:09

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Wooh! Are the volume files >20GB each?
Can you please provide a dump of the contents of Vols.rar? 
How did you get/dev/sdb2 to contain Vols.rar?

Original issue reported on code.google.com by hans.beckerus on 2014-02-21 15:33:14

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

1. I used that for storing many multimedia datas. I experienced some serious problems
when using traditional filesystems such as NTFS, ext2/3/4, jfs, xfs which is vulnerable
to breaking some data especially metadata. I think that storing RAR archive in partition
may an alternative to storing data.
Under *nix, disk partition data is accessible with similar interface to regular file.
That means, I can access archived data via rar2fs, or directly extract that to another
directory. RAR format has great feature for recovering partial damaged data, too. It
requires some traditional filesystem stroage to recover data in practice, though.

2. File lists are in issue 29.

3. dd if=Vols.rar of=/dev/sdb2 bs=4M

Original issue reported on code.google.com by jyhpsycho on 2014-02-21 15:50:10

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Ok, will have some hard time reproducing this though :(
But it is strange that it crashes! You said it worked if you mounted it as a single
archive on local file system, right? Can not see what would be the difference.

Original issue reported on code.google.com by hans.beckerus on 2014-02-21 16:24:13

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Yes, it works when mount using original single file archive. I think that's very strange,
too...

I'll test if it occurs with smaller file. When it occurs, I'll report that and attach
that example for you.

Original issue reported on code.google.com by jyhpsycho on 2014-02-21 16:53:48

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I can't reproduce it with small files... Now I creating 16GB random file for test.

Original issue reported on code.google.com by jyhpsycho on 2014-02-21 17:19:47

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Yea, I tried a simple setup using /dev/ram but that could not reproduce the crash :(

Original issue reported on code.google.com by hans.beckerus on 2014-02-21 20:12:03

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Try to reproduce the problem using the latest version in trunk (1.19.10). I discovered
a missing check for error thrown by fmemopen(). It might help, but this is really a
long shot. Would be great if you could reproduce the problem and run rar2fs through
gdb for a proper stack trace.

Original issue reported on code.google.com by hans.beckerus on 2014-02-21 21:32:24

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I finally can reproduce similar problem with following instructions:

1) Create dummy file with following command:
# truncate -s 64K '[TEST] Vol.1.rar'
# truncate -s 64K '[TEST] Vol.2.rar'
# truncate -s 64K '[TEST] Vol.3.rar'

2) Create archive contains dummy file created before.
   I used alias rar="wine 'C:\Program Files\WinRAR\Rar.exe'".
   I'm not tested with Linux version, but I guess that's not important for this case.
# rar a -ma5 -m0 -ep1 TEST.rar \[TEST\]\ Vol.*

3) Adjust size of archive file for mounting as loop device since loop device requires
512-byte-aligned file size. Following command rounds up archive file size.
# truncate -s $((($(stat -c '%s' TEST.rar) + 511) & -512)) TEST.rar

4) Set up loop device. In my case that assigned to /dev/loop0.
# losetup -f TEST.rar

5) Mount loop device created before.
# rar2fs -o ro,allow_other,debug /dev/loop0 TEST

6) Run "ls -la TEST/". That crashes.



I don't know why it says 500 Error when attaching example file...

Original issue reported on code.google.com by jyhpsycho on 2014-02-22 05:16:50

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Thanks. I will look into it.

Original issue reported on code.google.com by hans.beckerus on 2014-02-22 08:38:00

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I'll attach sample file. I changed file's extension becuase it occurs 500 internal server
error...

Original issue reported on code.google.com by jyhpsycho on 2014-02-22 09:21:19

- _Attachment: [TEST.rar.tmp](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-28/comment-12/TEST.rar.tmp)_

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I tried to extract this using WinRAR but the Vol.x.rar files does not seem to be valid
RAR archives. Is that expected?

Original issue reported on code.google.com by hans.beckerus on 2014-02-22 10:33:09

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Yes. That has just null data despite .rar extension. See instructions above. It seems
to be occured when parsing directory hierarchy, not accessing actual data.

Original issue reported on code.google.com by jyhpsycho on 2014-02-22 10:44:23

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Yes. Nasty one. Seems to crash in glibc, something with fmemopen().

[Switching to Thread 0x7fffe6851700 (LWP 15262)]
__memcpy_sse2 () at ../sysdeps/x86_64/multiarch/../memcpy.S:273
273     ../sysdeps/x86_64/multiarch/../memcpy.S: No such file or directory.
(gdb) where
#0  __memcpy_sse2 () at ../sysdeps/x86_64/multiarch/../memcpy.S:273
#1  0x00007ffff6ed0639 in fmemopen_read (cookie=0x7ffff0095480, b=<optimized out>,
s=8192) at fmemopen.c:107
#2  0x00007ffff6ed6ff8 in _IO_new_file_underflow (fp=0x7ffff0095530) at fileops.c:619
#3  0x00007ffff6ed827a in _IO_default_xsgetn (fp=0x7ffff0095530, data=<optimized out>,
n=1048560) at genops.c:540
#4  0x00007ffff6ecb783 in _IO_fread (buf=<optimized out>, size=1, count=1048560, fp=0x7ffff0095530)
at iofread.c:44
#5  0x00007ffff7961695 in File::DirectRead(void*, unsigned long) () from /usr/lib/libunrar.so
#6  0x00007ffff796172a in File::Read(void*, unsigned long) () from /usr/lib/libunrar.so
#7  0x00007ffff796295e in Archive::Read(void*, unsigned long) () from /usr/lib/libunrar.so
#8  0x00007ffff7963b68 in Archive::IsArchive(bool) () from /usr/lib/libunrar.so
#9  0x000000000040596e in RARInitArchiveEx (r=0x7fffe6847790, fh=0x7ffff0095530) at
dllext.cpp:119
#10 0x000000000040fa23 in listrar_rar (path=0x7ffff0029b90 "/", buffer=0x7fffe6850cb8,
arch=0x698bf0 "/dev/loop0",
    hdl=0x7ffff0029bb0, next=0x7fffe684c3e0, entry_p=0x626878, mh_flags=16) at rar2fs.c:1913
#11 0x0000000000411617 in listrar (path=0x7ffff0029b90 "/", buffer=0x7fffe6850cb8,
arch=0x698bf0 "/dev/loop0")
    at rar2fs.c:2246
#12 0x0000000000413e5d in rar2_readdir2 (path=0x7ffff0021a20 "/", buffer=0x7ffff00219a0,
filler=0x7ffff7bacde0, offset=0,
    fi=0x7fffe6850d40) at rar2fs.c:2884
#13 0x00007ffff7bb0149 in fuse_fs_readdir () from /lib/libfuse.so.2
#14 0x00007ffff7bb038c in ?? () from /lib/libfuse.so.2
#15 0x00007ffff7bb7596 in ?? () from /lib/libfuse.so.2
#16 0x00007ffff7bb5d65 in fuse_session_loop () from /lib/libfuse.so.2
#17 0x00000000004189cd in work_task (data=0x7fffffffeab0) at rar2fs.c:4422
#18 0x00007ffff7222e9a in start_thread (arg=0x7fffe6851700) at pthread_create.c:308
#19 0x00007ffff6f4fccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#20 0x0000000000000000 in ?? ()

Original issue reported on code.google.com by hans.beckerus on 2014-02-22 11:01:05

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Actually, mounting TEST.rar the normal way does not work for me either. There is no
crash but it does not list cleanly.

ls: cannot access [TEST] Vol.3.rar: No such file or directory
total 136
-rw-r--r-- 1 root root 65536 Feb 22 05:43 [TEST] Vol.1.rar
-rw-r--r-- 1 root root 65536 Feb 22 05:43 [TEST] Vol.2.rar
?????????? ? ?    ?        ?            ? [TEST] Vol.3.rar

So there is something more to this than just the crash.
Can you reproduce my observation?

Original issue reported on code.google.com by hans.beckerus on 2014-02-22 11:20:45

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Aha. It is when I turn off fmemopen() support that it fails also without using a loop
device. This is related to the path for issue22 which still seem to hide a bug :(

Original issue reported on code.google.com by hans.beckerus on 2014-02-22 11:29:40

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Hmm... I've never seen before; but, there's some another issue related on issue 22.

When I used rar2fs first time(~1.19.0, I remember), that shows directory structure
- Mount rar archives in wrapper-archive automatically. Prevent to that behavior, there's
--flat-only option. right? But, right now that doesn't mount automatically at all even
without --flat-only option! Actually, file list on issue 29 (rar2fs' output) are outputs
without --flat-only option. Is that expected behavior on issue 22 fixed?

Original issue reported on code.google.com by jyhpsycho on 2014-02-22 12:57:37

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I can reproduce that with explicitly disabled fmemopen() support.



LOOKUP /[TEST] Vol.3.rar
getattr /[TEST] Vol.3.rar
rar2_getattr2()   /[TEST] Vol.3.rar
MISS    /[TEST] Vol.3.rar
syncrar()   /
listrar()   /   arch=/dev/loop0
Looking up /[TEST] Vol.3.rar in cache
Adding /[TEST] Vol.3.rar to cache
65536 byte RAR file /[TEST] Vol.3.rar found in archive /dev/loop0
File inside archive is [TEST] Vol.3.rar
Looking up /[TEST] Vol.3.rar in cache
File inside archive is [TEST] Vol.2.rar
Looking up /[TEST] Vol.2.rar in cache
File inside archive is [TEST] Vol.1.rar
Looking up /[TEST] Vol.1.rar in cache
Invalidating cache path /[TEST] Vol.3.rar
Looking up /[TEST] Vol.2.rar in cache
Looking up /[TEST] Vol.1.rar in cache
   unique: 22, error: -2 (No such file or directory), outsize: 16
unique: 23, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 32218
   unique: 23, success, outsize: 16
unique: 24, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
releasedir[140437780339568] flags: 0x0
rar2_releasedir()   
   unique: 24, success, outsize: 16



Why it invalidates cache for Vol.3.rar?

Original issue reported on code.google.com by jyhpsycho on 2014-02-22 19:36:33

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Got it! rar2fs tries to parsing Vol.x.rar inside TEST.rar, but that's not RAR file -
contains just null data. Then it throws error when open that file recursively. It needs
some file header check to distinguish RAR inside RAR instead of extension check currently.

With --flat-only option and fmemopen() disabled, Even original file which I found this
issue (~130GB RAR file) works! (It has valid RAR files, though) I think there's some
bug on fmemopen() implementation in glibc, not the rar2fs' bug. Can you investigate
some more?

Original issue reported on code.google.com by jyhpsycho on 2014-02-22 19:51:42

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

There's an error without --flat-only option, even valid RAR file inside...

Original issue reported on code.google.com by jyhpsycho on 2014-02-22 21:27:49

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Yes, there is a bit more to this than you might realize ;)
The reason behind issue22 is actually due to a new feature introduced by libunrar5.
Before this version it was working just fine. In libunrar5 the check for a valid archive
modifies the file pointer and that is causing severe problems since rar2fs relies on
the fact that it is kept unmodified. I have already had a conversation with Eugene
at RAR labs about this, and in the upcoming version of libunrar it will be addressed.
But it still requires some workaround for current release(s). I thought I identified
what is needed to workaround this problem, but I now realize that the current solution
is not enough :( I have to find a solution to this problem first! Then it would be
possible to also workaround the crash by forcing rar2fs not to use fmemopen() for device
special files. At least until it can be identify why it does not work.

Original issue reported on code.google.com by hans.beckerus on 2014-02-22 23:11:23

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I think I managed to solve the problem with systems not providing fmemopen() now. The
crash is still a mystery though :( What I can do now is to detect the device special
file and by-pass fmemopen() to avoid the crash. At least until I know what is going
on.

Original issue reported on code.google.com by hans.beckerus on 2014-02-23 02:55:30

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Hmm, intresting...
In my current version of rar2fs I can only reproduce the crash if fmemopen() is enabled
and --flat-only is *not* specified! If I specify --flat-only it works just fine!
I need to figure out why. One problem might be the fact that the workaround for issue22
still has one caveat, and that is that the isArchive() function does not detect the
invalid format! But for some reason when using fmemopen() that results in a crash,
but not when disabling it, and only for device special mounts!?

Original issue reported on code.google.com by hans.beckerus on 2014-02-23 03:06:46

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Nah, when using --flat-only it works because then fmemopen() is not called.
When it is, things go bad :( I will look into it more deeply but for now I believe
the fastest way around it is to avoid fmemopen() for device special files!

Original issue reported on code.google.com by hans.beckerus on 2014-02-23 03:31:40

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

(No text was entered with this change)

Original issue reported on code.google.com by hans.beckerus on 2014-02-23 03:31:54

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I now see why it crashes! The reason is that I use stat() to get the size of the master
RAR archive to be able to mmap()+fmemopen() it. But stat() for /dev/loop0 returns only
4096 resulting in buffer overrun. I need to figure out how to obtain the actual size
of the archive when mounted like this!

Original issue reported on code.google.com by hans.beckerus on 2014-02-23 16:03:08

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I think there's no reason for enforce using fmemopen() to open device file unless there's
some critical performance degradation. I'll test latest version as soon as possible
and report it back...

Original issue reported on code.google.com by jyhpsycho on 2014-02-25 02:45:32

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Ok, r459 works. ;-)

Original issue reported on code.google.com by jyhpsycho on 2014-02-27 12:27:12

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Thanks.

Original issue reported on code.google.com by hans.beckerus on 2014-02-27 12:55:25

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

There's still exist an issue on FreeBSD 10. But, it's not rar2fs' bug, thus I do not
file new issue. This is just for report, NOT RAR2FS' BUG.

FreeBSD accepts only block-aligned access on block device - it means if you seek to
arbitrary location on block device and try to read, it fails. More precisely, following
code should fail:

fp = fopen("/dev/ada0", "r");
fseek(fp, 511, SEEK_SET);
fread(buf, 1, sizeof(char), fp);

But, file operation functions in libunrar does not consider this. On Linux, there's
no issue such as above, it works. On FreeBSD, it should works with proper libunrar
patch I think.

Original issue reported on code.google.com by jyhpsycho on 2014-03-11 14:10:38

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Thanks for the info. But I really doubt that libunrar was designed with block special
devices in mind. And adding support for it is probably not to expect either. Maybe
I should turn off support for block special device mount completely unless it is Linux?
OSX is BSD based so my guess is that it too is limited to block aligned seek operations?

Original issue reported on code.google.com by hans.beckerus on 2014-03-11 14:17:12

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Hmm... I need this if it still needs patch... I think it's ok that configure script
provides option to support block device mount, and print warning if that option is
specified and environment is not Linux.

I'm not have OSX, thus I can't test it now...

Original issue reported on code.google.com by jyhpsycho on 2014-03-11 14:28:23

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Am I understanding you correctly that you already have a patch to support block devices
in libunrar?

Original issue reported on code.google.com by hans.beckerus on 2014-03-11 14:44:46

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I made patch before reinstalling BSD, but I forget to backup it, then it needs re-workaround.
I remember only the concept of patch, thus it takes some time to reconstruct it...

Original issue reported on code.google.com by jyhpsycho on 2014-03-11 15:21:32

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Fine. Can you please post the patch whenever you have something ready? Then I can forward
it to Eugene at RARLabs and maybe convince him to include it in some upcoming release?
But how does it tell the difference between normal and block special file case? Or
was your patch hard-coded to handle only the BSF?

Original issue reported on code.google.com by hans.beckerus on 2014-03-11 15:55:41

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Basically, block device is accessible only multiple of blocks - sector in this case
- at once, not arbitrary location. Therefore when reading data from block device, it
should calculate location and size to read includes actual needed data. Regular file
system driver does that internally, but rar2fs delegates I/O to libunrar, and libunrar
does not that; it does not needed when accessing regular file just causes I/O overhead
because file system driver does that already when accessing regular file. Note that
RAR's original platform - DOS and Windows - does not support accessing disk device
directly as file!

My patch is just force to do that when libunrar tries to reading data. It causes some
I/O overhead when accessing regular file, I said before. It's not hard-coded for BSD,
but currently only considered 512-byte sector devices. That's easy to fix to support
4K sector, but it needs some workaround to generalize...

On Linux, it seems to kernel does that internally when accessing block device, similar
as regular file. That's just my guess, though.

There's some issue that file pointer management of libunrar's API... I used existing
logics as possible as I can, but that's not works well now. It seems to be more dirty
workaround.

I'll attach my patch when ready.

Original issue reported on code.google.com by jyhpsycho on 2014-03-11 16:35:26

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I found that libunrar API issue when seeking archive is related on quick open feature.
It takes some more time to fix...

Original issue reported on code.google.com by jyhpsycho on 2014-03-11 17:24:22

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I can mount block device with this patch, but it cannot read file data. I checked only
if it mountable before reinstalling OS, too...

Is there any raw archive access not through libunrar? That should be fixed as same
manner if exists. I'll attach libunrar part of patch here. I'll work with rar2fs part
later because I'm very sleepy now...

Original issue reported on code.google.com by jyhpsycho on 2014-03-11 18:20:49

- _Attachment: [libunrar-blockdev.patch](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-28/comment-41/libunrar-blockdev.patch)_

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Previous patch has some bug I think.

I finally maked patch seems to work for rar2fs... but, it completely disables 'seek-as-needed'
functionality - file pointer must be adjusted before read data. Is that sufficient?
I'll attach patches when I'm sure they are work.

Original issue reported on code.google.com by jyhpsycho on 2014-03-12 11:05:00

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I do not really know what you mean by "seek-as-needed", but reading from a given file
position should still be possible even if you need to align start offset on a block
size boundary. It should simply be some overhead, but data should be possible to retrieve
correctly? Please attach the patch. I can not promise anything though. Getting this
kind of support into upstream version of libunrar might be hard considering the narrow
user base. If it inflicts a huge performance degradation for common use it will most
likely be rejected.

Original issue reported on code.google.com by hans.beckerus on 2014-03-12 11:22:50

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Hmm... what "seek-as-needed" I mean is, currently rar2fs seems to emit seek operation
under some conditions - it just my understand - using variable force_seek. libunrar
itself sometimes returns different seek pointer I expected; I can't exactly understand
what they does and why they works currently despite of strange behavior, but it should
be adjusted correctly when accessing block device because it requires exactly block
boundary aligned access.

It seems that I can attach patches soon. I'm not good at C++, and I faced on some issue
related to Array<byte>... Using malloc() and free() seems to work, but it isn't preferred
way to fix C++ project like libunrar I think.

There's no notable performance degradation for me.

Original issue reported on code.google.com by jyhpsycho on 2014-03-12 14:28:46

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Right. Well force_seek if for archives with store compression (e.g. no compression at
all). For those archives rar2fs handles the file pointer internally and does not use
libunrar since no "extraction" is needed. I guess you might need to look into that
logic too :(

Original issue reported on code.google.com by hans.beckerus on 2014-03-12 14:38:03

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Here's my patches. One is libunrar side, another is rar2fs side. I checked under following
condition:

1) RAR archive data is in /dev/ada4p1, its size is ~45 GB, and it contains another
RAR archive.
2) Mount it using patched rar2fs compiled with patched libunrar.
3) Perform Test operation with patched unrar and not patched unrar.



My patches has some limitation:

1) It's just Proof-of-concept, not optimized one. I guess it's possible that increase
performance by modifying lread_raw() function entirely. But, I think it's sufficient,
and more preferable to optionize(?) it.
2) It considers just 512-byte-sector device. For 4K sector devices, there's need few
workarounds I think, but now most of 4K storage devices also supports 512-byte emulation,
thus it also works I guess. It's possible to determine device's minimum I/O size and
apply it, too.
3) It has following overhead:
   1] Seek to block boundary before read data, Seek to correct location after read
data - it simulates regular read to minimize code fix.
   2] Allocating another read buffer that fits corrected data range.
   3] Copy data from temporary read buffer to buffer used by outside of fixed code.
   It doesn't degrade the performance visibly for me, though.
4) It does not check whether malloc() is succeed or failed when allocating read buffer
in lread_raw() because I don't know the proper way to treats that exception. In most
cases it doesn't fail, but it still needs complementary measures.



Can you test if my patches works properly?

Original issue reported on code.google.com by jyhpsycho on 2014-03-12 18:26:26

- _Attachment: [rar2fs-blockdev.patch](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-28/comment-46/rar2fs-blockdev.patch)_ - _Attachment: [libunrar-blockdev.patch](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-28/comment-46/libunrar-blockdev.patch)_

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I don't test with multi-volume archive currently. I hope that you may test with it.

Original issue reported on code.google.com by jyhpsycho on 2014-03-12 18:50:36

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Oh, I misunderstand csh's time command output... that's very slow, rar2fs part at least;
I'll test another performance issues.

Original issue reported on code.google.com by jyhpsycho on 2014-03-13 01:36:59

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Following data is gathered by using:
1) Archive file - not patched version doesn't handle block device.
2) Test order - Testing -m5 archive then -m0 archive for invalidate page cache effectively.
3) Size - About 45GB both.

Format of csh's time command output:
(User time) (System time) (Elapsed time) (CPU Usage) (Memory Usage) (I/O Usage) (Page
fault/swap)

Without any patches:
1) -m5 compressed archive
1258.795u 108.472s 8:09.09 279.5%   304+614k 0+0io 0pf+0w
2) -m0 stored archive
372.915u 45.310s 6:55.69 100.6% 305+616k 345282+0io 5pf+0w

With libunrar patch only, and does not patch libunrar linking against rar2fs (It means
only unrar itself used for testing archive is patched):
1) -m5 compressed archive
1285.820u 131.100s 9:00.30 262.2%   303+611k 0+0io 0pf+0w
2) -m0 stored archive
381.708u 46.008s 7:09.31 99.6%  304+613k 356772+0io 5pf+0w

It seems that there's at least ~20% performance degradation with compressed file...
It needs more optimization before accepted by upstream and applied to rar2fs. Is there
any ideas?

Original issue reported on code.google.com by jyhpsycho on 2014-03-13 02:42:24

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

The patch to rar2fs should only affect random accesses and a possible volume file change
for archives in store (m0) mode. Sequentially decompressing an archive in store mode
should not make much of a difference in performance. The problem here are the compressed
streams that is accessed through libunrar. I will look into your libunrar patch more
carefully to see if something can be done to improve the performance.

Original issue reported on code.google.com by hans.beckerus on 2014-03-13 10:18:31

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Simply curious, but are you sure you wish to do this on FreeBSD?
http://www.freebsd.org/doc/en/books/arch-handbook/driverbasics-block.html

Original issue reported on code.google.com by hans.beckerus on 2014-03-13 11:30:02

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I think I found more suitable way to deal with. I'll attach the patch when performance
test is done.

Original issue reported on code.google.com by jyhpsycho on 2014-03-13 11:31:28

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Hmm... that says about hardware cache, doesn't?

I don't deal with entire block device layer on FreeBSD; If I want to do that, I should
report this issue to FreeBSD kernel maintainer, not here. I want just rar2fs works...

Original issue reported on code.google.com by jyhpsycho on 2014-03-13 11:39:38

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Yes, but is it not also saying that support for block special devices on FreeBSD more
or less may be removed sooner than later?

Original issue reported on code.google.com by hans.beckerus on 2014-03-13 11:41:44

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

Block devices is probably going to be supported also on FreeBSD, but not direct access
to them.

Original issue reported on code.google.com by hans.beckerus on 2014-03-13 11:45:33

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

I just failed... trying to think another idea... :(

Original issue reported on code.google.com by jyhpsycho on 2014-03-13 13:15:53

@hasse69
Copy link
Owner Author

hasse69 commented May 23, 2015 

On libunrar, there's some read-ahead code to deal with encrypted archive. I made some
code changes to read not encrypted archive as same manner, but it failed when reading
comment - There's no problem extracting archive contents, though. I'll investigate
some more...

Original issue reported on code.google.com by jyhpsycho on 2014-03-25 14:32:08

@Nischi85 Nischi85 mentioned this issue Oct 9, 2022
Closed
fdegros pushed a commit to fdegros/rar2fs that referenced this issue Nov 29, 2023
@hasse69
* Released 1.20.0
39d51f1 
* Turned off support for block special device mounts other
  than for Linux platforms. It does not make much sense trying
  to support it any where else due to possible read alignment
  restrictions.

* Built and tested against UnRAR source 5.10 beta 1 (5.1.1)

* [rar2fs hasse69#34]

  Fixed a problem with old style archives split into more
  than 101 files.

* Now returning "No such file or directory" instead of
  "Input/output error" when trying to read information
  for a file outside an archive or which does not exist
  at all (libunrar5 only).

* Fixed an issue when retrieving file information (#info)
  that could have resulted in an infinite loop.

* Fixed an issue with resolving symbolic links for
  legacy archives in combination with libunrar5.

* Corrected some issues that caused compilation failures
  for UnRAR source 4.1.x.

* Corrected a problem seen on e.g. FreeBSD for which closedir()
  was called with a NULL pointer resulting in a crash.

* [rar2fs hasse69#31]

  Corrected some compilation errors on FreeBSD (and also other
  platforms that sets HAVE_SETXATTR).

* Mounting block special files on Linux can now benefit from
  fmemopen() when supported.

* Enabled syslog support

* [rar2fs hasse69#22]

  Now may fall back to the old workaround from 1.19.2 that
  should be working also for (some!?) uClibc versions of
  fmemopen().

* Corrected some additional problems with the solution
  for issue hasse69#22.

* Added support for presenting high-resolution time stamp in
  RAR5 archives (thanks to jyhpsycho for the initial patch).
  Legacy archives still require a new version of UnRAR source
  before it can be enabled.

* [rar2fs hasse69#28]

  Fixed problem with mounting a block special file that
  resulted in a crash due to incorrect use of mmap() and
  fmemopen().

* Added a missing check for errors from fmemopen()

* [rar2fs hasse69#29]

  Fixed a minor calculation error in the file time stamp
  'seconds' part as provided by libunrar (MS-DOS time format).

* Removed 'INSTALL' from SVN trunk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Defect Priority-Medium
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hasse69