random crashes

[hasse69]

rar2fs v1.19.0 (DLL version 6)    Copyright (C) 2009-2013 Hans Beckerus
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under
certain conditions; see <http://www.gnu.org/licenses/> for details.
FUSE library version: 2.8.5
fusermount version: 2.8.5
using FUSE kernel interface version 7.12

Native build on NAS (DNS323) ARM machine with unrar-5.0.14, also natively built.

I get random crashes just trying to list the directory of the mounted rar file. Rebuilt the package with --enable-debug and debug symbols enabled, so can trace in gdb, but so far haven't gotten much wiser. Seems like it doesn't like multi-part rar files? f I set a seek-length=1 I do get at least a directory listing. 
I tried mounting single-file rar files, that seems to work better, however at some point I will get the same backtrace as below. Any ideas to solve this? So would like to have rar2fs working with minidlna[*] and stream films to my tv :)

Looking up /Captain.Phillips.2013.720p.BluRay.X264-AMIABLE.mkv in cache
Adding /Captain.Phillips.2013.720p.BluRay.X264-AMIABLE.mkv to cache
listrar()   /   arch=/mnt/md0/downloads/Captain.Phillips.2013.720p.BluRay.X264-AMIABLE/captain.phillips.2013.720p.bluray.x264-amiable.r00
[New Thread 0x40b194c0 (LWP 1889)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x40b194c0 (LWP 1889)]
0x402cd080 in wcscpy () from /ffp/lib/libc.so.0
(gdb) bt
# 0  0x402cd080 in wcscpy () from /ffp/lib/libc.so.0
# 1  0x4004a720 in StringList::AddString(wchar_t const*) () from /ffp/lib/libunrar.so
# 2  0x40076264 in RAROpenArchiveEx () from /ffp/lib/libunrar.so
# 3  0x0001a0ac in listrar (path=0x9d0b8 "/", buffer=0x40b18ce4, 
    arch=0x40b18b20 "/mnt/md0/downloads/Captain.Phillips.2013.720p.BluRay.X264-AMIABLE/captain.phillips.2013.720p.bluray.x264-amiable.r00") at rar2fs.c:2024
# 4  0x0001d20c in readdir_scan (dir=0x9d0b8 "/", 
    root=0x40b18c80 "/mnt/md0/downloads/Captain.Phillips.2013.720p.BluRay.X264-AMIABLE/", next=0x40b18ce4) at rar2fs.c:2470
# 5  0x0001db58 in rar2_readdir (path=0x9d0b8 "/", buffer=0x60540, filler=0x40014968, offset=0, fi=0x40b18dc8)
    at rar2fs.c:2682
# 6  0x40017cb8 in fuse_fs_readdir () from /ffp/lib/libfuse.so.2
# 7  0x40017ee0 in ?? () from /ffp/lib/libfuse.so.2
Cannot access memory at address 0x0
# 8  0x40017ee0 in ?? () from /ffp/lib/libfuse.so.2
Cannot access memory at address 0x0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[*] although when I did try the mounted file was not scanned, so I think I need some work there.

Original issue reported on code.google.com by fafarago on 2014-01-11

[hasse69]

Thanks for the issue report.
I might need access to a at least a few of your RAR volume files in order to reproduce
the problem. Would that be possible somehow? 
I have no access to an ARM based system so I will need to try this on some other host.
There has not been any similar reports before, but from the first look it seems to
be related with libunrar.so. Are you sure you are running towards the version of libunrar
that you built against? That is, did you also build libunrar.so and then installed
it in some proper system location? I do recall that similar issues were seen when using
incompatible library versions. Can you please provide the config.log file that was
created when rar2fs was built?
And more questions. Was this your first experience with rar2fs? I.e. does this seem
to be unique to 1.19.0 or libunrar 5.0.14 or do can you not tell?

Original issue reported on code.google.com by hans.beckerus on 2014-01-11 14:28:32

[hasse69]

(No text was entered with this change)

Original issue reported on code.google.com by hans.beckerus on 2014-01-11 14:28:48

[hasse69]

To answer a few of your questions:
- I am running against the libunrar version that I've built against. I know this for
sure because my system didn't have libunrar, so I built it myself on ARM. I did have
to change the makefile for libunrar as several functions weren't exported to the .so
file, so it needed recvol.o and rs.o added to the LIB_OBJ targets. I have also built
the unrar library using the same process and that seems working fine.
- I have attached the config.log file for the rar2fs build. This is with --enable-debug
and -g for debugging symbols.
- This is my first experience with rar2fs, just found the project and gotten really
excited :) I could try different versions if that is the way to be going for a search.
However as I am compiling everything on my NAS natively it is not the fastest process.
- I've tried different rar files, and all of them exhibited the same behaviour. If
--seek-length is not set (to 1 for example) then I cannot even get a directory listing
and the stack trace is as follows; doesn't even look like it accesses libunrar at all.
syncdir_scan()   /
listrar()   /   arch=/mnt/md0/downloads/file.rar
Looking up /contents.mp4 in cache
listrar()   /   arch=/mnt/md0/downloads/file.r00
listrar()   /   arch=/mnt/md0/downloads/file.r01
[New Thread 0x40b194c0 (LWP 832)]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x40b194c0 (LWP 832)]
0x402ea924 in raise () from /ffp/lib/libc.so.0
(gdb) bt
#0  0x402ea924 in raise () from /ffp/lib/libc.so.0
#1  0x402e4750 in abort () from /ffp/lib/libc.so.0
#2  0x00000000 in ?? ()


It's a bit of a hit&miss. I've successfully copied a video from a multi-part file out
of rar2fs, queried metadata with 'file <file>', however a simple 'ls -l' threw me an
exception, but only after the third try. I am stuck to an OSX machine right now but
will try to create a simle rar file to upload.

Original issue reported on code.google.com by fafarago on 2014-01-11 17:19:33

---

- _Attachment: [config.log](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-21/comment-3/config.log)_

[hasse69]

Quickly going through your config.log does not reveal anything I am afraid :(
But I got really surprised when you tell me you need to change things in the libunrar.so
makefile!?? You should *never* need to to that!
All you need is to look in the makefile, use whatever target applies to you (gcc is
the default I think) and then issue 'make lib'. If you need to make modification to
the makefile in order for libunrar to build then that should probably be report to
RAR labs. I can not answer to how changing things in there might affect your system.
Anyway, I was able to get hold of the same archive as you reported in the initial report.
I will try it here and see how it goes. But if you can provide a very simple reproducer
archive then that of course is the best way forward. I do have some slight worries
that I will not be able to reproduce your specific problems :( I seems to be very platform
or environment specific. Nothing of what you describe have been reported before, but
again ARM targets is nothing that I have been able to test myself. But let me perform
some local test and take it from there...

Original issue reported on code.google.com by hans.beckerus on 2014-01-11 18:09:58

[hasse69]

Nope, could not reproduce your problem using the archive you originally reported :(
It works just fine at my end. Must be something with your environment, and not just
a simple bug in rar2fs ( I never thought so either ;) ). Also you report very different
type of problems, SIGSEGV and SIGABRT etc. Very different types of crashes indeed.
I think we need to focus on one error at the time. Exactly how did you invoke rar2fs
when reporting the first problem?

Original issue reported on code.google.com by hans.beckerus on 2014-01-11 18:24:03

[hasse69]

A quick question, is your target (NAS) system 32- or 64-bit?
Just curious since I can see that LPARAM is used to hold the value of a pointer. LPARAM
is defined as a signed long. Could break on a 64-bit system if long is not also defined
as being 8 bytes (which it should according to the LP64 standard, be but there is also
the LLP64 exception for which 'long long' is needed to define a container large enough
to hold a pointer).

Original issue reported on code.google.com by hans.beckerus on 2014-01-11 19:54:25

[hasse69]

Hi Hans, thanks so much for giving stuck fast feedback. I won't be able to generate
an example rar until tomorrow when I go into work, but I agree it is not the rar file
itself that is at fault here but something else. First off:
- using an armv5tel, so 32bits
- I had to add two other object files to libunrar because rar2fs configure would fail
in testing for RARGetDllVersion on missing dependency RecVolumesRestore. I've looked
around and seems the libunrar source is not really supported. The makefile is really
Crap too. I will report this to rarlabs though 
- I run the following command "rar2fs -f -o allow_other --seek-depth=1 src_dir dst_dir
- did some guy last night and it looks there is some buffer overflow somewhere or threading
issues, see below. When you try to open the rar file in libunrar some data gets totally
fcked up. In dll.cpp Data just gets allocated and everything initialized nicely, but
it's not. It works the first two times by then I get this. Is it possible to run rar2fs
in single threaded mode? The unrar correctly works so it must be something between
that and rar2fs and fuse. 
- anyways will try and set up something better. Just ask if you need more info. 

(gdb) p Data->Cmd.FileArgs
$1 = {StringData = {Buffer = 0x42 <Address 0x42 out of bounds>,
    BufSize = 110, AllocSize = 117, MaxSize = 82}, CurPos = 97,
  StringsCount = 121, SaveCurPos = {46, 88, 50, 54, 52, 45, 65, 77,
    73, 65, 66, 76, 69, 47, 99, 97}, SavePosNumber = 112}
(gdb) f 2
#2  0x40097e64 in RAROpenArchiveEx (r=0x40b3ca08) at dll.cpp:42
42          Data->Cmd.FileArgs.AddString(L"*");
(gdb)

Cheers

Original issue reported on code.google.com by fafarago on 2014-01-12 00:54:32

[hasse69]

Right. I definitely do not think it has anything to do with FUSE, nor the interface
between rar2fs and fuse. It is more likely that something has gone bad in the interface
between rar2fs and the libunrar API extensions. Could have something to do with the
changes you had to make. Actually, the crash really looks similar to what I experienced
when I (by mistake) happened to get a member inconsistency in a struct that was referenced
both from rar2fs and libunrar, but the size became different and resulted in various
crash scenarios. Can you please describe exactly what changes you had to make to have
libunrar compile properly on your platform?

Original issue reported on code.google.com by hans.beckerus on 2014-01-12 17:03:25

[hasse69]

Also, rar2fs is already running in single threaded mode by default. But that is only
regarding its file system implementation. How libunrar handles threads internally is
nothing that we have control of. I see that you using the libunrar 5.x.x version, would
it be possible to try libunrar 4.x.x instead? It might be something in the support
of RAR5 archives that is not yet stable on all platforms. Not saying you should permanently
switch to libunrar4 but it might give a hint were to start searching for the problem.

Original issue reported on code.google.com by hans.beckerus on 2014-01-12 18:50:21

[hasse69]

And, did you build both the unrar utility and libunrar.so at the same time? I do recall
that there is a bug in the unrarsrc makefile that requires you to do a clean in between
if you first build the unrar utility and then the library! This is because is it sharing
object files but they are built differently! 

make
make clean
make lib

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 09:59:30

[hasse69]

See attached the changes to the unrar 5.0.14, which is basically this:
-LIB_OBJ=filestr.o scantree.o dll.o qopen.o
+LIB_OBJ=filestr.o scantree.o dll.o qopen.o recvol.o rs.o

Trying with unrar-4.2.4 I have to rename makefile.unix to makefile; it seems to work
:)

Note: in light of your most recent reply let me rebuild unrar-5 without the makefile
change that doesn't rebuild the lib after making the source.

Original issue reported on code.google.com by fafarago on 2014-01-13 12:26:00

---

- _Attachment: [fix-makefile.patch](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-21/comment-11/fix-makefile.patch)_

[hasse69]

Ok. Seems very odd that your platform should require objects files not required on other
platforms. recvol.o is only for volume recovery, which is not supported by the library
and should not be needed. rs.o is also used for recovery/repair of archives which is
a unique feature to the unrar utility.
But it is good to hear that you got 4.2.4 working! That means we have something more
concrete to work and compare against. I am looking forward to your new test results
of 5.0.14.

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 13:04:01

[hasse69]

Hmm, you did quite some rework of the makefile ;) Actually, I think the bug with clean
in between was fixed but it might be that your changes actually re-introduced it :(
It is very important that all object files are removed in between builds of unrar and
libunrar.so since they share object files but has different #defines set. That may
result in conflict of common data types and sizes etc. It is also most likely why you
had to add object files too! Actually the proper way it so change it so that output
is split between the two different builds or placed in different output folders. I
have already been discussing this with Eugene at RARLabs and he is aware of the problem
but does not put it on priority. I think a proper prepared autoconf project would be
what the UnRAR source tree needs ;)

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 13:15:56

[hasse69]

Ok, so I've fix the build. It was indeed me not using clean between building the binary
and the librar. Now the diff is only as attached. A bit stupid of me to not think about
the impact the changes I've made...

I have noticed one difference between using unrar 4 and 5. If the rar file contains
another rar then something is off. Below is are the contents of the rar file, I have
attached the command output and rar2fs log. Basically the inner rar is processed wrongly
(the contents of the original rar file are used instead of those in the rar). Don't
know where the fault lies, if it's unrar just tell and I'll revert back to libunrar-4.

root@alexandria:/mnt/md0/downloads# unrar l test/bob.rar 

UNRAR 5.01 freeware      Copyright (c) 1993-2013 Alexander Roshal

Archive: test/bob.rar
Details: RAR 4

 Attributes      Size    Date   Time   Name
----------- ---------  -------- -----  ----
    ..A....    303029  08-01-14 21:06  file.idx
    ..A....   7884165  08-01-14 21:31  inner_archive.rar
----------- ---------  -------- -----  ----
              8187194                  2

Original issue reported on code.google.com by fafarago on 2014-01-13 14:50:18

---

- _Attachment: [rar2fs_log.txt](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-21/comment-14/rar2fs_log.txt)_

[hasse69]

heh, wrong upload...

Original issue reported on code.google.com by fafarago on 2014-01-13 14:51:35

---

- _Attachment: [fix-makefile.patch](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-21/comment-15/fix-makefile.patch)_

[hasse69]

It surely looks like rar2fs is confused about something here!
Can you please provide me with your example archive so I can play with locally.

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 14:59:19

[hasse69]

See attached. Still no real test file; was very busy at work :( Thanks so much so far!

Original issue reported on code.google.com by fafarago on 2014-01-13 15:19:33

[hasse69]

Another thing, can you tell if you have fmemopen() support on your system.
In config.h (generated by configure) do you see this:

/* Define to 1 if you have the `fmemopen' function. */
#define HAVE_FMEMOPEN 1

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 15:27:18

[hasse69]

Works for me! :( 
But I have support enabled for fmemopen(). Actually, coming to think about it, I do
not think I ever performed regression test of this feature without it...
That could explain why it works for me but not for you. I will force this feature to
disabled and hopefully I can reproduce your problem.

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 15:34:01

[hasse69]

Ouch! After disabling support for fmemopen() it goes equally wrong for me as for you.
Good thing is that it means I can reproduce it! Bad thing is that I need to fix it
;)
Interesting that this passed unnoticed for such a long time? Maybe libunrar5 is not
used that much yet or fmemopen() support is more widely spread than I first expected.
Anyway, this is my bad and will look into it ASAP. Can you please file a new issue
report so I can close this one?

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 15:49:04

[hasse69]

I think I have corrected the problem now! I however still wish for you to file a new
issue report describing your RAR in RAR problem. Can you do that?
But until an official patch is available you can try this version of dllext.cpp and
rebuild rar2fs.

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 17:58:40

---

- _Attachment: [dllext.cpp](https://storage.googleapis.com/google-code-attachments/rar2fs/issue-21/comment-21/dllext.cpp)_

[hasse69]

(No text was entered with this change)

Original issue reported on code.google.com by hans.beckerus on 2014-01-13 18:01:54

[hasse69]

/* Define to 1 if you have the `fmemopen' function. */
/* #undef HAVE_FMEMOPEN */

With the patch I get the correct directory listing:

tomi@alexandria:~$ sudo ls -l /mnt/md0/downloads/test2
total 25064
-rw-r--r-- 1 root root   303029 Jan  8 21:06 file.idx
-rw-r--r-- 1 root root 25358336 Jan  8 21:06 inner_file.sub

Original issue reported on code.google.com by fafarago on 2014-01-13 23:32:16

[hasse69]

FWIW, I had to add recvol.o and qs.o to LIB_OBJ to get things working on amd64 / x86_64
too, in addition to adding -fPIC to the linker arguments, so it's not just the ARM
platform that has issues with building the lib.

Original issue reported on code.google.com by fastcat on 2014-01-14 16:05:59

[hasse69]

Yes, but the conclusion is that that is not needed. If you did a proper clean of the
source tree between building of unrar utility and the libunrar.so you should not need
to add recvol.o and qs.o on any platform. The library does not use anything from those
objects files. The only reason you had to add them was because you were trying to link
together a library based on object files built for the stand-alone utility, and it
requires recvol.o and qs.o. But I think that later versions of the UnRAR source tree
has corrected the problem with both -fPIC and also performs an automatic clean between
different builds?

Original issue reported on code.google.com by hans.beckerus on 2014-01-14 17:15:28

[hasse69]

...and the fact that linking towards the wrong objects also resulted in crashes! I would
assume that any build of libunrar.so that had to include anything out of the default
is error prone and may result in random crashes :(

Original issue reported on code.google.com by hans.beckerus on 2014-01-14 17:18:18

[hasse69]

Hmm ... the version of unrar I have does do the auto-clean between builds, but you're
right, I don't need those objects any more for things to work.  My need for those on
x86_64 may have been a hold-over from a prior version of unrar.  I'll stop hijacking
this ticket now :)

Original issue reported on code.google.com by fastcat on 2014-01-14 18:46:17

[hasse69]

No problem. Your comment is noted and available for others to see, which is the whole
point :) Thanks for your feedback.

Original issue reported on code.google.com by hans.beckerus on 2014-01-14 21:43:01