stacked file system gets unmounted #177
Comments
|
Unfortunately there is very little to work on here. Never seen anything similar before. That the file system seems to be gracefully unmounted is really strange and should only happen if the FUSE main loop exit for some reason. What version of FUSE is this btw? The INTERRUPT operations look unfamiliar to me. So there are no other traces of problems in e.g. the syslog? |
|
rar2fs v1.29.5 (DLL version 8) Okay so this is from the syslog(unraid) from the first attempt that errored out: And this is from the second attempt which just flat out quit without any error(dont mind the eth0 renaming, its just me stoping/pausing the emby docker container): |
|
The last one is a mystery. I have no answer to why the file system is actively unmounted. That is nothing rar2fs would ever attempt by itself as I have already explained. The first one is a crash though. |
|
Okay, I'll read up on how to do that. #!/bin/bash Get, make and install unrarwget https://www.rarlab.com/rar/unrarsrc-6.1.4.tar.gz Get, make and install rar2fswget https://github.com/hasse69/rar2fs/releases/download/v1.29.5/rar2fs-1.29.5.tar.gz Cleanuprm -rf $WORKDIR |
|
Try to not use the --with-unrar-lib option so that you are guaranteed to compile and link statically to the correct version of unrarsrc. What does the output of Also note that if you run from gdb, run the compiled binary (from the src directory) and not the installed version of rar2fs since the latter will be completely stripped from any symbols. |
Rest will come tomorrow. |
|
Okay sorry for the delay. I just ran my usual test but with new binaries compiled without the --with-unrar-lib options as you said. I'm not too well versed in gdb tho. But I will make an effort to learn if that is what it takes. It crashed somewhat gracefully again. rar2fs 22690549372 17955481232 4733171076 80% /mnt/user/rar2fs_level1 When it crashes df ends up like this: syslog: rar2fs debug: |
|
Yes signature looks the same. But looking at it from the bright side is that it seems pretty easy to reproduce. Simply google how to run a binary through gdb and you should be good to go. There are not that many steps involved to get a nice stack trace. |
|
thank you for bearing with me. |
|
Not really what we need. I think that is what you did but in this case you did not capture the crash, this was something else. |
|
Okay sorry. I ran this command: Hmm maybe something else is crashing first? I'm not sure... |
|
Yes, it is not this process that crash since it simply terminates without catching any signals. |
|
I will try to test some more but I'm traveling for a few days and won't be able to post something new until the weekend. |
|
No worries. Take your time. What you need to do is to run both instances of rar2fs through gdb so that you trap the crash in the correct instance since it seems only one of them is in fact crashing. |
|
Okay so I tried again, by running both instances of rar2fs in gdb as described before. I enter the --args command, and follow it by typing "run" in gdb, I hope that is correct. I fixed another error I had in some of my rar-archives by setting --seek-length=0 . It seems to have cleared out some of the errors I've gotten for files in the emby log, and also made stacking fuse systems hold for longer. Instead of randomly unmounting rar2fs in 1-5 minutes as before. It seems to be more stable and hold for 30-60 minutes. Here is an excerpt of the gdb instance of rar2fs_level (the first of the fuse-stack). (The second stack never stopped working) |
|
I have a theory. And it mostly seems to be like this: `LOOKUP /Movies/Rogue.One.2016.1080p.BluRay.x264-SPARKS/Subs/rogue.one.2016.1080p.bluray.x264-sparks.sub /Movies/Dirty.Grandpa.2016.1080p.BluRay.x264-GECKOS/Subs/ getattr /Movies/Let.Me.Eat.Your.Pancreas.2017.1080p.BluRay.x264-REGRET/Subs/let.me.eat.your.pancreas.2017.1080p.bluray.x264-regret.sub getattr /Movies/Project.Almanac.2014.1080p.BluRay.x264-SPARKS/Subs/Project.Almanac.2014.1080p.BluRay.x264-SPARKS.sub In other words, always .sub giving errors. ls printout of the different levels at the same directory below.
Is it possible that the errors somehow cumulatively causes the process to automatically unmount? CPU is fairly low whilst scanning, around 6-13% usage, and about 20GB of ram still available on the server. On the last run the second level of rar2fs crashed... Thread 5 "rar2fs" received signal SIGSEGV, Segmentation fault. |
|
Can you please make sure to pull master/LATEST from git repo. It looks like the crash is something that has already been addressed and was first spotted in v1.29.5. |
|
I did the following:
|
|
Sound to me that you have corrupt .rar files like myself in some folders. You can start rar2fs but it kills it self after 10-30min. Works fine if i skip this folder on my side tho. |
|
Please build with symbols and continue to run through gdb. We need to make sure there is no crash on any level. Again, rar2fs would never unmount itself. The only thing that could result in a controlled unmount is if the FUSE main loop exit. Just to confirm; I assume you point Emby to the second- and never the first- level mount, right? |
Thanks for your feedback. However, no matter how corrupt things are, the file system should never unmount itself. There simply is no such logic in rar2fs to accomplish that. |
I thought so too actually, but I've run sfv-checks on some of the errored files, and they are all correct. However applying --seek-length=0 definitely helped Emby find more files and improved playback of video files in some cases. I will look into how to build with symbols. unraid SHFS - /mnt/user/ Docker container for Emby is mapped to /mnt/user/rofs. I will add for clarity that if I only mount 1 RAR2FS layer, and then rofs-filtered as described above. I never have any issues like this at all. The crash/unmount only happens when I stack 2 RAR2FS on top of each other with rofs-filtered on top. |
|
To build with symbols just repeat what you did last time. Configure with --enable-debug and make sure to run through gdb and use the built binary directly from the src directory since the installed binary is automatically stripped from any symbols. Any particular reason to why you need this rofs mount? Can you skip that to reduce the use-case somewhat? The rar2fs file system is mounted read-only by default anyway. What I first wish to conclude is that the crash is no longer observed. |
|
You should never point configure to /usr/lib! There is a great chance you compile against a source tree that does not match the library linked to.
You should not need to use the —with-unrar-lib since build should by default link statically to the libunrar.a that was built in your unrarsrc directory.
|
|
I'm currently compiling from inside a VM running slackware (which unraid is based on). I think more than a few packages and dependencies are missing from unraid to compile directly on target. |
|
Oh and about the rofs-mount. I don't care for the read only part, In fact I'd prefer if it was writeable. But I need to filter out a lot of files that are unnecessary for Emby. |
That is perfectly alright, my ask was only about the possibility for you to temporarily skip that extra layer for the simple reason to reduce the scope of your use-case. |
Here I am not sure I fully understand. Then how did you compile last time since you obviously got a better stack trace with full symbol information? Anyhow, let's skip that for now. All I wish to conclude is if the crash you experienced earlier is now gone. I think the syslog was pretty obvious here, showing that something crashed. Let's check that again. |
I compile in the VM and move the binaries out to unraid in appropriate locations. I'm trying to compile with the --enable-debug option but I get an error. Perhaps I'm not doing it correct?
|
|
No, --enable-debug is an option/switch to the configure script you normally run before you build and invoke make. But as I proposed earlier, never mind this step. We only need this if you still observe the crash or we wish to instrument the code in order to get a better idea to what is going on when file system is magically unmounted for no apparent reason. |
thank you for your' patience. This is new territory for me. I think I successfully ran ./configure The crashes do still occur. Since I started this thread I went from a pre-compiled binary supplied through "nerdpack" on unraid, to compiling my own binaries per your instructions. I guess that's why perhaps it looks a bit different. Will try to "crash it again with gdb running" for both instances of RAR2FS. |
|
Lets make sure we agree on the difference between the crash and the unmount. They might be related or they might also not be. |
git clone https://github.com/hasse69/rar2fs.git won't do? I just got an unmount again Nothing special in syslog. Will do a second run just to be sure. Second run done, and it just unmounted again without crashing. |
|
Yes, seems like the crash is gone at least. But the unmount remains a mystery. Can you run outside of gdb now and also mount using the -d flag to collect some debug output from FUSE itself. If possible also skip the rofs mount. |
|
Getting really confused right now. The only plan left I have is that I will set up a log of all the processes running, and see if any process at all is being changed when the unmount happens. |
|
Try the opposite then? Let rofs become your first level, then mount rar2fs as level 2 and 3. |
|
Seems to be working. Still none the wiser why the reverse wont work though. I do loose some functionality though.
I have looked at cmdfs as an alternative to rofs-filtered, but havn't got it working yet, not sure of the state of it since it's been free of updates since 2018. My dream scenario would of course be if rar2fs had an option to point to a config file with regex based filters much like rofs-filtered does ;-) |
That makes two of us I am afraid. The only thing that comes to mind is if rofs-filtered for some reason is getting punished by accessing rar2fs level 2 and 1, possibly caused by some low level timeout in FUSE? Something that ripples down to render some unexpected tear down on kernel level. I simply cannot seem to ignore the fact that those INTERRUPT ops logged by FUSE look very suspicious. I guess the only way to get some answers here would be to raise a question to the FUSE support mailing list. Something you could try is to create three levels of rar2fs mounts, and skip rofs-filtered completely. Then point Emby to your third level. That would then basically become a dummy mount but would also mimic something rather similar to what rofs-filtered does. Just to see if this in some way is related to rofs or in fact something with FUSE as such. That this would boil down to some issue with rar2fs sounds a bit far fetched, but cannot be out-ruled of course. But then we need more facts in order to pursue this issue further.
I can only give you the same answer I have given other users with similar requests in the past. Having a feature to filter out files would be outside scope of what rar2fs was designed to do. Not only does it make the design more complex but also there should be other file system or tools available that would do a much better job at it specifically designed for that particular task. I am a bit surprised that Emby itself does not provide such a feature. It cannot be unique to your use case that files might cause unwanted "noise" in the media libraries. Maybe there is some plug-in that actually would deliver what you want? Have you looked into that possibility? |
|
Have you checked this? https://emby.media/community/index.php?/topic/109245-auto-organize-20-public-beta/ |
|
Okay so first. I will look into stacking 3 layers of RAR2FS like you suggested, I will also try to log ps -e for any differences during the unmount. Second. I did search yesterday night after seeing your suggestion about there possibly being some plugin or emby function for this. And I did not find anything, now I come back and see you have potentially found something. Thanks for that. I will try this out, but also report back about the unmounting part. |
|
Any progress here? |
|
Yes a little bit. I stacked 4 layers of RAR2FS, no errors. I looked into filtering-capability plugins for emby, but the consensus when reading the forum is that it's something the filesystem should deliver. I also looked into the plugin you found as well, but it does not seem to deliver such functionality . |
Sounds to me rofs is involved somehow, could it be too slow when filtering out content? There are no mount options that could be used for rofs that can be related to performance?
Sounds strange, there are no standard file systems delivering this functionality so how can consensus be to rely on such? |
|
The plugin is more about sorting unsorted files into the correct libraries. For example, dumping everything into downloads, and then letting the plugin sort it out to whatever appropiate library, like tv or movies. I asked in the thread about the use of filtering, and the creator confirmed that the plugin won't do what I'm looking for. rofs do have some mount options, and I've tried a few over the last 1-2 years. What i've tried to change is the more standard fuse-options like splice_read. I havn't tried to change the various read options tho. like: Anyway, I ran some more tests and logged the output of ps -e and pstree -a into seperate files and compared them with winmerge. I'm not that fluent in linux processes, but there is definitly some difference between the outputs of ps-e. Do note that the system is not used in any other way during the time than re-scanning the library inside emby (well I have a VM running pfsense also). |
|
In the middle of a house move. Will get back to this a bit later. Is there anything else you can think of that I can log or look into given the running processes? |
|
Not at this moment, no. |
|
I'm a having difficulties lately to find the time to further investigate this issue. I had an idea of trying to somehow make cmdfs work, and try it together with rar2fs. But haven't had they time. Probably fine to close this issue IMO. |
|
Seems like cmdfs might cover your needs, despite the fact that it does not seem to be a very active project. I will close this issue since there are nothing so far pointing to it being a problem with rar2fs. Feel free to re-open if there are any new findings. |
So I've been having issues with my first stack of rar2fs getting unmounted when I scan my library in Emby together with rofs-filtered.
Emby is mounted on the 2 rofs-filtered systems.
This is my setup (although I've tried various combinations, such as removing cache options, and without splice options)
rar2fs --exclude=.lock --seek-length=1 --relatime -d -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0 /mnt/user/share/fin /mnt/user/rar2fs_level1rar2fs --exclude=.lock --seek-length=1 --relatime -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0 /mnt/user/rar2fs_level1 /mnt/user/rar2fsrofs-filtered /mnt/user/rofs -o source=/mnt/user/rar2fs -o config=/etc/filter1.rc -o kernel_cache -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0rofs-filtered /mnt/user/rofs_ful -o source=/mnt/user/share/ful -o config=/etc/filter1.rc -o kernel_cache -o auto_cache -o direct_io -o max_write=131072 -o splice_read -o splice_write -o splice_move -o allow_other -o nonempty -o umask=0000 -o uid=0 -o gid=0The scan in emby runs for a few minutes, then comes to a halt.
The 2 last attempts with debug enabled on the rar2fs_level1 resulted in the output below.
I'm not sure what I am looking at here, the first attempt clearly has errors, but the second seems to unmount without errors.
Any pointers on what I can read out from this, or look further into?
read[23159808100864] 32740 bytes from 1440712 flags: 0x8000 read[23159808100864] 32740 bytes from 1440712 unique: 377498, success, outsize: 32756 unique: 377500, opcode: FLUSH (25), nodeid: 4390, insize: 64, pid: 12911 flush[23159808100864] unique: 377500, success, outsize: 16 unique: 377501, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 unique: 377502, opcode: RELEASE (18), nodeid: 4390, insize: 64, pid: 0 release[23159808100864] flags: 0x8000 INTERRUPT: 377500 unique: 377502, success, outsize: 16 unique: 377504, opcode: OPEN (14), nodeid: 4390, insize: 48, pid: 12920 unique: 377501, error: -11 (Resource temporarily unavailable), outsize: 16 unique: 377505, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 377504 open flags: 0x8000 /Movies/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE/Subs/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE.rar open[23159807905184] flags: 0x8000 /Movies/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE/Subs/Extremely.Loud.and.Incredibly.Close.2011.720p.BluRay.X264-AMIABLE.rar unique: 377504, success, outsize: 32 unique: 377506, opcode: FLUSH (25), nodeid: 4390, insize: 64, pid: 12920 flush[23159807905184] unique: 377507, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 377506 unique: 377506, success, outsize: 16 unique: 377508, opcode: RELEASE (18), nodeid: 4390, insize: 64, pid: 0 release[23159807905184] flags: 0x8000 unique: 377508, success, outsize: 16 rar2fs[6743]: unmounted /mnt/user/rar2fs_level1And
read[22712525407936] 32740 bytes from 1571641 flags: 0x8000 read[22712525407936] 32740 bytes from 1571641 unique: 374466, success, outsize: 32756 unique: 374468, opcode: LOOKUP (1), nodeid: 1, insize: 47, pid: 18693 LOOKUP /Movies getattr /Movies NODEID: 2 unique: 374468, success, outsize: 144 unique: 374470, opcode: FLUSH (25), nodeid: 1818, insize: 64, pid: 22959 unique: 374471, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0 INTERRUPT: 374470 flush[22712525407936] unique: 374470, success, outsize: 16 unique: 374472, opcode: RELEASE (18), nodeid: 1818, insize: 64, pid: 0 release[22712525407936] flags: 0x8000 unique: 374472, success, outsize: 16 rar2fs[17365]: unmounted /mnt/user/rar2fs_level1The text was updated successfully, but these errors were encountered: