Mergerfs and dax

[dimi-be]

Hi,

I'm thinking of using mergerfs as it would be a lot simpler than my current setup of spreading folders over drives using bind mounts.

I've been reading through the Readme page and read that data will be cached twice. I only use ext4 on all my drives. Would it be a good idea to mount them with the dax option so all data is accesed directly? This would turn off all read and write caching and would eliminate the issue with the double caching. Did anyone try this? Are there disadvantages?

It's mentioned in the Readme that directly accessing the drive is preferred so that mergerfs can do all the caching.

Regards,
Dimitri

[trapexit]

If you set cache.files=off then it won't cache twice. But if you need mmap support that won't currently fly. In my experience disabling page caching in mergerfs leads to better performance than cached oddly enough. Something I've been talking to the kernel folks about. As for DAX: I've not gotten around to trying it. Not sure if there are any edge cases that need to be accounted for.

[dimi-be]

Thanks for answering.

I would like to use mergerfs on my home folder. I think browsers, media libraries and apps like that use sqlite, so I think I will need mmap and because of that caching.

Currently I have my home folder on an nvme drive. I use bind mounts to map folders like music, documents, pictures,... to slower ssds. I used to mess around with stuff lvm, but I like this solution better and something like mergerfs would be even better.

I'm guessing it's faster without caching because there is less copying happening in memory?

I won't have time to test some stuff today, but I will try and do some tests tomorrow. I will do some tests with and without dax and let you know.

[dimi-be]

I like to have everything in my home folder, it's just a preference/habit I guess. I got the idea of spreading my files over multiple drives from using unraid.

What I would like to do is have most of my home folder by default on a partition on my nvme drive. Then I would like to have stuff that takes up more space but doesn't require speed to my slower ssds.

Do you think mergerfs is not well suited for this use case? I looked at other solutions like aufs and overlayfs and mergerfs looked like the one that could do what I needed.

[trapexit]

I don't think any level of indirection is appropriate for your literal home directory and I don't know any technology that I'd feel secure using to replace a home directory except traditional RAID1/5/6. Even just home directory encryption has its issues. See the latest Linux Mint release. aufs is a dead project and has been for years. It doesn't work with modern kernels last I checked. overlayfs is a CoW filesystem. It doesn't do the same things as mergerfs or unraid's fs.

If you have enough storage for the basic homedir config I'd just create a separate mergerfs mount inside your homedir? Gives you all the features without most of the limitations. It's the same setup used by most people except instead of mounting to /media or /mnt/pool or whatever it's in /home/USER/media.

[dimi-be]

Hmm I see. I guess the amount of disc access that's happening on the home directory can't be underestimated.

It seems like my original idea is not a good solution for me then. I'm going to put the idea on hold and think of a better folder structure that I can live with or maybe just stick with what I have now because that's what is working 😁

I need to think things through a bit. Maybe I can live with the one directory or with mounting something in /media 🤔

Thank you a lot for your suggestions and advice.

[trapexit]

It's not the amount of IO. That's not the issue at all. People run massive servers with mergerfs all day every day reading and writing terabytes a day. It's the type of IO. Home directories are accessed by lots of different software and some can be sensitive to any sort of odd behavior even if totally legit. NFS is not uncommonly used for home directories and its sometimes atypical behavior breaks software not prepared for it somewhat regularly. Moreover you have to properly orchestrate the loading of said filesystem to a location that most of the time is just a dedicated partition which has fewer moving pieces to setup.

It's not that you can't do it. I just don't understand the need for the complexity or risk it brings. I wouldn't want to worry about if some piece of software isn't handling some situation correctly and I need to be concerned that it will freak out or corrupt something as a result. People have and do use mergerfs as their home directory but I wouldn't and I really don't see the point. You can trivially mount to some random location and symlink or bindmount the locations into your home directory and not have to worry about large media storage vs millions of random browser cache files or SQLite3 or anything else. It's more work to get it to be your home directory than not and would only give you more storage (which for just software config and cache is rarely an issue.

[dimi-be]

Ok, I never tried anything like mounting my home dir over nfs. I've also never encrypted any partitions so haven't experienced any of these issues.

After hearing this I agree with you that using mergerfs on my home directory is probably worth the potential issues it could cause.

I'm think of mounting a volume /media and work from that directory instead of my home dir. I could symlink the sub folders to my home dir, but I would need to make sure this doesn't cause any issues. I would like to get rid of the bind mounts, it's getting out of hand.

Another advantage of that option is that I can turn the cache off in mergerfs.