mergerfs v3.X

[trapexit]

I'm not sure when it will be coming out since I really need to focus on my 3DO ODE menu work but I wanted to start a discussion regarding v3.X.

Here is a rough list of things I have in mind. Some of them are user facing and others not so much but listed to be thorough.

• Remove policy path preservation: just have the policies narrowly defined as they have become and make the behavior of rename and link explicitly defined. In effect: remove ignorepponrename and create special rename and link policies or flags per function indicating path preservation behavior.
• add support for upcoming file descriptor passthrough feature (may seriously improve read and write performance, though will lead to loss of certain features)
• remove deprecated options (all the libfuse options)
• on Linux use statx to grab stat info (could be perf improvement when used with network filesystems)
• Support rename2 if possible.
• port libfuse to C++
• replace libfuse thread management
• use thread pools to process certain filesystem calls concurrently (such as readdir)
• better support for tiered storage (list of list of branches?)
• general policy caching as part of policies to limit syscalls
• remove redundant policies: IE those "create" policies that exist for search and action but don't make much sense.
• live mirroring (maybe, still not a fan given the complicated error handling)
• remove "debug" mode and replace with proper logging that can be switched at runtime
• feature to turn link calls into "reflink" copies (like cp --reflink)
• documentation: move to readthedocs.io
• toml config file (which would likely result in removal of most mount options, leaving it to the config file)
• remove xattr & ioctl runtime interface in favor of reloading toml config file?
• move files on failed rename: with overlapping files across branches if one rename succeeds the whole call succeeds but will try to remove failed renamed files to keep consistency. may be be better to try to move them to a special directory.
• freedesktop.org compatible trash
• async writes (return to kernel that a write succeeded but do the actual writing in a separate thread out of band to decrease latency)
• buffered writes (like cache.writeback but in mergerfs to handle larger buffers)
• synology synoacl support
• FOPEN_NOFLUSH support
• new FUSE supplemental groups extension (could improve behavior by removing need to cache supplemental groups which leads to inconsistent behavior occasionally, though recently I did add cache clearing to help work around changing of groups while mergerfs is running)
• lookup/getattr policy cache to remove always needing to search all branches to find a file. though this would have limited value if mergerfs needs to combine details of numerous branch content (like directories)
• optimized / bespoke policies per function. would provide more opportunity for function specific behavior and optimized lookup
• config wizard: a simple tui to walk users through updating config
• rebalance / relocation / duplication daemon... similar to drivepool or unraid
• ptrace / ld_preload based shim to simulate the yet to be create kernel file passthrough feature.

Looking for suggestions / ideas. I think v3.0.0 will mostly be the changes to the arguments since that's the real breaking change and the features will come later.

[kinghat]

front end dashboard and management 😁

[topin89]

Well, I just want my qBittorrent to work well on HDDS and I need mmap support and I want not severe performance degradation while torrents are downloading to the point where speed is barely tenth of channel maximum. Here. That's what I really want.

So, let's walk the list.

TL;DR: Yes for removing unneded bugsflags. Big yes for better performance (fd passthrough, thread pools, caching). Meh for features that are not useful for torrenting and Samba share. Honest why for code reorganization. And random ideas not in the list at the end.

Remove policy path preservation: just have the policies narrowly defined as they have become and make the behavior of rename and link explicitly defined. In effect: remove ignorepponrename and create special rename and link policies or flags per function indicating path preservation behavior.

Just read what this mean. TL;DR for those who have no time: mergerfs have different policies on path (folders) preservation. Some will create new folders if needs be, like on move/rename, some won't. This flag says ignore chosen policy and allow link and rename syscalls and therefore tools like mv and mklink to always make new folders on a drive. Given that those with preserve path policies most likely don't want that anyway, and those without preserve won't care, and finally that's one flag and bugspace less, I vote yes.

add support for upcoming file descriptor passthrough feature (may seriously improve read and write performance, though will lead to loss of certain features)

YES! Every bit of improvement is good, and serious improvements are exceptionally good.

remove deprecated options (all the libfuse options)

Less flags, less bugs. Yes, please.

on Linux use statx to grab stat info (could be perf improvement when used with network filesystems)

As long as local improvement does not noticeably degrade, I have nothing against.

Support rename2 if possible.

Atomic path names exchange, syscall-level checks for existence, and some specifics for union file systems with multiple read-only layers one read-write layer like unionfs or overlayfs. That last flag support is meaningless (unless there is a person how uses mergerfs on top of unionfs), first two sounds fine, but low priority for me.

port libfuse to C++

Will it improve readability? Performance? Reduces frustration? I have no opinion on this, but I really want to know why?

replace libfuse thread management

Again, what will be better then.

use thread pools to process certain filesystem calls concurrently (such as readdir)

Faster means better. Better means yes.

better support for tiered storage

Don't need that, low priority for me. Anyone else?

general policy caching as part of policies to limit syscalls

Caching means faster, but delayed sync with underlying fs. Still, faster-better-yes.

remove redundant policies

Policies. They need better docs, with emphasis on why use this or that, what's better when and why. Like, why use rand policy? Then again, policy as I see are two not entirely independent settings, "path preservation", "use most shared", "where to place new files". I suggest just split them.

live mirroring (maybe, still not a fan given the complicated error handling)

I don't need it.

remove "debug" mode and replace with proper logging that can be switched at runtime

I don't need it either.

feature to turn link calls into "reflink" copies (like cp --reflink)

That is not useful for me, with my underlying ext4 partitions. Still, that could be useful in some cases with underlying xfs or btrfs.

I noticed mergerfs has at least Ubuntu and Arch packages. More than sure there is more. So, if any backward incompatible changes are there, I suggest merger3 for a new packages. And maybe security fixes for 2.x branch for a time, as long as it's like once a three month or less. If more, just drop support, not worth it anyway.

Also, I really want benchmark/profiler. Dead simple at that, language is irrelevant. It should test some load first on relevant FSes independently, then on merger overlay, compare and show results.

Load like: 50 new files from 1M to 1Gb, opened independently and random write with mmap and regular write. Then same for read. Then simultaneous read and write. I swear, on my old AMD box I can't watch what I already downloaded if torrents are active. That could be because of qBittorrent itself, could be because of drives themselves, could be because of merger being that less effective than direct syscalls. I want to know, I need a benchmark.
Also, with benchmark, we could actually test what flags and flag combinations are working, and what aren't.

On documentation. There should be some quickstart guide. And build/install section should be close to the top. Or even better, all settings should be on a separate page with link to it in the readme.

Well, hope this post is at least helpful.

[trapexit]

I noticed mergerfs has at least Ubuntu and Arch packages. More than sure there is more. So, if any backward incompatible changes are there, I suggest merger3 for a new packages. And maybe security fixes for 2.x branch for a time, as long as it's like once a three month or less. If more, just drop support, not worth it anyway.

I don't own or manage those and most of the time they are out of date. I can try to track them down and suggest such things but not really in my control.

[trapexit]

Also, I really want benchmark/profiler. Dead simple at that, language is irrelevant. It should test some load first on relevant FSes independently, then on merger overlay, compare and show results.

You mean included in the docs? Any random benchmark can work. I have suggestions in the docs. I'm not how useful it is to include actual numbers given real world workloads vary greatly as does hardware. It isn't trivial to nail down the precise reason why something is slow. What you're talking about is a tool which would attempt to find bottlenecks. It's a given that mergerfs is worse than not. It adds an additional layer of indirection and does so by requiring the back and forth from user and kernel space. What you're looking for is more of a bottleneck detection tool which is a project all in itself.

[trapexit]

On documentation. There should be some quickstart guide. And build/install section should be close to the top. Or even better, all settings should be on a separate page with link to it in the readme.

https://github.com/trapexit/mergerfs#basic-setup

Suggestions and PRs welcome.

Most users wouldn't be building it themselves. I'm not sure I understand the reasoning for putting it closer to the top.

Regardless, the docs are getting overhauled anyway. The purpose for one document was so searching via one's browser or in the man page was simple. Apparently that's not what people do / want so I'm likely to strip the README to something smaller and create a readthedocs.io page. The fact is the product is very flexible and that's not going away. There will always be a lot of documentation.

[topin89]

Some quick reply today, everything else after a well-deserved sleep.

What bugs?

Only potential. The less configure options, the less chance to make a bug on some weird option combination. I didn't see a single bug so far.

It doesn't fundamentally change anything nor remove flags. It simply changes the manner in which you config the behavior.
But won't it remove ignorepponrename and all related ifs?

That shouldn't happen. Many people including myself are using mergerfs at large scale at high speeds.

That's why I want a bench. I want to know at exactly what kind of load this happens, and especially if it happens even without mergerfs. Maybe it's just SMR HDD being SMR HDD. Maybe 500+ torrents is not something my old two core AMD E-350 can run in general (despite htop saying only 70% of cpu are used).

I'll need to know basically everything about your setup

Basically, Debian testing, three almost full HDDs 2, 3 and 4 TB, old two core AMD E-350, 4 GB RAM. Details in attachment. In nspawn container with lives qBittorrent, download folder binded to some folder in mergerfs. Container uses host network and has no uid mappings, essentially autostarting chroot.

I'll give you whatever you ask.

I can try the O_DIRECT thing as well as reach out again to the FUSE maintainer and see if there are any new suggestions in handling this situation.that make using it in a general way difficult. FreeBSD interestingly supports mmap without page caching in their latest FUSE rewrite though it has had other issues.

Well, maybe someday. I get that it's hard and I definitely won't press for it. It's just, libtorrent2.0 uses it, and so will qBit at some point. I'll try to turn off caching, maybe I won't need this as much for now.

But it won't be in the kernel for a bit of time and has been discussed for years so... don't hold your breath. If people want to compile their own kernels to use it I'm fine adding it to the main release in whatever API the patch currently offers.

Pity. Maybe DKMS patches? It requires some build tools, but at least it's a standard way to apply a patch. At least Intel Realsense uses it in its debs. I would definitely go for that, but again, I compiled statically boost, qt, libtorrent and qBit just because I can and because non-zero chance on some linker optimizations.

You keep saying bugs. There are no known bugs in mergerfs at the moment. Flags != bugs. All flags are functional and have purpose. Just some have been simplified or superseded.

Potential bugs. Yup, I should have said that clearly. Sorry. Still, I'm pretty sure it's easier to make some mistake on some combination if there are more flags.

It's one of the most requested features.

😅 Then go for it. Most requested features should be top priority.

In most cases it probably wouldn't be materially faster. It'd more likely reduce load.

On a really old PC, that may be even better.

Creating a per uid/gid pool of threads would help there. The ways in which data is pulled off /dev/fuse and moved around could be improved as well which can help with raw throughput.

Cool. Definitely do want.

I've been slowly removing / absorbing libfuse as it's API was lacking for certain needs (outside a full rewrite.)

So it's less like porting from scratch, and more like accept reality and port remaining parts?

I don't know what you mean by "split" them. Split what?

Policy as I see it, consist of two settings. Path : existing path, most shared path, create missing folders. With corresponding prefixes ep-, msp- and without a prefix. And placement: first found, least free space, least used space, most free space, percentage random, newest, random. With a special read-only case (which is technically "no placement" placement). I'm not insisting on anything, that's just how I see this.

ext4 supports it. It is xfs that just recently gained support.

What? Ext4 supports reflinks? As in, same file until first write, CoW reflink? In ext4? There is no mention of that anywhere, except for a seven year old proposal. When did that happen? If it's not a mistake, that's wonderful news.

I don't own or manage those and most of the time they are out of date. I can try to track them down and suggest such things but not really in my control.

I suppose a warning on backward incompatibility and suggestion for maintainers on Release page and Readme should be fine. It's generally good to know for everyone anyway.

What you're looking for is more of a bottleneck detection tool which is a project all in itself.

Not insisting. Still, it's hard to meaningfully optimize without some kind of a profiler. There was a good chance you had some internal tool that could be adapted.

Most users wouldn't be building it themselves. I'm not sure I understand the reasoning for putting it closer to the top.

Still, those who want to, would want to found the way to build before a way to use it. I know I would. OK, reasoning is weak here.

Just to be clear, you asked for opinions and suggestions. I gave mine. If there a lot of people like with the same opinions, they are good to know. If not, they are good to ignore.

And thanks for the mergerfs in general. A real saver from "where to save this" problem, and simple at that.

[trapexit]

But won't it remove ignorepponrename and all related ifs?

It will remove that flag but will add others to control their behavior.

That's why I want a bench.

If the CPU is high'ish but performance drops check load. Load is a decent indication of latency since it includes outstanding IO requests. You can change the number of threads in mergerfs too to see how that changes. I've seen systems with slow, high latency devices, like SMR drives, cause the load to shoot up. It can also increase load by queuing up a bunch of requests to FUSE but then mergerfs can only respond so fast. There are some things with the kernel that can be tweaked but never found it to make much difference. Adding options splice_read, splice_move, and/or splice_write might help.

Pity. Maybe DKMS patches?

Perhaps but the patch and interface are still in the works. V13 of the patches dropped this week but they are still discussing the things.

Policy as I see it, consist of two settings. Path : existing path, most shared path, create missing folders. With corresponding prefixes ep-, msp- and without a prefix. And placement: first found, least free space, least used space, most free space, percentage random, newest, random. With a special read-only case (which is technically "no placement" placement). I'm not insisting on anything, that's just how I see this.

Still not quite following. There are three general classes of function. Those that create a file, those that modify a file, and those that return information on a file. Each behavior has certain things that need to be accounted for such as "search" functions not being able to return more than 1 piece of data but there possibly being more than one source of information. The policies were created in the code to be interchangeable originally to make things cleaner but I've given up on that due to the inherent differences. This is why I'm saying I'm going to remove the redundant policies in 3 and just go with more bespoke policies where needed. Like rename and link which require all 3 'categories' of behavior.

ext4

I thought I had saw work a while back on cow for ext4 but maybe it never got in or finished. My bad.

Not insisting. Still, it's hard to meaningfully optimize without some kind of a profiler. There was a good chance you had some internal tool that could be adapted.

I've got the same profiling tools anyone else does. Cold spot optimization isn't the easiest to do (which I suspect is the typical issue). And I've not really tried to profile the kernel as I'm not super familiar with the space. In other cases I simply haven't been able to replicate issues. People have complained about high end systems being slow when some random ARM SBC runs fine. With so many combinations of variables it's been difficult to get good reading on those issues. If someone has suggestions on measuring that I'm all ears.

Just to be clear, you asked for opinions and suggestions. I gave mine. If there a lot of people like with the same opinions, they are good to know. If not, they are good to ignore.

Sure. I'm just discussing what was asked/said.

[topin89]

The policies were created in the code to be interchangeable originally to make things cleaner but I've given up on that due to the inherent differences. This is why I'm saying I'm going to remove the redundant policies in 3 and just go with more bespoke policies where needed. Like rename and link which require all 3 'categories' of behavior.

Then whatever I tried to say would be irrelevant in the nearest future. Good, actually, it's kind of convoluted topic.

Still not quite following.

What I tried to say, there are 20 policies, each can be applied to a three categories of FUSE functions(action, create and search):

all
epall
epff
eplfs
eplus
epmfs
eppfrd
eprand
erofs
ff
lfs
lus
mfs
msplfs
msplus
mspmfs
msppfrd
newest
pfrd
rand

Of them, there are two special cases: erofs, and newest.
Everything else is two modifiers combined in one, path preservation policy("don't preserve", "preserve", "preserve most shared") and drive selection policy ("all possible", "first found", "least free space", "least used space", "most free space", "percentage free random distribution", "random"). They feel like two separate settings for me, like "ep+ff" or "msp+pfrd". If they are not, then they are not, but it could help the docs with some clarity. Again, if this is all going to be changed, that's a moot point.

[trapexit]

Ah. I understand now your point. I'm not sure I see it that way in that at any time T0 there exists or doesn't exist certain relative paths on a branch. One can argue that is "drive selection policy". The behavior of mergerfs is not to create or not create paths based on the policy. It is that the policy considers those things when choosing a branch and if that target branch doesn't have the relative path it is created/cloned. Path preservation isn't outside the policy. It is inside it. A policy is exclusively an algo to choose a branch. Nothing more. It is logically pure.

Sure, you could break them down like that but ultimately policies are single, bespoke algos. f(branches,fusepath) = [branch]. That is the signature. To abstract things like "newest", "erofs", "all" and the others requires boiling it down to that. "all" is different from all others in that it returns N entries. Others return 1. It's somewhat complex to manage all these. People often forget that the platform has to manage overlapping files and what to do isn't always clear. Most people don't have that situation but it is a common pattern for a number of usecases. mergerfs could be far more opinionated and my life would be far easier but then there would just be a ton of people asking me to expand the functionality.

So yes. I could describe policies as you split them but internally it'd still be as today. And I honestly don't know that'd be easier for people to grasp.

There is also a practical aspect in that some of those algo would be less efficient to implement as a combination of things. Not always significantly but it would be there.

[chrishoage]

better support for tiered storage (list of list of branches?)

Not sure if this is feasible, as my knowledge of filesystems is very limited.

When doing category.search operations I wonder if it's feasible to do ff checks in order of the configuration.

So for example if you had /mnt/cache:/mnt/data it would forward to /mnt/cache first then if not found it searches /mnt/data after that.

If I had to guess though the "if not found" part is the part that is not feasible or too error prone to be worth it.

I'd be willing to trade absolutely terrible seek time for energy savings of not hitting a disk if it isn't needed

based on my read through of https://github.com/trapexit/mergerfs/wiki/Limit-Drive-Spinup the suggestion of giving things like plex the raw filesystems is a little problematic with moving files from /mnt/cache to /mnt/data as sometimes plex gets confused when files change to a new location, even if it is moved first, scanned, then deleted (then scanned).

In a perfect unicorn world I can move things on the backing filesystems and plex doesn't see any move.

I very much understand if this just can't work for any number of reasons, but if it can work it would be interesting to see in some (far) future version.

Thank you for your work here, it's amazingly solid.

[trapexit]

Not sure what you mean. ff IS in order of config. It wouldn't make sense otherwise. There is no "first" without order.

[trapexit]

I'd be willing to trade absolutely terrible seek time for energy savings of not hitting a disk if it isn't needed

How do you possibly determine such a thing? You can only know something by looking.

[chrishoage]

Not sure what you mean. ff IS in order of config. It wouldn't make sense otherwise. There is no "first" without order.

My (poor) understanding is that ff searches though all backing filesystems first, then does a sort afterward, using the configuration to give the path found first.

Is that not the case? Will it not hit all first then returned the best match?

How do you possibly determine such a thing? You can only know something by looking.

I perhaps have explained poorly. Basically what I am curious about is if is possible to do a "blocking" sequence search of the configured disks.

[trapexit]

ff works exactly as listed in the docs. No more, no less.

Given the order of the drives, as defined at mount time or configured at runtime, act on the first one found.

It doesn't do any more scanning than needed. Nor does any other policy.

I perhaps have explained poorly. Basically what I am curious about is if is possible to do a "blocking" sequence search of the configured disks.

I don't know what you mean. ff is a sequential search of configured branches. Just as described in the docs.

[chrishoage]

Thank you for the clarification. I think I had confused myself from the mentions of "sort branches" in the docs. I had understood it to search all backing drives then sort the result and return it.

Apologies for the confusion on my part!