Project status?

[s-m-e]

Same question as 16 months ago - what's the status of the project?

Since I last asked, some stuff has happened: New maintainers, a number of accepted pull requests, even more discussions about the project and its future. That's good. On the other hand side, the past 6 month have yet again seen stagnation and long standing, apparently not reviewed, significant pull requests. That's, well, not so good.

fusepy - as it is today - does still not allow a POSIX compliant file system on top of it due to unfixed issues. Meanwhile, libfuse 3 has been out there for a rather long time now, which is not reflected in fusepy at all. Linux Kernels 4.20 and 5.1 have just seen a massive improvement in FUSE functionality (and performance), very unlikely to ever be picked up by libfuse 2. Besides, fusepy still does not even have rudimentary tests. Yet strangely, there has been a 3.0.1 release of fusepy with some relatively minor changes on PyPI in September 2018 directly following 2.0.4 from 2016. This does not look trustworthy at all.

After more than a year of testing and bug fixing, I got my Python FUSE-pass-through file-system finally to work: LoggedFS-python has had its first release, passing the entire pjdfstest-test-suite for POSIX compliance and, even more important to me, surviving stress tests with fsx-linux. It is build on top of rejected pull request 79 against fusepy. It can now serve as a proper foundation for tests on fusepy if desired, effectively testing against the bulled-proof EXT4. My test procedure is based on a file-system-image-file / loop-back-device design, allowing to bisect the EXT4 file-system after I have run tests of my pass-through file-system on top of it. I have written quite a bit of infrastructure code for that.

Why have not I contributed any further to fusepy? I prefer to not spend time on pull requests that get rejected or not even acted upon while the direction the project has not been agreed upon by its maintainers.

My plan (still) is to build another, somewhat larger project on top of LoggedFS-python, which would require stable, mature and well-maintained FUSE-bindings underneath. I have the options of (a) re-basing my project to the now again sort-of maintained FUSE-bindings offered here, (b) hard-forking fusepy and maintain the fork myself, (c) writing my own bindings, (d) re-basing my project to the well-maintained but also overly-complicated FUSE-low-level Python bindings offered here, or (e) wait for you guys to get this project straight.

I sincerely apologize for the nagging. Yes, it's an open source project without any commercial backing. So, in all brutal honesty, what are your recommendations?

Would someone reading this be interested in actively supporting option (b)? Same game plan as already detailed in this old comment.

[s-m-e]

@terencehonles @eestrada I take three weeks of utter silence from the maintainers as a sign of no interest in this project.

I am forking.

@paulo-raca @rianhunter @N-Coder @billziss-gh and everybody else who is interested in this project: This is an open, wholehearted invitation to join the effort.

[eestrada]

@s-m-e I had recently been considering removing my name from the project because I have had personal/familial concerns over the past year or so that have consumed any time or energy I may have had available for this project. Your hard fork gives me as good a reason as ever to follow thru on those considerations. Given the scope of what you hope to accomplish, a hard fork seems like a better idea anyway.

If you (or someone else) wants to have maintainer privileges on this project, @terencehonles is the one to talk to. Good luck with the new project. 🙂

[rianhunter]

I've already forked this project:

https://github.com/rianhunter/fusepyng
https://pypi.org/project/fusepyng/

My fork supports OpenBSD. Feel free to pull my changes, or collaborate on my fork.

[paulo-raca]

Hello, all.
I didn't reply before because I'm not really a maintainer, I just did minor contributions in the past.

I did, however, write a high-level wrapper for fuse: Fusetree.
It's based on fuse-lowlevel + asyncio, and each inode is represented by a object.

It tries to cut the boilerplate for simple filesystems by doing the right thing: str/bytes arrays becomes read-only files, dicts become folders, generators become non-seekable read-only files, etc -- Yet, you can also write fully-customized inodes ;)

It is currently based on fusepy, as it was the best implementation I found at the time, but it may change if something else comes up ;)

---

Said that, I think this is a good moment for a new library:

• Python2 is finally gone(-ish)
• Fuse3 is finally on mainstream linux distros

I'm quite busy with other projects right now, but I could become a contributor in the future ;)

A few suggestions:

• Coordinate so that we don't start too many forks :P
• Drop Python2 and fuse2 support.
• Take advantage of @DataClass instead of dicts on various data structures.
• Use asyncIO instead of threads.

[billziss-gh]

@s-m-e, @rianhunter and others.

I will be happy to help bring refuse and/or fusepyng to Windows. It is my understanding that fusepy already had some patches to run on Windows so likely both projects have some support for it already.

Unfortunately I rarely work with Python these days so I would not be able to make any more major contributions.

It would be great if the two forks found a way to work together :)

[s-m-e]

@eestrada Thanks.

@rianhunter Yes, I saw your fork. Two major points of criticism, if you do not mind: First, your repository does not allow people to report issues. It is my understanding that issues are now disabled on GitHub by default for new forks, so you might just need to actively activate them. If there had been issues, I would have pinged you there ;) Second, you published a 1.x release on PyPI. Yes, you added OpenBSD support, which is a serious and pretty cool new feature, but otherwise, it's still fusepy with all is quirks in its current condition - meaning without tests of any kind. So this is still alpha, as far as I am concerned, and people should not be mislead into thinking otherwise. It's not the kind of fresh start that I envision. Feel free to contact me directly via e-mail and let's discuss how we can bring one of our forks forward :)

@paulo-raca Fusetree looks cool. Thanks for the hint. This is something I have been looking for ... Besides, FUSE3 has been picked up by the faster moving Linux distros, yes, but some of the enterprise ones still do not have it: openSUSE Leap 15.1 (the upcoming release of Leap) and SLES for instance. It is not quite there yet. Besides, I agree with your suggestions and I'd be very happy to welcome you as a contributor. Meanwhile, there is also some new work coming from the libfuse people: pyfuse3. It's async and uses cython. Pretty nice design but still beta, not exactly trivial and requires a compiler to work. I'll stick with the "simple" cross-platform ctypes approach of fusepy, at least for a while, and will keep observing how their work progresses.

@billziss-gh As far as I can tell, a lot of the changes you made in fusepy got merged in 2018. I'll have a close look at the differences that are left and come back to you, if you do not mind. I am also going to need some help with testing / setting up CI tests for Windows.

I opened up a literally first issue in refuse for continuing this conversation :)

[billziss-gh]

@s-m-e yes, feel free to ping me. I added myself already to your "kick off" issue in refuse.

[rianhunter]

@s-m-e The issue thing was not intentional. I'm not sure it's reasonable to consider fusepy alpha software given it's stable, more than 5 years old and has thousands of users, but perhaps that gives you motivation. fusepyng exists purely so that new dbxfs features can be released without blocking on fusepy maintainers. I don't intend to spend my time proactively rewriting fusepy or changing its API without any clear tangible benefit to dbxfs, but I also won't ignore contributions from others if they want to contribute.

[s-m-e]

@rianhunter What makes it alpha in my eyes is the fact that its (fusepy's) test strategy is (1) release and (2) see if users' stuff breaks ;) "Very careful editing" is not a test strategy. A filesystem is about trusting a piece of software with your data, is not it? At this point, I only trust my own old fork, because I have literally tested the hell out of it. But this is only my personal use-/edge-case of fusepy. And there we are: Many serious projects, which work with fusepy, somehow end up maintaining their own fork. So do you as well, I'd say. I am very much in favor of consolidating the effort, but in a well maintained, solid and tested way. No release before it is as clear as humanly possible that there is not risk of loosing or corrupting data. That would be my motivation.

[paulo-raca]

Thanks for pointing me to fusepy3, it fits my needs perfectly -- however I understand it is a "low-level" API and maybe unsuitable for most people

[lavdnone]

Have to use this for some semi production cases because pyfuse3 often cracks under real loads, not mentioning that it kills performance 2.5 x, so @paulo-raca I would not jump to pyfuse3 for non recreational purposes