Linux Status

[nlieb]

I'm just curious to know whether there has been any internal progress or designs for supporting Linux. I'd like to start prototyping a stackable kernel mode file system for GVFS based on wrapfs, but I don't want to be duplicating work or diverging from internal design decisions.

[sanoursa]

We have not yet done any serious design work for Linux. I would love to see a prototype of a stackable file system on Linux! Our goal is to be able to reuse the same user mode code, which implies needing a very similar driver model, on all three OS's. The beginnings of our Mac kext is now available in the repo, though it does not yet reflect all the events that we require to make GVFS fully functional.

To get started, we would need the ability to intercept directory enumerations, and file reads/writes/renames/deletes, with the ability to fail those operations if needed after interacting with a user mode process.

[nlieb]

Sounds good, I'll be sure to reference the MacOS implementation and GVFS service and see what can be done as a rough prototype. Unfortunately there aren't any Kauth-like APIs in Linux like there are in MacOS (as far as I can tell), but the general flow should be good enough to start with.

Thanks!

[sanoursa]

A good starting point may be the MirrorProvider project (https://github.com/Microsoft/GVFS/tree/master/MirrorProvider). It currently implements the set of callbacks that are available on both Windows and Mac (which for now is only dynamic enumeration and file hydration), and we'll keep extending it as we add Mac capabilities.

Once you've got that covered, you can look at the full set of callbacks currently used on Windows here: https://github.com/Microsoft/GVFS/blob/master/GVFS/GVFS.Platform.Windows/WindowsFileSystemVirtualizer.cs. Keep in mind that not all of those need a direct port, e.g. enumeration on Windows requires 3 callbacks but on Mac requires 1 callback. But that file will give you a sense of the specific notifications we require.

[nlieb]

Thanks, I really appreciate the guidance. MirrorProvider seems ideal for prototyping.

Out of curiosity, what purpose does the MirrorProvider serve? I see that it basically mirrors files without hitting the network. Is it just to stub out the callbacks for driver testing?

[sanoursa]

Exactly right, it's a test bed for the driver. It serves 3 main purposes:

• It is a simple provider implementation, so it makes it easier to test driver changes by eliminating any other sources of errors
• It allows us to run real builds on top of the virtual file system, to test out those file system interactions. The most common test we run with MirrorProvider is to do a normal git clone of any real-world repo, then mirror it, and then run a build of that repo on top of the mirror. This tests the enumeration and hydration callbacks, as well as everything else that the build wants to do in the file system.
• It will grow into a sample for creating a cross-platform provider.

[wrighty]

@nlieb I just wanted to say I'm watching this issue with interest!

At GitHub we're hiring for people to work on the Linux client (I'm the hiring manager). When we have people in that role we'll be working in the open on plans to bring VFS for Git support to Linux and collaborating with anyone who's already started on this work.

[cedric05]

after watching https://www.youtube.com/watch?v=cwnM5HDymik&index=3&list=PLtvmFy4Ed_DJiFvo3_TztWzDMZYMBAHdg . for porting to linux , native file changes can be achieved using kqueue i suppose? just curious

[sanoursa]

A linux port is now underway: https://github.com/Microsoft/VFSForGit/tree/features/linuxprototype. Closing out this issue as a general question.

[chrisd8088]

See also libprojfs for ongoing work.

[WesleyYue]

1. What is the current status of this? The linuxprototype branch has not received an update since Oct 2019 and libprojfs hasn't been touched since Jun 2019.
2. Can this issue be re-opened to track the status as a feature request and remain open until linux support is added so that interested individuals can subscribe to the issue for updates?

[chrisd8088]

Work on VFSForGit has largely transitioned over to the Scalar project instead, for which there's an excellent introduction by @derrickstolee.

There's also a specific issue (microsoft/scalar#323) dedicated to collecting use cases for Scalar on Linux, so if you are considering Scalar and have Linux requirements, please do add them there!

That said, the VFSForGit linuxprototype branch passes the functional test suite (with some inapplicable tests excepted, and one flaky test in the SparseTests), and I've been aiming to keep it up-to-date with master, but have been slightly delayed of late; I'll try to tackle that soonish.

Going forward we would have needed to tackle one significant challenge, described in github#34 (comment), which relates to behaviour after renaming. My guess is that Scalar avoids that issue entirely (but am not 100% on that myself).

As for libprojfs, while not especially performant, it does support VFSForGit now as it stands. If you or anyone has additional contributions to improve performance, we will of course look at them; just open a PR in that project.