[RESEARCH] Dokany vs WinFsp

[sergeevabc]

Dear @bailey27,
Consider evaluating Dokany vs WinFSP, probably an outperforming alternative, would you?

[bailey27]

I've looked at WinFSP. In my tests it's at least 2x faster than Dokany.

I might port cppcryptfs to use WinFSP in the future, but so far I haven't had any problems with Dokany other than the slower performance.

[sergeevabc]

Dokany released version 2 beta, which seems to be focused on speed enhancements. How does it compare with Dokany 1.x in your tests, @bailey27? Is it still slower than @billziss-gh's WinFSP?

[sergeevabc]

And, by the way, recent changes of GoCryptFS include mention of using memory pools for buffer handling which doubles the streaming read speed on its developer’s machine. Have you tried that, @bailey27?

[bailey27]

Thanks for the heads up. I'll test Dokany 2.

If it looks like it's going to be a lot of work to move to Dokany 2, and if Dokany 2 is still significantly slower than winfsp, then that will be a good motivation to switch to winfsp.

cppcryptfs does not use a buffer pool, but I think using a buffer pool would be a bigger boost in go than in c++. A buffer pool would be easy to implement, though. I will add it to my list.

But I think the streaming read and write performance is quite good already. It's when cppcryptfs is dealing with lots of small files that it is slow. Maybe Dokany 2 or winfsp will solve that.

[billziss-gh]

@bailey27 as discussed before, if you do decide to go the winfsp route you can count on my support.

BTW, while I am not familiar with the buffer pool implementation in gocryptfs, I would expect that a file system that uses the NT cache manager would far outperform anything that can be done in user mode.

[Liryna]

@bailey27 Yes Dokan 2 is still a beta and the stability may not be here for now.
The migration of cryptdokan may not be so hard to do since you based your code on the mirror.
Anyway, there is still some work to do but it would be really great to have feedback/help on it !

[bailey27]

@Liryna , I think the big change in Dokany 2.0 was to make it asynchronous, right?

I did some testing with Dokany 2.0 beta (mirror.exe) yesterday, and it looks to me that it isn't any faster than 1.5 when dealing with lots of small files.

I think the reason why Dokany is slow when dealing with lots of small files is the latency. I think making it asynchronous wouldn't help with that.

I also tested WinFSP (passthrough-64) again.

It looks to me that WinFSP is about 1.5 to 2x faster than Dokany when dealing with lots of small files.

But WinFSP (passthrough-64) is 3 to 4 times slower than native NTFS in my tests.

So I think if I changed cppcryptfs to use WinFSP, it will be faster, but it will still not be fast.

I think gocryptfs doesn't have these problems because it runs on Linux and uses Fuse which is supported by the Linux kernel. In other words, I think gocryptfs is fast because your are actually supposed to be able to have filesystems in user space on Linux, but on Windows you aren't.

@billziss-gh, I realize that passthrough-64 is not the reference fileystem implementation for WinFSP (memfs is). But I think passthrough-64 would be my starting point if I were to change to WinFSP because I am not really implementing a whole filesystem. I'm basically doing what passthrough or Dokany mirror does, but with encryption/decryption.

I'm going to add a i/o buffer pool to cppcryptfs. But I am sure it won't help with the lots of small files speed issue. It may help speeding up streaming reads and writes a little bit, but those are already fast.

[bailey27]

@sergeevabc , I also meant to say that I think if you need performance like native NTFS, then you maybe should try something like VeraCrypt or Bitlocker.

[billziss-gh]

But WinFSP (passthrou#64) is 3 to 4 times slower than native NTFS in my tests.

So I think if I changed cppcryptfs to use WinFSP, it will be faster, but it will still not be fast.

NTFS is hard to beat! After all both WinFsp passthrough and Dokany mirror use NTFS to satisfy the I/O.

But NTFS is fast because it uses the "cache manager" extensively. Effectively when an I/O comes into the file system it becomes a simple "memcpy" from the OS's cache buffers into the application buffers. [In memory mapped cases even that "memcpy" is eliminated as the user mode application accesses the cache mapping directly!]

I think gocryptfs doesn't have these problems because it runs on Linux and uses Fuse which is supported by the Linux kernel. In other words, I think gocryptfs is fast because your are actually supposed to be able to have filesystems in user space on Linux, but on Windows you aren't.

I have not tested gocryptfs, so I am not familiar with its performance. I do have my own FUSE (low-level) file systems and I have not found FUSE to be as performant as you are describing it. I often end up implementing my own user mode buffer cache under FUSE, which seems to improve matters. Some times I even implement a dirname cache as I find getattr too chatty.

At the end of the day a FUSE implementation has to do the same thing that WinFsp and Dokany do. Context switch between the process that issues a file system call to the process that implements the file system, then context switch back. The primary optimization opportunity is: try to satisfy the I/O without context switching to the file system. A secondary optimization goal is: if you do context switch to the file system do it really FAST!

But of course FUSE is very mature by now and probably optimized as well as it can be (at least on Linux). I might try to recreate MEMFS for low-level FUSE and run some benchmarks to get a baseline if I ever get the time.

@billziss-gh, I realize that passthrou#64 is not the reference fileystem implementation for WinFSP (memfs is). But I think passthrou#64 would be my starting point if I were to change to WinFSP because I am not really implementing a whole filesystem. I'm basically doing what passthrough or Dokany mirror does, but with encryption/decryption.

It is actually trivial to enable the cache manager for a WinFsp file system and gain a huge performance boost for free. Unfortunately passthrough will not work well under the cache manager without modifications. The primary reason is that passthrough simply maps its own file system handles onto (different) handles in the underlying (NTFS) file system. Unfortunately the cache manager and other Windows components have the tendency to keep files open for longer periods than those requested by user mode applications and consequently passthrough also keeps its own handles open for long times; this often creates conflicts in the underlying (NTFS) file system.

I believe it is possible to modify passthrough to work around those problems. But it is not a trivial task to do correctly. I may do it some time in the future to enable file systems like yours to easily use it as a base for their development.

---

Some of the reasons that Windows likes to keep files open:

• It will often buffer write requests and "lazy" write them at a later time, often after the file is closed!
• Open, I/O, Close, ... is a very common pattern in applications, so if the file is not "really" closed on the OS side (despite what the application thinks) the cache remains intact and does not need to be "reprimed" at the next Open.

---

Lastly can you describe the scenario that you use to measure performance in more detail? You mention lots of small files, but you do not mention if you just open and close them or you also do some I/O on them, etc. I would like to add a similar scenario to my own testing and see if there are optimization opportunities that I have missed. Thanks!

[bailey27]

Thanks for your explanations. They are really helpful and well-written.

My tests were somewhat informal, and revolved around linux-3.0.tar.gz
https://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.gz

That's because that what gocryptfs uses for benchmarks. It has over 36,000 files in it.

I timed untaring it with cygwin tar, ls -lR (piped in to /dev/null), dir /s/b (piped into >nul), rm -rf, rd /s/q, and also I used native Windows (but 32bit) info-zip to zip it up from the virtual drive onto the virtual drive.

I did the tests with Windows defender realtime scanning turned off.

I wasn't particularly thourough, and the benchmarks were run in a 2 cpu 2GB Virtualbox VM with the VDI file residing on an SSD in a USB 3.0 enclosure.

I was mainly interested in determining if Dokany 2.0 was any faster than 1.x in these cases, and also to re-validate what I remembered from testing passthrough before.

[billziss-gh]

Thanks for the test scenario. It is very useful.

[sergeevabc]

@bailey27, thanks for mentioning notable VeraCrypt by @idrassi and Bitlocker by Microsoft, but the former is about one large container which might be tricky to backup/sync and the latter is a proprietary.

Also there is a promising undertaking for transparent encryption called Cryptomator, but is dependent on WebDAV with so far unresolved 4 GiB hitch. even thought I suggested switching to FUSE or even smooth and solid Pismo File Mount (used by EncFSMP, for example). Be sure to benchmark PFM, by the way.

Meanwhile, Windows users are fortunate to have a FUSE-based SecureFS by @netheril96 to play with.