Slow for many small files

[jp341]

I performed a benchmark between Boxcryptor Classic v1.5 (modified encfs using CBFS), EncFSMP (encfs using PFM), and cppcryptfs (gocryptfs using dokan).

As follows:
(1) source directory on SSD 1 containing 29,289 files measuring 1.68GB (some large files, but many small files, including some .git and .csv repositories)
(2) create new encrypted drive on SSD 2 + then mount
(3) ensure no cpu heavy processses running
(4) using FreeFileSync synchronize the SSD1 source to mounted destination

Results as follows:

1st - Boxcryptor
Time Elapsed: 4mins 43 sec
Average items/sec: 103
Average MBytes/sec: 6.08

2nd - EncFSMP
Time Elapsed: 6 mins 55 sec
Average items/sec: 71
Average MBytes/sec: 4.15
(* note: Chained IV and Unique IV disabled per author's recommendation for speed)

3rd - cppcryptfs
Time Elapsed: 39 mins 57 sec
Average items/sec: 12
Average MBytes/sec: 0.74

Two main observations:
(1) particularly slow for many small files, <20Kb/s
(2) cppcryptfs does speed bytes/sec up for large files (reaching above 10 Mb/sec), but notably # items processed per second stays low

Attached progress charts.

[jp341]

Ran another test for encfs4win using dokany 1.0.1, and the results were the worst of all candidates.

4th - encfs4win
Time Elapsed: 1hr 20 mins 36 sec
Average items/sec: 6
Average MBytes/sec: 0.37

This would seem to suggest that dokany is a critical reason for the slowness.

[bailey27]

Thanks for running these benchmarks. It's interesting to see how cppcryptfs compares to BoxCryptor and EncFSMP.

I think the performance problem with small files is caused by Dokany. cppcryptfs performs about the same as the Dokany mirror sample program.

I think the "Extract linux-3.0.tar.gz" benchmark I put on the main cppcryptfs github page is similar to your test.

My benchmarks compared cppcryptfs to native NTFS and Dokany mirror.

I think the performance is slow with small files because there is too much latency in the communication between the Dokany client-side DLL and the Dokany kernel mode driver.

If you want, you could maybe do the same test using mirror.exe, and if you get similar results, then file an issue on the Dokany github page.

The mirror program is installed by Dokany here

C:\Program Files\Dokan\Dokan Library-1.0.1\sample\mirror\mirror.exe

[CGsama]

Hi, is there any update on this case? like does dokany did anything on it? cppcryptfs is very useful for store photo against ai training by cloud drive provider. I was thought the problem is cloud drive app, but turns out on two local ssds the problem still, and the disk utilization near zero, made me crazy until see this issue. Or the small file performance better be mentioned at readme? Thanks!