Crash when deleting 10,000+ files at once

[tullebo]

Google File Stream remained stable and open, but cpp crashed fully, albeit gracefully. This occurred on two separate occasions when deleting a 5-10 thousand files at once of a single folder.

[bailey27]

I really wish Google would make file stream available to people like me who don't have a school or work gmail account.

I cannot figure out any way for me to get at a Google File Stream drive to test and debug, so I am afraid I cannot work on this issue.

As I understand it, Google File Stream is an installable filesystem (like cppcryptfs or encfs). I'm not sure if it's user mode like or if it's kernel-mode.

But either way, using it with cppcryptfs is running cppcryptfs on top of another installable filesystem. Like running it on top of encfs.

So I am not surprised that it isn't 100% solid due either to mis-behavior of itself or bugs in the underlying fs.

I created 10,000 files and deleted them on cppcryptfs (with a normal NTFS volume as the backing-store). It didn't crash. I did this twice.

One thing I did notice was that when you delete a bunch of files in Windows Explorer, it appears to find all the files, if there are any files, then it deletes ONE file, and then it finds all the files again. It does this over and over until there are no more files.

This is extremely expensive and silly. cppcryptfs has to decrypt all the file names every time it finds all the files.

It takes several minutes to delete 10,000 files this way.

If I go to a command prompt and do

del *.*

It is really fast.

[tullebo]

Thanks for your thoughts! I know it's not a typical use-case, and happened during some file cleaning. The Google File Stream layer is itself extremely slow and requires massive CPU just to explore simple folders with a few files. I'm surprised it's still in the state it's in after a full year.

But it good to know that the del command handles the deletion so much more efficiently. I used robocopy and Beyond Compare for most of my backup - Windows Explorer's file handler hasn't been the most reliable.

Thanks again.

[trumblejoe]

@bailey27 Possible you could replace Google Drive File Stream with Rclone Mount? It features a debug mode. I use it over GDFS in my own setups on linux. NaiDrive is another free alternative which I use on windows, but it doesn't feature debug mode that I'm aware of.

[ccchan234]

@bailey27
Hi~ then may I ask how you put files on google drive, and read in 2nd PC?

What I think after reading your post:
PC A: (HDD--plain text) cppcryptfs <---google sync and backup ---> google drive?
then PC B: google drive sync --> cppcryptfs (plain text --- HDD)?

For the last year I use just like OP,
i use by installing GFS, then put the cppcryptfs container there and mount it, ON THE FLY.
there could be some minor errors as GFS didn't really 100% sync/update files with certainty. (it
use cache, and I think sometimes it didn't flush well) The good point is,
I got 7TB of file on google drive, using GFS, I can work with my laptop which only have a C drv
of 128GB.

thank you.

[ccchan234]

@bailey27

btw, onedrive + win10 can also have on demain file streaming.

yet, onedrive ONLY support around 260-400 character,
I found cppcryptfs could easily got error as encrypted file name is too long.

I am screwed because I wanna to use rclone to mirror my files from google drive to onedrive,
but the file/path name in google drive is 3xxxx, while on onedrive is 260-400, so no luck.

indeep cppcryptfs mainly only works OK with google drive, and seems also open drive (but slow for me in Asia).

other trials of mirror from GD to onedrive, box/dropbox are no luck.

Thanks

[BrsyRockSs]

I have encountered similar problems, but some things are different

Software ver.
cpp 1.4.3.11
Dokany 2.0.0.2000
ntfs
No file name encryption

When I delete any file, I will stay for a long time at the beginning. If I delete multiple files at the same time, it will get stuck for a while after each file is deleted

I have noticed your previous reply

One thing I did notice was that when you delete a bunch of files in Windows Explorer, it appears to find all the files, if there are any files, then it deletes ONE file, and then it finds all the files again. It does this over and over until there are no more files.

The solution I'm using now is to create a bat file with the following contents:

@echo off
DEL /F /A /Q \\?\%1
RD /S /Q \\?\%1

It's really fast to delete

But in terms of this problem itself, the deletion function of win Explorer is almost unavailable.
Is this a common phenomenon? Because I remember that the previous version was a little slow, but at least it could be used

[BrsyRockSs]

I tested other versions
cpp 1.4.3.10
dokan 1.5.x
Deleting empty files is very fast without any catons, after I changed to a new version, the problem reappeared