Manage folders

[ncw]

Some of rclones remote fs do understand the concept of folders, eg

• drive
• local
• dropbox

Make an optional interfaces (eg Mkdir, Rmdir) for these FS to manage the creation and deletion of folders. This would enable empty folders, and deletion of empty folders on sync.

[mullenkamp]

I just noticed this issue when I sync to drive. I would definitely like to have folders deleted in addition to the contained files.
Thanks a lot for the copy program, it's the most reliable I've used so far for google drive.

[ncw]

@mullenkamp thanks for the vote of confidence! I can see a plan coming together for this.

[austinginder]

Been using Rclone regularly for the past few months. Works solid. There are a good number of empty directories left in my Dropbox. This would be awesome if it was built in with the sync command.

[memeplex]

This is specially important for synchronizing databases (for example, a git repo) which may:

1. require an empty directory to be there.
2. leave behind a large number of empty directories.

[Wowfunhappy]

Is there any chance that this will be prioritized in the near-ish future?

The longer that I use rclone to sync the contents of my local hard drive with ACD, the more of a problem this is becoming. As files naturally get reorganized and moved around, more and more empty folders pile up on ACD. An aesthetic problem, perhaps, but it's becoming difficult to actually find the files I want, and there's no way to programmatically find and delete the empty folders short of completely deleting the remote copy and re-uploading from scratch, which would take literal months over my internet connection.

Rclone is still useful for creating remote backups should my local hard drive fail, but it's a lot less useful than it once was.

[ncw]

I think what I might do for this is just delete empty folders on the destination at the end of the sync.

This is relatively straight forward and will satisfy nearly all cases, except for that of syncing an empty directory.

[cento79]

I think it'll be a useful workaround. I'd go for a dedicated switch to enable the deletion so that people actually wanting empty folders will not be affected. BTW, great job: I synced almost 700GB in less that 3 days; I used to use rsync before and it kept crashing every few GBs. Excellent tool.

[dav1303]

Sure adding a parameter to let user choose what he wants to do

• Delete empty folders or not delete

[Wowfunhappy]

Deleting empty folders would definitely solve the problem for my use case.

[mcrisc]

I agree with @cento79: a switch to enable deletion would be a good solution. Started using rclone a few days ago. Good tool! Congrats!

[winternet-studio]

I definitely also vote for a switch to enable folder deletion.

[calisro]

I am for folder deletion but I would rather see rclone delete a folder when asked to rather than delete the contents of a directory and then the empty directory. The reason that I have for this:

In my use case, I use Amazon Cloud Drive to store home camera data (videos/jpgs) from my home's automation. I store everything in directories sort of like this:

/cams//2016-02-01
/cams//2016-02-02
/cams//2016-02-03
/cams//2016-02-04
/cams//2016-02-05

etc. So daily folders for each camera. In the GUI, when I send a folder to the trash, it literally sends the folder to the trash and if I were to try to purge that folder or even restore that folder it would be VERY simply to do. i.e. select the folder and restore/delete depending on my wishes.

IF I perform the deletes via RCLONE, It deletes and puts in the trash up with tens of thousands of individual files contained in those folders. Good luck trying to restore them when they are all intermixed with different days/cameras. and there is no method available in amazon cloud drive to 'purge trash' (It needs to be done in batches of 1000 and takes a long time via the GUI).

It would be more efficient to have rclone delete the contents or folder depending on what you specify. Like this:

This could delete individual files
rclone -delete robacd:/cams/playroom/2016-03-29/*
OR
rclone -delete robacd:/cams/playroom/2016-03-29/

This could delete the FOLDER itself (no trailing slash and/or no star)
rclone -delete robacd:/cams/playroom/2016-03-29

Thanks for the great tool.

[Wowfunhappy]

Perhaps this should be split into two separate tickets? I think that properly managing folders is clearly the ideal long-term goal, but it would be great if rclone could simply delete empty folders in the interim.

[kardapoltsev]

+1 for deleting empty directories.

[rtg20]

I too would like empty directories to be deleted. In the meantime, does anyone have a good script for accomplishing the deletion under Windows? Perhaps a Powershell that I can run immediately after rclone?

[rtg20]

...ok I found one:

gci -R . | where { $.PSISContainer -and @( $ | gci ).Count -eq 0 } | ri

reference: http://stackoverflow.com/questions/7831286/how-to-delete-empty-folders-using-windows-command-prompt

[ghost]

+1 for not leaving empty directories behind.... it's indeed very troublesome over long period of sync

[mehargags]

sorry to nudge this thread, but I needed an urgent help to sync all my directory strcuture to Google Drive.
Have also opened a new thread Sync subdirectories recursively excluding some

I have a deep directory structure with 1000s of images inside 100s of subdirectories to be sync'd recursively to my Google Drive, but leaving out ./cache and ./thumbnails directory.

Issue #1 rClone does not copy subdirectories

rclone sync /synctest/images GDrive:/images
this ... only sync files in the dir specified
Why is it not sync'ng and creating the directory structure ? What arguments need to be passed to sync all subdir recursive?

issue #2 How to exclude certain directories

is this right ?
rclone sync /synctest/images GDrive:/images --exclude "/thumbnails/**"

I read you can make and load and exclusion file... but if I only need to exclude one or two directories, what will be the right parameter to pass ?

Thanks

[R3yn4ld]

@mehargags you should open your own issue for subdir, or go to the community forum to have help about exclusion (I cannot help here, I've personally didn't play with this feature).

@ncw , all
Back to the folder issue/feature: I like to work either on the web interface of my cloud provider, or, on a mounted remote.
In the mounted remote, deleting empty folder is as simple as run find -type d -empty -delete ! (on a linux machine, like solution proposed above)
I like the @Wowfunhappy idea of creating a .keep file, like the .placeholder in android roms

BTW, when moving files from a remote to another one, empty folder are not cleaned.

[ncw]

@rollus note that you can delete empty directories with rclone rmdirs now (note extra s).

https://rclone.org/commands/rclone_rmdirs/

[ncw]

Here is a beta which should fix the problem and delete empty directories. It does this after the sync phase provided there were no IO errors.

https://beta.rclone.org/v1.37-082-g265fb8a5/ (uploaded in 15-30 mins)

Let me know if you find any problems with it!

[netinial]

I confirm that there are still same issue for GDrive, (leaving empty folders.)

[winternet-studio]

If anyone is interested in a PHP script that will ensure an exact copy by removing folders on the destination that no longer exist in the source I have posted one here: #1642 (comment)

[ghost]

How's the current progress on syncing empty folders?

[adrianog]

@ncw why is this the case: "provided there were no IO errors"?

Should we not (attempt to) remove directories from destination regardless, if they are not in the source, irrespective of previous IO errors?

[ncw]

@adrianog "IO errors" can mean the source was read wrong. If you read an empty directory incorrectly in the source, you don't want to delete stuff in the destination. The same logic doesn't quite apply for source directories, but I'd like to keep the logic the same, as in "IO errors" mean no deletions.

[archon810]

https://rclone.org/bugs/ refers to this ticket and should probably be updated.

[ncw]

https://rclone.org/bugs/ refers to this ticket and should probably be updated.

Well spotted! I haven't looked at that bugs page for years!

[trademark18]

Is there a workaround for this? In my use case I recursively process files in each directory, then when it's empty, I'd like to delete the "folder" on IBM COS. I've tried various versions of rmdirs or rmdir and nothing at all seems to delete that "folder" on the ICOS console. Any info on how to run a command to delete the "folder" on ICOS would be appreciated.

[ncw]

Is there a workaround for this? In my use case I recursively process files in each directory, then when it's empty, I'd like to delete the "folder" on IBM COS. I've tried various versions of rmdirs or rmdir and nothing at all seems to delete that "folder" on the ICOS console. Any info on how to run a command to delete the "folder" on ICOS would be appreciated.

I suspect that there is a 0 sized directory marker object that rclone is ignoring.

You should find that rclone purge icos:bucket/path/to/folder will remove it. However be careful with purge - it really will delete everything!

[trademark18]

@ncw Thanks for the tip. I tried it and sadly I got the same result. Oddly I'm not having this issue on a "Standard" bucket, but only on "Smart Tier" buckets. Maybe those are different under the hood?

[ncw]

@ncw Thanks for the tip. I tried it and sadly I got the same result. Oddly I'm not having this issue on a "Standard" bucket, but only on "Smart Tier" buckets. Maybe those are different under the hood?

I remembered wrong - sorry! The S3 backend doesn't support Purge. I suspect this directory was created with something other than rclone - is that correct?

[trademark18]

@ncw Yes, the directories were created by an Aspera upload.

[ncw]

@ncw Yes, the directories were created by an Aspera upload.

What will have happened is that "directory markers" have been created. These are empty files with names like dir/. Rclone doesn't deal with these properly at the moment.

If you would like to make a new issue @trademark18 then we can get rclone purge to remove them fairly easily which would be a step forward.