Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manage folders #100

Closed
ncw opened this issue Aug 16, 2015 · 73 comments

Comments

Projects
None yet
@ncw
Copy link
Owner

commented Aug 16, 2015

Some of rclones remote fs do understand the concept of folders, eg

  • drive
  • local
  • dropbox

Make an optional interfaces (eg Mkdir, Rmdir) for these FS to manage the creation and deletion of folders. This would enable empty folders, and deletion of empty folders on sync.

@mullenkamp

This comment has been minimized.

Copy link

commented Aug 26, 2015

I just noticed this issue when I sync to drive. I would definitely like to have folders deleted in addition to the contained files.
Thanks a lot for the copy program, it's the most reliable I've used so far for google drive.

@ncw

This comment has been minimized.

Copy link
Owner Author

commented Aug 26, 2015

@mullenkamp thanks for the vote of confidence! I can see a plan coming together for this.

@austinginder

This comment has been minimized.

Copy link

commented Sep 11, 2015

Been using Rclone regularly for the past few months. Works solid. There are a good number of empty directories left in my Dropbox. This would be awesome if it was built in with the sync command.

@memeplex

This comment has been minimized.

Copy link

commented Dec 28, 2015

This is specially important for synchronizing databases (for example, a git repo) which may:

  1. require an empty directory to be there.
  2. leave behind a large number of empty directories.
@Wowfunhappy

This comment has been minimized.

Copy link

commented Feb 3, 2016

Is there any chance that this will be prioritized in the near-ish future?

The longer that I use rclone to sync the contents of my local hard drive with ACD, the more of a problem this is becoming. As files naturally get reorganized and moved around, more and more empty folders pile up on ACD. An aesthetic problem, perhaps, but it's becoming difficult to actually find the files I want, and there's no way to programmatically find and delete the empty folders short of completely deleting the remote copy and re-uploading from scratch, which would take literal months over my internet connection.

Rclone is still useful for creating remote backups should my local hard drive fail, but it's a lot less useful than it once was.

@ncw

This comment has been minimized.

Copy link
Owner Author

commented Feb 10, 2016

I think what I might do for this is just delete empty folders on the destination at the end of the sync.

This is relatively straight forward and will satisfy nearly all cases, except for that of syncing an empty directory.

@ncw ncw added this to the Soon milestone Feb 10, 2016

@cento79

This comment has been minimized.

Copy link

commented Feb 10, 2016

I think it'll be a useful workaround. I'd go for a dedicated switch to enable the deletion so that people actually wanting empty folders will not be affected. BTW, great job: I synced almost 700GB in less that 3 days; I used to use rsync before and it kept crashing every few GBs. Excellent tool.

@dav1303

This comment has been minimized.

Copy link

commented Feb 10, 2016

Sure adding a parameter to let user choose what he wants to do

  • Delete empty folders or not delete
@Wowfunhappy

This comment has been minimized.

Copy link

commented Feb 10, 2016

Deleting empty folders would definitely solve the problem for my use case.

@mcrisc

This comment has been minimized.

Copy link

commented Feb 16, 2016

I agree with @cento79: a switch to enable deletion would be a good solution. Started using rclone a few days ago. Good tool! Congrats!

@winternet-studio

This comment has been minimized.

Copy link

commented Apr 7, 2016

I definitely also vote for a switch to enable folder deletion.

@calisro

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2016

I am for folder deletion but I would rather see rclone delete a folder when asked to rather than delete the contents of a directory and then the empty directory. The reason that I have for this:

In my use case, I use Amazon Cloud Drive to store home camera data (videos/jpgs) from my home's automation. I store everything in directories sort of like this:

/cams//2016-02-01
/cams//2016-02-02
/cams//2016-02-03
/cams//2016-02-04
/cams//2016-02-05

etc. So daily folders for each camera. In the GUI, when I send a folder to the trash, it literally sends the folder to the trash and if I were to try to purge that folder or even restore that folder it would be VERY simply to do. i.e. select the folder and restore/delete depending on my wishes.

IF I perform the deletes via RCLONE, It deletes and puts in the trash up with tens of thousands of individual files contained in those folders. Good luck trying to restore them when they are all intermixed with different days/cameras. and there is no method available in amazon cloud drive to 'purge trash' (It needs to be done in batches of 1000 and takes a long time via the GUI).

It would be more efficient to have rclone delete the contents or folder depending on what you specify. Like this:

This could delete individual files
rclone -delete robacd:/cams/playroom/2016-03-29/*
OR
rclone -delete robacd:/cams/playroom/2016-03-29/

This could delete the FOLDER itself (no trailing slash and/or no star)
rclone -delete robacd:/cams/playroom/2016-03-29

Thanks for the great tool.

@Wowfunhappy

This comment has been minimized.

Copy link

commented Apr 23, 2016

Perhaps this should be split into two separate tickets? I think that properly managing folders is clearly the ideal long-term goal, but it would be great if rclone could simply delete empty folders in the interim.

@kardapoltsev

This comment has been minimized.

Copy link

commented May 11, 2016

+1 for deleting empty directories.

@rtg20

This comment has been minimized.

Copy link

commented May 16, 2016

I too would like empty directories to be deleted. In the meantime, does anyone have a good script for accomplishing the deletion under Windows? Perhaps a Powershell that I can run immediately after rclone?

@rtg20

This comment has been minimized.

Copy link

commented May 16, 2016

...ok I found one:

gci -R . | where { $.PSISContainer -and @( $ | gci ).Count -eq 0 } | ri

reference: http://stackoverflow.com/questions/7831286/how-to-delete-empty-folders-using-windows-command-prompt

@ghost

This comment has been minimized.

Copy link

commented May 30, 2016

+1 for not leaving empty directories behind.... it's indeed very troublesome over long period of sync

@kazeuraki

This comment has been minimized.

Copy link

commented Jan 16, 2017

#100 (comment)
Same with this guy.
Failed to rmdirs: directory not empty: (*drive.ChildReference)(0xc420664480)}
Empty folder A in folder B (dont have any file, just empty folder)

@balazer

This comment has been minimized.

Copy link

commented Jan 16, 2017

@ncw, using rmdirs in rclone v1.35-33-g47ebd07 with Google Drive, one thing I noticed is that the command line help says

rmdirs Remove any empty directoryies under the path.

But the command will remove the path itself also, not just the directories under the path. Also 'directories' is misspelled.

@balazer

This comment has been minimized.

Copy link

commented Jan 16, 2017

@ncw, using rmdirs in rclone v1.35-33-g47ebd07 with Google Drive I'm experiencing the same Failed to rmdirs: directory not empty errors as others. The directory appears to be empty.

The problem is that the directory contained an item that had been moved to the trash. Once the item is deleted from the trash, rmdirs works without errors.

For the same reason, rmdirs fails when the --drive-use-trash option is specified.

@Stele77

This comment has been minimized.

Copy link

commented Mar 11, 2017

Hey, i am just a noob, so i dont know about the technical backgrounds.. But Rclone not syncing empty folders is sadly a deal breaker for my usecase: Backup of my NAS-Shares to ACD.
This messes up my folder structure, and is even more sad IMHO, because RClone works so flawless otherwise.
I understand from this or other issue tickets, that many others would want to auto delete empty dirs, but others like me would need them to be synced.

IMHO it would be a viable solution to offer a switch, to let the users decide, how they want to handle this.

I think i am not alone with my workflow need for exact backup, and it otherwise breaks the purpose of this ingenious tool. And it es even more annoying, because new users and noobs like me will struggle, and many wont find the answer here. So everyone not so tech savy and geeky has a problem..

Just my 2 cents..

PS: Just saw another user opening a new ticket on the same problem:
#1228

@Fmstrat

This comment has been minimized.

Copy link

commented Mar 12, 2017

@Stele77 An easy way for this not to be a deal-breaker:

find /tmp/A -type d -empty -printf "mkdir -p \"%p\";chown %u:%g \"%p\";chmod %m \"%p\"\n" > /tmp/A/rclone-restore-empty-directories.sh;

Run that before you backup to ACD. It's a one-liner that makes a script to restore your empty directories with ownership and permission. It will backup to ACD, so if you ever have to restore, you just run it after the restore completes.

Of course, replace /tmp/A with your backup folder.

As a note, I also do a find command for every file, so that I can do one post restore and compare the results.

EDIT
Figured I'd give that command, too, if you want it:

find /tmp/A -printf "\"%p\"|%s|%u|%g|%m|%a|%c|%t|%y\n" > /tmp/A/storage-filelist.txt
@ncw

This comment has been minimized.

Copy link
Owner Author

commented Mar 13, 2017

I've put this issue on the 1.37 milestone since all the pieces are in place now :-)

@vesper8

This comment has been minimized.

Copy link

commented Mar 29, 2017

couldn't we have some enhanced functionality in combination with --delete-excluded so that an rmdir on excluded directories is attempted in each folder?

rmdirs could be removing empty directories that i don't want to remove, i only want to remove the ones that are listed in my excluded-filter

@mehargags

This comment has been minimized.

Copy link

commented Apr 2, 2017

sorry to nudge this thread, but I needed an urgent help to sync all my directory strcuture to Google Drive.
Have also opened a new thread Sync subdirectories recursively excluding some

I have a deep directory structure with 1000s of images inside 100s of subdirectories to be sync'd recursively to my Google Drive, but leaving out ./cache and ./thumbnails directory.

Issue #1 rClone does not copy subdirectories

rclone sync /synctest/images GDrive:/images
this ... only sync files in the dir specified
Why is it not sync'ng and creating the directory structure ? What arguments need to be passed to sync all subdir recursive?

issue #2 How to exclude certain directories

is this right ?
rclone sync /synctest/images GDrive:/images --exclude "/thumbnails/**"

I read you can make and load and exclusion file... but if I only need to exclude one or two directories, what will be the right parameter to pass ?

Thanks

@R3yn4ld

This comment has been minimized.

Copy link

commented Apr 8, 2017

@mehargags you should open your own issue for subdir, or go to the community forum to have help about exclusion (I cannot help here, I've personally didn't play with this feature).

@ncw , all
Back to the folder issue/feature: I like to work either on the web interface of my cloud provider, or, on a mounted remote.
In the mounted remote, deleting empty folder is as simple as run find -type d -empty -delete ! (on a linux machine, like solution proposed above)
I like the @Wowfunhappy idea of creating a .keep file, like the .placeholder in android roms

BTW, when moving files from a remote to another one, empty folder are not cleaned.

@ncw

This comment has been minimized.

Copy link
Owner Author

commented Apr 10, 2017

@RolluS note that you can delete empty directories with rclone rmdirs now (note extra s).

https://rclone.org/commands/rclone_rmdirs/

@ncw

This comment has been minimized.

Copy link
Owner Author

commented Aug 9, 2017

Here is a beta which should fix the problem and delete empty directories. It does this after the sync phase provided there were no IO errors.

https://beta.rclone.org/v1.37-082-g265fb8a5/ (uploaded in 15-30 mins)

Let me know if you find any problems with it!

@ncw ncw closed this in 265fb8a Aug 9, 2017

@netinial

This comment has been minimized.

Copy link

commented Sep 18, 2017

I confirm that there are still same issue for GDrive, (leaving empty folders.)

@winternet-studio

This comment has been minimized.

Copy link

commented Sep 20, 2017

If anyone is interested in a PHP script that will ensure an exact copy by removing folders on the destination that no longer exist in the source I have posted one here: #1642 (comment)

@ghost

This comment has been minimized.

Copy link

commented Nov 19, 2017

How's the current progress on syncing empty folders?

@adrianog

This comment has been minimized.

Copy link

commented Dec 11, 2017

@ncw why is this the case: "provided there were no IO errors"?

Should we not (attempt to) remove directories from destination regardless, if they are not in the source, irrespective of previous IO errors?

@ncw

This comment has been minimized.

Copy link
Owner Author

commented Dec 12, 2017

@adrianog "IO errors" can mean the source was read wrong. If you read an empty directory incorrectly in the source, you don't want to delete stuff in the destination. The same logic doesn't quite apply for source directories, but I'd like to keep the logic the same, as in "IO errors" mean no deletions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.