Manage folders #100

Open
ncw opened this Issue Aug 16, 2015 · 60 comments

Projects

None yet
@ncw
Owner
ncw commented Aug 16, 2015

Some of rclones remote fs do understand the concept of folders, eg

  • drive
  • local
  • dropbox

Make an optional interfaces (eg Mkdir, Rmdir) for these FS to manage the creation and deletion of folders. This would enable empty folders, and deletion of empty folders on sync.

@ncw ncw added the enhancement label Aug 16, 2015
@mullenkamp

I just noticed this issue when I sync to drive. I would definitely like to have folders deleted in addition to the contained files.
Thanks a lot for the copy program, it's the most reliable I've used so far for google drive.

@ncw
Owner
ncw commented Aug 26, 2015

@mullenkamp thanks for the vote of confidence! I can see a plan coming together for this.

@austinginder

Been using Rclone regularly for the past few months. Works solid. There are a good number of empty directories left in my Dropbox. This would be awesome if it was built in with the sync command.

@memeplex

This is specially important for synchronizing databases (for example, a git repo) which may:

  1. require an empty directory to be there.
  2. leave behind a large number of empty directories.
@Wowfunhappy

Is there any chance that this will be prioritized in the near-ish future?

The longer that I use rclone to sync the contents of my local hard drive with ACD, the more of a problem this is becoming. As files naturally get reorganized and moved around, more and more empty folders pile up on ACD. An aesthetic problem, perhaps, but it's becoming difficult to actually find the files I want, and there's no way to programmatically find and delete the empty folders short of completely deleting the remote copy and re-uploading from scratch, which would take literal months over my internet connection.

Rclone is still useful for creating remote backups should my local hard drive fail, but it's a lot less useful than it once was.

@ncw
Owner
ncw commented Feb 10, 2016

I think what I might do for this is just delete empty folders on the destination at the end of the sync.

This is relatively straight forward and will satisfy nearly all cases, except for that of syncing an empty directory.

@ncw ncw added this to the Soon milestone Feb 10, 2016
@cento79
cento79 commented Feb 10, 2016

I think it'll be a useful workaround. I'd go for a dedicated switch to enable the deletion so that people actually wanting empty folders will not be affected. BTW, great job: I synced almost 700GB in less that 3 days; I used to use rsync before and it kept crashing every few GBs. Excellent tool.

@dav1303
dav1303 commented Feb 10, 2016

Sure adding a parameter to let user choose what he wants to do

  • Delete empty folders or not delete
@Wowfunhappy

Deleting empty folders would definitely solve the problem for my use case.

@mcrisc
mcrisc commented Feb 16, 2016

I agree with @cento79: a switch to enable deletion would be a good solution. Started using rclone a few days ago. Good tool! Congrats!

@winternet-studio

I definitely also vote for a switch to enable folder deletion.

@calisro
calisro commented Apr 21, 2016 edited

I am for folder deletion but I would rather see rclone delete a folder when asked to rather than delete the contents of a directory and then the empty directory. The reason that I have for this:

In my use case, I use Amazon Cloud Drive to store home camera data (videos/jpgs) from my home's automation. I store everything in directories sort of like this:

/cams//2016-02-01
/cams//2016-02-02
/cams//2016-02-03
/cams//2016-02-04
/cams//2016-02-05

etc. So daily folders for each camera. In the GUI, when I send a folder to the trash, it literally sends the folder to the trash and if I were to try to purge that folder or even restore that folder it would be VERY simply to do. i.e. select the folder and restore/delete depending on my wishes.

IF I perform the deletes via RCLONE, It deletes and puts in the trash up with tens of thousands of individual files contained in those folders. Good luck trying to restore them when they are all intermixed with different days/cameras. and there is no method available in amazon cloud drive to 'purge trash' (It needs to be done in batches of 1000 and takes a long time via the GUI).

It would be more efficient to have rclone delete the contents or folder depending on what you specify. Like this:

This could delete individual files
rclone -delete robacd:/cams/playroom/2016-03-29/*
OR
rclone -delete robacd:/cams/playroom/2016-03-29/

This could delete the FOLDER itself (no trailing slash and/or no star)
rclone -delete robacd:/cams/playroom/2016-03-29

Thanks for the great tool.

@Wowfunhappy

Perhaps this should be split into two separate tickets? I think that properly managing folders is clearly the ideal long-term goal, but it would be great if rclone could simply delete empty folders in the interim.

@kardapoltsev

+1 for deleting empty directories.

@rtg20
rtg20 commented May 16, 2016

I too would like empty directories to be deleted. In the meantime, does anyone have a good script for accomplishing the deletion under Windows? Perhaps a Powershell that I can run immediately after rclone?

@rtg20
rtg20 commented May 16, 2016

...ok I found one:

gci -R . | where { $.PSISContainer -and @( $ | gci ).Count -eq 0 } | ri

reference: http://stackoverflow.com/questions/7831286/how-to-delete-empty-folders-using-windows-command-prompt

@ghost
ghost commented May 30, 2016

+1 for not leaving empty directories behind.... it's indeed very troublesome over long period of sync

@Wowfunhappy
Wowfunhappy commented Jul 19, 2016 edited

Any chance this could get added to the 1.34 milestone? It has been sitting in "soon" for six months now. :)

@mclassdesigns

Empty directories are also not being deleted on OneDrive. Otherwise, amazing tool!

@peixotorms

+1 for this, please.

@ponyesteves

+1 for this feature! Great tool

@memeplex
memeplex commented Sep 8, 2016

Please add a thumbs up instead of replying +1. This way the developer will be aware of the number of votes for the feature and less likely to close the issue to further comments. Also interested subscribers won't receive uninteresting comments.

This was referenced Sep 12, 2016
@lynndixon

I would really like to see the ability of files to be deleted from the destination via sync when they are no longer on the source. I added my thumbs up as well.

@ncw
Owner
ncw commented Oct 28, 2016 edited

I wrote a little script to delete all empty dirs under a remote: http://pub.rclone.org/rclone-delete-empty-dirs.py3

Run it with python3.

Use the --dry-run flag with it first to see what it does.

It isn't very efficient but should get the job done. You may need to run it more than once if you have empty directories within other directories that only have empty directories in.

I'll probably integrate the logic into rclone into a new command, maybe rclone rmdirs or something like that - see #831

@lanrat
lanrat commented Oct 28, 2016 edited

I can't access http://pub.rclone.org/rclone-delete-empty-dirs.py3 it gives a 404.

Update: Was the link meant to be http://pub.rclone.org/rclone-delete-empty-dirs.py ?

@cemsbr
cemsbr commented Oct 29, 2016

The lsd command in the script of @nsw fails for me. Executing it outside the script gives the message level value not supported.

@calisro
calisro commented Oct 31, 2016 edited

I wish I would have saw the python script... but for anyone else wanting a work around. Here is a pure bash one. It uses a rclone mount and then deletes from acd directly via rclone (because of writing bugs on rclone crypt mounts).

https://forum.rclone.org/t/so-is-there-any-way-to-clean-up-those-empty-left-behind-directories/114/9?u=calisro

@ShapeShifter499

New to this issue...

Shouldn't there be a 1:1 sync option? ANY changes on source leads to a mirrored change on remote?

For example, I remove a directory completely on source and next time I run rclone sync it removes that directory on remote. But if a directory is empty on source it gets created as a empty directory on remote.

@calisro
calisro commented Nov 3, 2016

@ShapeShifter499 That is the way the 'sync' option works. The only caveat is there are issues handling directories right now which will get taken care of at some point.

@ShapeShifter499
ShapeShifter499 commented Nov 3, 2016 edited

@calisro Maybe I should file a new issue then? ACD sync doesn't delete files or folders on remote when source changes.

EDIT: Ok I'll make a new issue about it

@ncw
Owner
ncw commented Nov 6, 2016

@lanrat that download works for me - try again!

@cemsbr

The lsd command in the script of @ncw fails for me. Executing it outside the script gives the message level value not supported.

That indicates you are using an old version of rclone - you'll need to upgrade.

@cemsbr
cemsbr commented Nov 8, 2016

I used 1.33. I'm waiting 1.34 to available in Archlinux.

@nrk666
nrk666 commented Nov 19, 2016 edited

This python script seems to have some logic problems, I think relating to the root cause. I had some empty directories nested within otherwise empty directories and it tried to delete the top level directory (with nothing inside it but an empty directory) before deleting the empty directory inside itself, then error'd out with "Directory Not Empty" which was true. Wow, even I think that sentence is impossible to follow.

acd:FOLDER
- SomeFolder (child of FOLDER)
- Some Other Folder (empty child folder of "SomeFolder")

The python script tried to delete "SomeFolder" before deleting "Some Other Folder" and caused an error because "SomeFolder" was not empty (which was true).

I ran python3 script.py acd:FOLDER

@cemsbr
cemsbr commented Nov 21, 2016

With version 1.34-DEV, it fails the same way as my previous post.

@ncw ncw added a commit that referenced this issue Nov 27, 2016
@ncw Add directory parameter to Rmdir and Mkdir #100 #831
This will enable rclone to manage directories properly in the future.
aaa1370
@ncw
Owner
ncw commented Nov 27, 2016

I've made a new rclone rmdirs command which will delete any empty directories (in a nested way) from a remote.

You can try it in this beta: http://beta.rclone.org/v1.34-25-gf3365dd/ (uploaded in 15-30 mins)

Use it like this

rclone rmdirs --dry-run remote:

When happy then do

rclone rmdirs remote:
@ncw
Owner
ncw commented Nov 27, 2016

Oops messed up the build: try http://beta.rclone.org/v1.34-30-g943a093/ (uploaded in 15-30 mins)

@ncw
Owner
ncw commented Nov 28, 2016
@cemsbr
cemsbr commented Nov 30, 2016

Worked for me. Took 27 minutes in the first run for dropbox. After running for the first time, total objects and size are still the same, but a dry run doesn't show anything left.

@calisro
calisro commented Dec 5, 2016 edited

When using rmdirs, I am seeing errors that hte directories are not found.

2016/12/05 09:07:40 .unionfs/Pictures/Downloaded Albums/lindasnotepad: Failed to rmdir: directory not found
2016/12/05 09:07:40 Attempt 1/3 failed with 1 errors and: directory not found
2016/12/05 09:08:27

I don't have anything else running deleting the directories. This is a crypt ACD.

rclone lsd "robacd-crypt:.unionfs/Pictures/Downloaded Albums"
-1 2016-11-02 18:10:06 -1 lindasnotepad
2016/12/05 09:18:33
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 1.2s

but then running it manually directly at that folder works?

rclone rmdirs -v "robacd-crypt:.unionfs/Pictures/Downloaded Albums/lindasnotepad"
2016/12/05 09:21:42 rclone: Version "v1.34-36-g539853dβ" starting with parameters ["rclone" "rmdirs" "-v" "robacd-crypt:.unionfs/Pictures/Downloaded Albums/lindasnotepad"]
2016/12/05 09:21:43 Encrypted amazon drive root 'cloudm/t55mfpdhjnnpr9gcctbmus931o/prtmcteuum3panir771giakboo/d9saeilc969hoki1sdvar99b86i81mgfjo3qk0cbr4sfv7cqbrtg/pt9ds1q722h53k7vvk8f48r9nc': Modify window not supported
2016/12/05 09:21:43 amazon drive root 'cloudm/t55mfpdhjnnpr9gcctbmus931o/prtmcteuum3panir771giakboo/d9saeilc969hoki1sdvar99b86i81mgfjo3qk0cbr4sfv7cqbrtg/pt9ds1q722h53k7vvk8f48r9nc': Reading ""
2016/12/05 09:21:43 amazon drive root 'cloudm/t55mfpdhjnnpr9gcctbmus931o/prtmcteuum3panir771giakboo/d9saeilc969hoki1sdvar99b86i81mgfjo3qk0cbr4sfv7cqbrtg/pt9ds1q722h53k7vvk8f48r9nc': Finished reading ""
2016/12/05 09:21:44
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 1.9s
2016/12/05 09:21:44 Go routines at exit 7

rclone -V
rclone v1.34-36-g539853dβ

@ncw
Owner
ncw commented Dec 6, 2016

@calisro I can confirm that. Should be easy to fix - will post a beta here!

@ncw
Owner
ncw commented Dec 6, 2016

@calisro I've fixed that in http://beta.rclone.org/v1.34-51-gcb9f1ee/ (uploaded in 15-30 mins)

@Wowfunhappy
Wowfunhappy commented Dec 6, 2016 edited

Is there any reason to not do rmdirs automatically at the end of a sync?

Correct me if I'm wrong, but empty directories can't be created with rclone to begin with, so the only way an empty directory could appear is if its contents were deleted, in which case we probably want the empty directory removed as well.

@calisro
calisro commented Dec 6, 2016

@ncw Seems better now. Thanks.

@Wowfunhappy
Wowfunhappy commented Dec 7, 2016 edited

Whereas the older "third time lucky" beta worked beautifully for me, the newer one seems to not actually be deleting my empty folders.

(There wasn't any output from either version. I'll rerun with verbose and post results if I can, but I probably won't have time in the next few days.)

@tuanjr
tuanjr commented Dec 8, 2016

(Sorry for my english bad)
i really love this soft, only problem that it can't delete empty folder after sync
i tried rclone rmdir but failed
but i have many many empty folder like that: /folder1/folder2(empty)/folder3(empty)

@Fmstrat
Fmstrat commented Dec 11, 2016

Does rmdirs remove ALL empty directories? Or just empty directories that do not exist in the source? The latter is what I think is really required as there are many, many use cases where empty directories are required.

@Wowfunhappy

@fmstrat rmdirs removes all empty directories. Rclone doesn't really support empty directories atm—empty directories on source won't get synced to begin with. NCW has stated that he intends to support empty directories at some point, but rmdirs was added as an easy stop-gap.

@karan
karan commented Dec 27, 2016

rmdirs fails for me now:

2016/12/26 18:23:02 Failed to rmdirs: directory not empty: []*drive.ChildReference{(*drive.ChildReference)(0xc420664000), (*drive.ChildReference)(0xc420664080), (*drive.ChildReference)(0xc420664100), (*drive.ChildReference)(0xc420664480)}

It used to work until last week with v1.34-36-g539853dβ. Then it stopped working so I updated the binary. I'm on the latest beta version:

$ rclone --version
rclone v1.34-75-gcbfec0dβ

The remote is a folder in Google Drive, and I have confirmed that it is empty.

@dendenis82

if for the first: rmdirs remote:
then sync local: remote:

will last line add again all empty folders(which is needed by some software) to remote?

@eharris
eharris commented Dec 30, 2016 edited

I agree that preserving empty folders is necessary in some cases, and the interim solution of optionally deleting all empty directories is a reasonable temporary workaround.

One of the issues mentioned as an explanation of why directories aren't already handled properly was that some backends don't have any concept of them, and this resulted in a common interface that also disregarded them.

Even for backends that don't support the concept of folders, I don't think there should be any reason why they can't be tracked and emulated properly by the rclone code for those backends. Just a couple of possibilities that come immediately to mind:

  1. Creating a placeholder file (like somedir/._empty_) that is otherwise ignored by rclone except to mark that a directory exists. Potential collisions can be avoided by making the name of the placeholder file configurable per storage destination. For destinations that support some form of metadata tagging of files, they could also be explicitly marked with that somehow to remove any possible ambiguity.

  2. Creating a special file under the parent directory of any empty directories that contains a list of all the empty directories under it. Again, this special file would be ignored by other normal operations. This one has the possible benefit that the file may only be needed in the parent of the lowest level of a chain of otherwise empty directories, as the file itself would mark the tree above that as existing. However that might complicate the implementation enough not to be worth it.

This same kind of feature could be used to solve the other bug currently listed on http://rclone.org/bugs/ as it could also store the directory mtime in such a special file. While this would require a special file for every directory (or maybe just every parent directory), that seems like it would be a worthwhile tradeoff.

In my opinion interfaces for 'rmdir'/'mkdir' should not be optional on any backend. Directories need to be first-class citizens that rclone fully supports, even if they are only emulated as far as the actual storage backend is concerned.

@Wowfunhappy
Wowfunhappy commented Jan 4, 2017 edited

Creating a placeholder file (like somedir/.empty) that is otherwise ignored by rclone except to mark that a directory exists.

In the event that such invisible placeholder files will not interfere with your use case, you can do this yourself with Bash. Before running rclone, cd into your directory and type:

find . -type d -empty -exec touch {}/.keep \;

Source

NCW has said in the past he'd prefer not to create/store rclone-specific files on remotes, and I generally agree with this approach. It's easy enough to tell rclone to ignore a given file, but what about other programs? What about users going through the remote's native web interface?

@gordan-bobic

I don't think there is any disadvantage to keeping rclone specifics on remotes if the directories are encrypted because there is nothing else directly compatible with rclone's encryption (e.g. ecryptfs and encfs).

@Wowfunhappy

If you're using Crypt, I agree, but that's only a subset of use cases.

@calisro
calisro commented Jan 7, 2017

I wouldn't agree. it would screw up file counts. and "crypt" isn't propietery to rclone. Files shouldn't store metadata and shouldn't be directory placeholders. just my two cents. :)

@gordan-bobic

It is proprietary in the sense that there is no other software that can decrypt the files and file names. There is nothing stopping you from using the same library to implement a codec of your own that can read them, but it is proprietary in the sense that no other similar software can decode the data (e.g. ecryptfs or encfs).

@calisro
calisro commented Jan 7, 2017

Look up the word proprietary. It is ownership. crypt isn't proprietery. It is opensource. And its used in many tools. Anyone can write code to decrypt the data. This is off-topic though.

@gordan-bobic

OK, then:
s/proprietary/incompatible with all other currently available stackable file system encryption methods/

@kazeuraki

#100 (comment)
Same with this guy.
Failed to rmdirs: directory not empty: (*drive.ChildReference)(0xc420664480)}
Empty folder A in folder B (dont have any file, just empty folder)

@balazer
balazer commented Jan 16, 2017 edited

@ncw, using rmdirs in rclone v1.35-33-g47ebd07 with Google Drive, one thing I noticed is that the command line help says

rmdirs Remove any empty directoryies under the path.

But the command will remove the path itself also, not just the directories under the path. Also 'directories' is misspelled.

@balazer
balazer commented Jan 16, 2017 edited

@ncw, using rmdirs in rclone v1.35-33-g47ebd07 with Google Drive I'm experiencing the same Failed to rmdirs: directory not empty errors as others. The directory appears to be empty.

The problem is that the directory contained an item that had been moved to the trash. Once the item is deleted from the trash, rmdirs works without errors.

For the same reason, rmdirs fails when the --drive-use-trash option is specified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment