rclone copy creating empty directories it didn't create before #7689

YukiUnHappy · 2024-03-25T06:54:03Z

What is the problem you are having with rclone?

Rclone will now sync directory modification times if the backend supports it
That's a cool new feature, but I think it shouldn't be enabled by default, or it should ignore folders by default where no files need to be transferred.
In the previous version, I could simply copy to the remote within the folder where I needed to upload files, but now I have to remove the other folders in that folder that have no files first, otherwise these empty folders will also be created on the remote due to the sync modification time.
Conversely, if I only need to copy specific files from the remote to the local (using --include), but other irrelevant folders under that path on the remote will also be created on my local machine with the original structure. This did not exist in previous versions and is a bit annoying.

What is your rclone version (output from `rclone version`)

rclone v1.66.0

os/version: debian 10.0 (64 bit)
os/kernel: 5.14.9 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.22.1
go/linking: static
go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

The information has already been mentioned above.

Which cloud storage system are you using? (e.g. Google Drive)

This is not related to a specific backend.

The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

rclone copy -vv Source Remote --include E.txt

A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

root@localhost:~/TEST# tree
.
└── Source
    ├── A
    │   └── D
    ├── B
    ├── C
    └── E.txt

5 directories, 1 file
root@localhost:~/TEST# rclone copy -vv Source Remote --include E.txt
2024/03/24 23:51:52 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "-vv" "Source" "Remote" "--include" "E.txt"]
2024/03/24 23:51:52 DEBUG : Creating backend with remote "Source"
2024/03/24 23:51:52 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/03/24 23:51:52 DEBUG : fs cache: renaming cache item "Source" to be canonical "/root/TEST/Source"
2024/03/24 23:51:52 DEBUG : Creating backend with remote "Remote"
2024/03/24 23:51:52 DEBUG : fs cache: renaming cache item "Remote" to be canonical "/root/TEST/Remote"
2024/03/24 23:51:52 DEBUG : A: Making directory with metadata
2024/03/24 23:51:52 INFO  : A: Made directory with metadata (mtime=2024-03-24T23:50:07.378243503-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "A", newDst=A
2024/03/24 23:51:52 DEBUG : B: Making directory with metadata
2024/03/24 23:51:52 INFO  : B: Made directory with metadata (mtime=2024-03-24T23:50:01.04917523-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "B", newDst=B
2024/03/24 23:51:52 DEBUG : C: Making directory with metadata
2024/03/24 23:51:52 INFO  : C: Made directory with metadata (mtime=2024-03-24T23:50:03.246198928-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "C", newDst=C
2024/03/24 23:51:52 DEBUG : E.txt: Need to transfer - File not found at Destination
2024/03/24 23:51:52 DEBUG : A/D: Making directory with metadata
2024/03/24 23:51:52 INFO  : A/D: Made directory with metadata (mtime=2024-03-24T23:50:07.378243503-07:00)
2024/03/24 23:51:52 DEBUG : Added delayed dir = "A/D", newDst=A/D
2024/03/24 23:51:52 DEBUG : Local file system at /root/TEST/Remote: Waiting for checks to finish
2024/03/24 23:51:52 DEBUG : E.txt: md5 = d8e8fca2dc0f896fd7cb4cb0031ba249 OK
2024/03/24 23:51:52 DEBUG : E.txt.cihekil4.partial: renamed to: E.txt
2024/03/24 23:51:52 INFO  : E.txt: Copied (new)
2024/03/24 23:51:52 DEBUG : Local file system at /root/TEST/Remote: Waiting for transfers to finish
2024/03/24 23:51:52 INFO  : A/D: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  : C: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  : A: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  : B: Set directory modification time (using SetModTime)
2024/03/24 23:51:52 INFO  :
Transferred:              5 B / 5 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2024/03/24 23:51:52 DEBUG : 7 go routines active
root@localhost:~/TEST# tree
.
├── Remote
│   ├── A
│   │   └── D
│   ├── B
│   ├── C
│   └── E.txt
└── Source
    ├── A
    │   └── D
    ├── B
    ├── C
    └── E.txt

10 directories, 2 files

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

The text was updated successfully, but these errors were encountered:

cau118 · 2024-03-26T13:09:35Z

I'm on macOS 14.3.1 running rclone v1.66.0 and I am experiencing the same issue. Folders are now being copied/synchronized, and at the destination, they are empty. They are empty because the folders do not contain files match the extension I have specified in the include flag on the rclone command line.

rclone v1.66.0

os/version: darwin 14.3.1 (64 bit)
os/kernel: 23.3.0 (arm64)
os/type: darwin
os/arch: arm64 (ARMv8 compatible)
go/version: go1.22.1
go/linking: dynamic
go/tags: none

OS used: macOS 14.3.1.

Remote is SSH/SFTP.

Command line used:
rclone copy . pi:testfolder -P --include="*.txt" --dry-run -vv

Results of the command with dry-run:

2024/03/26 09:05:39 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "." "pi:testfolder" "-P" "--include=*.txt" "--dry-run" "-vv"]
2024/03/26 09:05:39 DEBUG : Creating backend with remote "."
2024/03/26 09:05:39 DEBUG : Using config file from "/Users/cmlkh/.config/rclone/rclone.conf"
2024/03/26 09:05:39 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/cmlkh/Downloads/testfolder"
2024/03/26 09:05:39 DEBUG : Creating backend with remote "pi:testfolder"
2024/03/26 09:05:39 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: New connection 172.16.0.227:51097->172.16.0.12:22 to "SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3"
2024/03/26 09:05:40 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: Shell type "unix" from config
2024/03/26 09:05:40 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: Relative path resolved to "/home/cmlkh/testfolder"
2024/03/26 09:05:40 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: Using root directory "/home/cmlkh/testfolder"
2024/03/26 09:05:40 DEBUG : 13.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 DEBUG : 19.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 DEBUG : 9.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 NOTICE: examplefolder01: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 NOTICE: 9.txt: Skipped copy as --dry-run is set (size 1.022Ki)
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder01", newDst=<nil>
2024/03/26 09:05:40 NOTICE: examplefolder02: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder02", newDst=<nil>
2024/03/26 09:05:40 NOTICE: examplefolder03: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder03", newDst=<nil>
2024/03/26 09:05:40 DEBUG : scratch.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 NOTICE: scratch.txt: Skipped copy as --dry-run is set (size 8.616Ki)
2024/03/26 09:05:40 NOTICE: 19.txt: Skipped copy as --dry-run is set (size 1.653Ki)
2024/03/26 09:05:40 NOTICE: 13.txt: Skipped copy as --dry-run is set (size 1.737Ki)
2024/03/26 09:05:40 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: Waiting for checks to finish
2024/03/26 09:05:40 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: Waiting for transfers to finish
2024/03/26 09:05:40 NOTICE: examplefolder03: Skipped set directory modification time as --dry-run is set
2024/03/26 09:05:40 NOTICE: examplefolder01: Skipped set directory modification time as --dry-run is set
2024/03/26 09:05:40 NOTICE: examplefolder02: Skipped set directory modification time as --dry-run is set
Transferred:   	   13.029 KiB / 13.029 KiB, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.2s
2024/03/26 09:05:40 NOTICE:
Transferred:   	   13.029 KiB / 13.029 KiB, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.2s

2024/03/26 09:05:40 DEBUG : 15 go routines active
2024/03/26 09:05:40 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: Closing 1 unused connections
2024/03/26 09:01:30 DEBUG : 15 go routines active
2024/03/26 09:01:30 DEBUG : sftp://cmlkh@172.16.0.12:22/testfolder: Closing 1 unused connections

sabitm · 2024-03-28T23:24:04Z

I'm not 100% sure if this is related, but bisync also has a weird behavior on 1.66.0. It keep producing a file conflict (which I have to resync to resolve it) between local and remote storage. I just assume that is because of experimental nature of bisync, hence I'm not opening an issue and will happily use the previous version for time being.

nielash · 2024-03-29T00:21:35Z

Hi @sabitm, bisync maintainer here. bisync received a very large update in v1.66, and handling of sync conflicts is indeed one of the things that changed. So it may be that the new behavior you're seeing is intentional, but if there's a bug then I'd definitely like to know about it. Please feel free to open an issue or post in the forum with more details!

As for the original issue -- FYI there is an ongoing discussion about it in the forum:
https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/9

sabitm · 2024-04-02T04:12:12Z

Hi @nielash, thanks for maintaining bisync!

I'm not entirely sure how to debug though. It happens randomly enough. I'll try to find a way to reproduce it and will opening an issue if I found something interesting.

ncw · 2024-04-02T10:49:53Z

There are several threads about this on the forum

For rclone v1.66.1 I am going to make sure it obeys the --create-empty-src-dirs flag.

What the defaults for this for rclone sync, rclone copy and rclone move I have yet to decide. Currently they are all false. There is an argument for making this true for rclone sync but that might be a step too far for a point release.

YukiUnHappy · 2024-04-03T09:36:46Z

Thank you @ncw for confirming the bug. Additionally, I have a small issue that I won't open a separate issue for.
The auto completion improvement in 1.66 has broken the ability to complete paths with spaces in bash.

root@localhost:~/TEST# tree
.
├── ABC
└── D EF

2 directories, 0 files
root@localhost:~/TEST# rclone copy D
D

If we use the old version of rclone to create a completion script and apply it, we can also complete paths normally in 1.66.

root@localhost:~/TEST# rclone-v1.65.2-linux-amd64/rclone completion bash
root@localhost:~/TEST# . /etc/bash_completion
root@localhost:~/TEST# rclone version
rclone v1.66.0
- os/version: debian 10.0 (64 bit)
- os/kernel: 5.14.9 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none
root@localhost:~/TEST# rclone copy D\ EF/

ncw · 2024-04-03T10:14:17Z

@YukiUnHappy

Additionally, I have a small issue that I won't open a separate issue for. The auto completion improvement in 1.66 has broken the ability to complete paths with spaces in bash.

I can confirm that. Can you open a new issue about this please? So we don't mix the two up - thank you. (BTW I think this is a bug in cobra and here is the fix spf13/cobra#2126 - Note also that if you type a " first eg "D<TAB> you'll get the correct completions.)

…false In v1.66.0 the changes to enable metadata preservation on directories introduced a regression, namely that empty directories were created despite the state of the --create-empty-src-dirs flag. This patch fixes the problem by letting the normal rclone directory creation create the directories and fixing up their timestamps and metadata afterwards if --create-empty-src-dirs=false. Fixes #7689 See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/ See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/

ncw · 2024-04-05T15:26:04Z

Here is a first attempt at a fix - this obeys --create-empty-src-dirs in the same way that 1.65.2 does

v1.67.0-beta.7854.8d02ad3ba.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

@nielash I've been unable to get the bisync tests to pass on Windows or macOS. I have no idea why they are failing. Running with -golden sorted out the linux tests but not Windows or macOS. Can you help? Also any code review gratefully received :-)

nielash · 2024-04-05T17:27:49Z

@nielash I've been unable to get the bisync tests to pass on Windows or macOS. I have no idea why they are failing.

It looks to me like the issue is that Size() for the dir is returning an unexpected value. It is expecting 4Ki but getting 96 on mac_amd64:

2024/04/05 15:31:30 | MISCOMPARE  -Golden vs +Results for  test.log
2024/04/05 15:31:30 | @@ -139 +139 @@
2024/04/05 15:31:30 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:31:30 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 96)
2024/04/05 15:31:30 | @@ -141 +141 @@
2024/04/05 15:31:30 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:31:30 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 96)

and 0 on windows:

2024/04/05 15:30:53 | MISCOMPARE  -Golden vs +Results for  test.log
2024/04/05 15:30:53 | @@ -139 +139 @@
2024/04/05 15:30:53 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:30:53 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 0)
2024/04/05 15:30:53 | @@ -141 +141 @@
2024/04/05 15:30:53 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:30:53 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 0)

This comment seems potentially relevant:

rclone/backend/local/local.go

Lines 1494 to 1497 in 05e5712

    
           // The value in info.Size() is not always correct 
        
           // - Windows links read as 0 size 
        
           // - Some virtual filesystems (such ash LucidLink) links read as 0 size 
        
           // - Android - some versions the links are larger than readlink suggests

I can take a closer look later!

…false In v1.66.0 the changes to enable metadata preservation on directories introduced a regression, namely that empty directories were created despite the state of the --create-empty-src-dirs flag. This patch fixes the problem by letting the normal rclone directory creation create the directories and fixing up their timestamps and metadata afterwards if --create-empty-src-dirs=false. Fixes #7689 See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/ See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/

nielash · 2024-04-06T15:39:44Z

@ncw I'm doing the closer look at fix-7689-empty-dirs now and I think there's an issue with the --create-empty-src-dirs=false case. Here's a test that demonstrates it: nielash@16d856a

The same test would have passed previously.

nielash · 2024-04-06T17:01:04Z

@ncw added a few other inline comments:
dd463b2
13c72a9

and a test that (I think) catches another regression: nielash@cd0891d

…false In v1.66.0 the changes to enable metadata preservation on directories introduced a regression, namely that empty directories were created despite the state of the --create-empty-src-dirs flag. This patch fixes the problem by letting the normal rclone directory creation create the directories and fixing up their timestamps and metadata afterwards if --create-empty-src-dirs=false. Fixes #7689 See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/ See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/

ncw · 2024-04-16T20:49:19Z

I finally got round to fixing up this code. Can you take a quick look @nielash ?

I think it is doing the right thing now. I left the fixes unsquashed so you can see them and I put your tests in (though I took one line out as I don't think it was correct any more).

v1.67.0-beta.7901.7042810e8.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

Thank you :-)

nielash · 2024-04-17T02:58:30Z

@ncw I think it mostly looks good, but the issue I mentioned here is still not fixed.

Here's a test that demonstrates it: nielash@d670d07 (same test would have passed previously)

...which passes once this is reverted: nielash@ff4198d

Otherwise I think it looks good!

…false In v1.66.0 the changes to enable metadata preservation on directories introduced a regression, namely that empty directories were created despite the state of the --create-empty-src-dirs flag. This patch fixes the problem by letting the normal rclone directory creation create the directories and fixing up their timestamps and metadata afterwards if --create-empty-src-dirs=false. Fixes #7689 See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/ See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/

ncw · 2024-04-17T16:01:00Z

Thanks for the review @nielash

This feels like the never ending bug!

I've fixed the issue that you pointed out (I realize I misunderstood the purpose of s.modifiedDirs it is to keep the modified status of the directories themselves, not the parents like we need for delayed modtime setting).

Please take another look!

I also fixed a bug pointed out in the forum https://forum.rclone.org/t/empty-dirs-not-wanted/45059/14

Hopefully if this checks out I'll squash it down, neaten it up and then that is the last bit needed for v1.66.1

v1.67.0-beta.7906.91d78fd38.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

nielash · 2024-04-17T22:40:55Z

Added a comment for you inline. This is a tricky one indeed!

ncw · 2024-04-22T17:16:07Z

@nielash I can't find your comment! I thought I saw a notification but it has gone! Please post a link - thank you :-)

nielash · 2024-04-22T17:29:51Z

@ncw here you go! 91d78fd#r141149941

…false In v1.66.0 the changes to enable metadata preservation on directories introduced a regression, namely that empty directories were created despite the state of the --create-empty-src-dirs flag. This patch fixes the problem by letting the normal rclone directory creation create the directories and fixing up their timestamps and metadata afterwards if --create-empty-src-dirs=false. Fixes #7689 See: https://forum.rclone.org/t/empty-dirs-not-wanted/45059/ See: https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/

ncw · 2024-05-03T11:32:25Z

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.66.1

Thanks for all your help @nielash :-)

ncw changed the title ~~1.66.0 has broken the default behavior from before~~ rclone copy creating empty directories it didn't create before Apr 2, 2024

ncw added the bug label Apr 2, 2024

ncw added this to the v1.67 milestone Apr 2, 2024

ncw added the Point Release Candidate label Apr 2, 2024

nielash mentioned this issue Apr 6, 2024

bisync: the tests are too noisy #7736

Open

ncw mentioned this issue Apr 9, 2024

SFTP slow with many directories #7746

Closed

ncw closed this as completed in 10eb474 May 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rclone copy creating empty directories it didn't create before #7689

rclone copy creating empty directories it didn't create before #7689

YukiUnHappy commented Mar 25, 2024

cau118 commented Mar 26, 2024

sabitm commented Mar 28, 2024

nielash commented Mar 29, 2024

sabitm commented Apr 2, 2024

ncw commented Apr 2, 2024 •

edited

Loading

YukiUnHappy commented Apr 3, 2024

ncw commented Apr 3, 2024

ncw commented Apr 5, 2024

nielash commented Apr 5, 2024

nielash commented Apr 6, 2024

nielash commented Apr 6, 2024

ncw commented Apr 16, 2024

nielash commented Apr 17, 2024

ncw commented Apr 17, 2024

nielash commented Apr 17, 2024

ncw commented Apr 22, 2024

nielash commented Apr 22, 2024

ncw commented May 3, 2024

rclone copy creating empty directories it didn't create before #7689

rclone copy creating empty directories it didn't create before #7689

Comments

YukiUnHappy commented Mar 25, 2024

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Which cloud storage system are you using? (e.g. Google Drive)

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

How to use GitHub

cau118 commented Mar 26, 2024

sabitm commented Mar 28, 2024

nielash commented Mar 29, 2024

sabitm commented Apr 2, 2024

ncw commented Apr 2, 2024 • edited Loading

YukiUnHappy commented Apr 3, 2024

ncw commented Apr 3, 2024

ncw commented Apr 5, 2024

nielash commented Apr 5, 2024

nielash commented Apr 6, 2024

nielash commented Apr 6, 2024

ncw commented Apr 16, 2024

nielash commented Apr 17, 2024

ncw commented Apr 17, 2024

nielash commented Apr 17, 2024

ncw commented Apr 22, 2024

nielash commented Apr 22, 2024

ncw commented May 3, 2024

What is your rclone version (output from `rclone version`)

The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

ncw commented Apr 2, 2024 •

edited

Loading