rclone sync --create-empty-src-dirs to b2 doesn't create empty directories

[gentoo-root]

What is the problem you are having with rclone?

I have a crypt remote called b2enc over a b2 remote called b2. rclone sync --create-empty-src-dirs to b2enc seems to have no effect on empty directories. When I rclone mount b2enc:, these directories don't show up. With rclone 1.68.2, sync silently ignores them and doesn't create them on remote. With rclone 1.69.1, sync prints messages like these, but still the directories are not created:

2025/02/22 13:16:35 INFO  : DIRECTORY/NAME: Making directory

What is your rclone version (output from rclone version)

rclone v1.69.1
- os/version: arch (64 bit)
- os/kernel: 6.12.9-arch1-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.0
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Arch Linux, 64-bit

Which cloud storage system are you using? (e.g. Google Drive)

Backblaze

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

exec rclone sync --create-empty-src-dirs --links --filter-from ~/.config/rclone/filter-home.txt --fast-list --progress --transfers 12 --verbose ~ b2enc:home

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2025/02/22 14:20:08 DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "sync" "--create-empty-src-dirs" "--links" "--filter-from" "/home/max/.config/rclone/filter-home.txt" "--fast-list" "--progress" "--transfers" "12" "-vv" "/home/max" "b2enc:home"]
2025/02/22 14:20:08 DEBUG : Creating backend with remote "/home/max"
2025/02/22 14:20:08 DEBUG : Using config file from "/home/max/.config/rclone/rclone.conf"
2025/02/22 14:20:08 DEBUG : Creating backend with remote "b2enc:home"
2025/02/22 14:20:08 DEBUG : Creating backend with remote "b2:backup-XXXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXX"

2025/02/22 14:20:10 DEBUG : .ssh: Excluded
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s
2025/02/22 14:20:10 DEBUG : .cache: Excluded
<...>
2025/02/22 14:26:04 INFO  : EMPTY/DIRECTORY/NUMBER/ONE: Making directory
Transferred:      581.914 MiB / 581.914 MiB, 100%, 2.636 MiB/s, ETA 0s
Checks:            532878 / 542881, 98%
Transferred:            9 / 9, 100%
Elapsed time:      5m54.0s
2025/02/22 14:26:04 INFO  : EMPTY/DIRECTORY/NUMBER/TWO: Making directory
Transferred:      581.914 MiB / 581.914 MiB, 100%, 2.636 MiB/s, ETA 0s
Checks:            532878 / 542882, 98%
Transferred:            9 / 9, 100%
Elapsed time:      5m54.0s
<...>
2025/02/22 14:26:23 DEBUG : Encrypted drive 'b2enc:home': Waiting for transfers to finish
Transferred:      581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4s
2025/02/22 14:26:23 DEBUG : Waiting for deletions to finish
Transferred:      581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4sTransferred:            581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4s
2025/02/22 14:26:23 INFO  : 
Transferred:      581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4s

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[kapitainsky]

Empty directories are not supported by B2.

You can switch to S3 compatibility layer and use https://rclone.org/s3/#s3-directory-markers

[gentoo-root]

Empty directories are not supported by B2.

I see... Thanks for the information. Can rclone do something on its side, e.g., upload directory markers like on S3 (or files with the .rclonedir extension if the trailing slash is not supported for filenames on B2)?

UX-wise, an error message when trying to use --create-empty-src-dirs on B2 would make sense, and the new behavior in 1.69.1 produces a lot of noise printing "Making directory" for all empty dirs on every sync (and the message is also misleading).

[gentoo-root]

You can switch to S3 compatibility layer

It seems it's not a drop-in solution, because it started reuploading and replacing many terabytes of my data, and I'm not sure why. After a file is uploaded via b2, the b2 remote shows its mtime, but the s3 remote shows the upload time. After the same file is reuploaded via s3, they switch: b2 shows the upload time, and s3 shows the actual mtime.

Edit: that's why:
For s3: The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns.
For b2: The modification time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard.

[kapitainsky]

For s3: The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns.
For b2: The modification time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard.

You could use --modify-window flag to rectify it. Sounds like 1ms should work here. But the best is to test.

https://rclone.org/docs/#modify-window-time

After a file is uploaded via b2, the b2 remote shows its mtime, but the s3 remote shows the upload time. After the same file is reuploaded via s3, they switch: b2 shows the upload time, and s3 shows the actual mtime.

Or maybe not.. Indeed it looks like you have to plan which API you are using from the start.

[gentoo-root]

Sounds like 1ms should work here.

No, not at all, it's not a matter of 1 ms, the upload timestamp and the mtime of the file can be years or even decades apart. When the file is uploaded via the b2 remote, b2 shows mtime, and s3 shows the upload date. When I start syncing via the s3 remote, rclone doesn't see the mtimes set in X-Bz-Info-src_last_modified_millis, because it queries X-Amz-Meta-Mtime, so it starts reuploading all files. Even worse, as I have versions enabled for my bucket, the paycheck will double until the old versions are cleaned up.

Indeed it looks like you have to plan which API you are using from the start.

Yeah, better luck next time 😅

Back on topic, there are two things that can be done on rclone side: directory markers for b2 (I guess, it should be doable, because I see b2 understands directory markers created with s3), and better UX for the messages (may be not needed if directory markers are implemented). What do you think?

[kapitainsky]

directory markers for b2

https://www.backblaze.com/docs/cloud-storage-files#file-names

"File names cannot start with /, end with /, or contain //."

Which means that it can not use the same method as used for S3 where marker ends with /.

Somebody would have to investigate it further I guess.

I wonder if creating some hidden empty file would be acceptable solution.. It would "create" these dirs.

[gentoo-root]

Which means that it can not use the same method as used for S3 where marker ends with /.

Hmm, I wonder then:

1. How S3 API creates those markers on B2? Maybe "File names cannot start with /, end with /, or contain //" is a limitation of B2 API, not the underlying storage, though.
2. How I see empty directories via B2 API after the markers were created with S3 API?

Which means that it can not use the same method

Even if so, there are still two alternatives possible:

1. Instead of a file called dir/, create dir.rclonedir as a marker.
2. Create the .rclonedir file inside the dir (as you suggsted).

(For this example, naming was chosen to resemble *.rclonelink stubs for symlinks; I don't insist on these particular names.)

[c3piio]

Hello, have a hack-y solution for this. Place a .keep file in every empty directory. Deepest level is fine if you only want to sync the structure. If you are on Windows 11, running the following in a .bat file will place a .keep file in empty directories. They will then sync to backblaze using the native b2 api. I went down this road today. I already backed up 750 gb via b2, only to realize that empty folders didn't copy over. I tried connecting via s3 api, but that wanted to re-upload everything because the metadata is different between the two. So if you are already using the b2 api, using the .keep file "trick" works fine and takes up a few kb. Here is the code for the .bat file:

@echo off
REM Batch file: Place a .keep file in every empty folder under ROOTDIR

setlocal enabledelayedexpansion

REM Set this to the path you want to scan:
set "ROOTDIR=D:\Folder"

REM For each folder (deepest first, so we catch truly empty ones)
for /f "delims=" %%F in ('dir "%ROOTDIR%" /ad /b /s ^| sort /r') do (
    REM List all contents (including hidden, system, etc.)
    dir /a /b "%%F" | findstr . >nul
    if errorlevel 1 (
        type nul > "%%F\.keep"
        echo .keep placed in: %%F
    )
)

echo.
echo Done: .keep placed in all empty folders.
pause

[ncw]

I'm going to close this as we can't make directory markers for b2, alas! #8414 (comment)

[gentoo-root]

as we can't make directory markers for b2, alas!

I don't quite understand why; could you elaborate? As I commented above, there are two alternative approaches to directory markers that don't use a slash. Did you consider them? Could you explain why they didn't work?

Also, as I said in that comment above, B2 API mounts understand dir markers created with the S3 API, so it might be possible to squeeze in a slash somehow, but I don't insist on that, because there are two other options not needing a slash.

Regarding c3piio's suggestion to manually place .keep files, that wouldn't work for me, as I need to preserve the exact directory tree locally. I can't just add some files that aren't supposed to be there. Even if I could, it would be a major maintenance burden to create those markers every time a new directory is created, and to remove them as soon as the first file is added. I'd very much prefer if rclone did that job on the remote filesystem, as one of my suggestions says.

[ncw]

Even if so, there are still two alternatives possible:

1. Instead of a file called dir/, create dir.rclonedir as a marker.
2. Create the .rclonedir file inside the dir (as you suggsted).

Option 2 you can do locally if you want.

Option 1 would work but it isn't how the other storage backends work. It could be made to work but I think it would be swimming against what the other providers do and what backblaze expect.

[gentoo-root]

Option 2 you can do locally if you want.

My previous message explains in details why creating dummy files locally is a bad option for me. Rclone is a great abstraction layer over various remote filesystems that have their peculiarities, so it's sort of weird to make the user do part of its job, when it could be part of rclone.

Option 1 would work but it isn't how the other storage backends work.

Could you elaborate? Is the only difference that other backends use suffix /, and B2 would use suffix .rclonedir? The suffix itself looks like something that can be easily abstracted: backends can provide specific strings for those suffixes. It also goes in line with *.rclonelink files that are used to encode symlinks.

it would be swimming against <...> what backblaze expect.

I'm sorry, but I fail to see why B2 would care what files I put there and what they are called. I.e. I don't understand how uploading an empty file named dirname.rclonedir breaks B2's expectations.