Bucket Based Remotes: empty directories

[yparitcher]

Progress

1. error on empty directory

• azureblob
• b2
• gcs
• qingstor
• s3
• swift

Current Situation:

Rclone does not officially support empty directories on BBR, however as many parts of the rclone api use file based systems there are many edge cases where rclone acts as if empty directories exist on BBR.
this causes non existent paths to be validated as rclone cant tell the difference between them and empty directories. this can cause inconsistencies with mount etc.
this has been discussed before, see #3427 #3415

Proposed solutions:

1. enforce non support of empty directories on BBR:

• make empty directories be an invalid path
• must fix list commands
• this will need work to make consistent with mount and similar commands which expect a filesystem (empty directories)

2. support empty directories on BBR via a hidden empty file:

• this is already done by some clients on other remotes (B2 web client, some S3 clients)
• need standardization to ensure that rclone uses the same files as others
• require refactoring of many BBR to include this logic
• cause some confusion about these files, but make filesystem commands (mount) work better
• requires more api calls - more expensive

3. 1 & 2 via a flag:

• this give the most options but will be slightly more confusing to use and require duplicate codepaths

---

if i missed anything please comment and i will add them in.

this will be a significant decision. rclone should have an policy which will help unified implementations for different backends.

i do not mind helping will this but it will definitely require help from many people.
thanks to all those that have made these remotes work until now
@ncw Please review

[ncw]

I think we should probably do 1. first - no flag needed.

As for 2. rclone mostly just contents itself with ignoring the 0 length directory files other programs make. Making rclone actively put them in would be quite a lot of work so if we want to do it, should probably be in phase 2. I think it needs a backend flag so you can set it on a per backend basis.

Unifying code between BBR backends will be important!

There are 6 BBR b2, s3, gcs, swift, qingstor, azureblob. It is possible to get free accounts for all of those except for gcs and azureblob. (you can test s3 against minio or alibabacloud). gcs has a free tier and azureblob may too. Signing up for free services will probably mean putting your credit card in though!

I'd love to have your help with this :-)

[yparitcher]

besides for the azureblob, b2, s3, gcs, swift, qingstor that are bucket based Features().BucketBased

googlephotos and hubic do not have the empty directory flag set: Features().CanHaveEmptyDirectories
what is their status regarding this?

[yparitcher]

Right now on BBR Mkdir is a wrapper for createbucket and always (if the bucket is valid) returns without error.

in order to not support empty directories,i would like to make Mkdir return an error on BBR when called on a non existent directory. this will require rewriting some tests etc that expect Mkdir to always succeed.

the same goes for Rmdir

does this sound reasonable?

[yparitcher]

operations.Rmdirs() has different behavior w & w/o --fast-list/listR:

rclone/fs/operations/operations.go

Lines 1316 to 1324 in 693112d

	func Rmdirs(ctx context.Context, f fs.Fs, dir string, leaveRoot bool) error {
	dirEmpty := make(map[string]bool)
	dirEmpty[dir] = !leaveRoot
	err := walk.Walk(ctx, f, dir, true, fs.Config.MaxDepth, func(dirPath string, entries fs.DirEntries, err error) error {
	if err != nil {
	fs.CountError(err)
	fs.Errorf(f, "Failed to list %q: %v", dirPath, err)
	return nil
	}

• without listR errors in walk.Walk() are trapped by the func passed to Walk() and are not propagated. (if a wrong directory is listed causing an error the error is printed here and the Rmdirs succeeds without error.)
• with listR errors in walk.Walk() are not trapped by the func and are propagated. (if a wrong directory is listed causing an error Rmdirserrors and fails.)

[ncw]

besides for the azureblob, b2, s3, gcs, swift, qingstor that are bucket based Features().BucketBased

googlephotos and hubic do not have the empty directory flag set: Features().CanHaveEmptyDirectories
what is their status regarding this?

hubic is a thin wrapper around swift so that is correct.

I think the google photos is a mistake - you can have empty directories.

[ncw]

Right now on BBR Mkdir is a wrapper for createbucket and always (if the bucket is valid) returns without error.

in order to not support empty directories,i would like to make Mkdir return an error on BBR when called on a non existent directory. this will require rewriting some tests etc that expect Mkdir to always succeed.

the same goes for Rmdir

does this sound reasonable?

If the directory doesn't exist when we do rclone mkdir bbr:bucket/dir that is probably the moment we should be creating the zero length directory marker

I think returning an error will break too much stuff - we want the directory made, we don't want an error returned saying that we can't make the directory.

[ncw]

operations.Rmdirs() has different behavior w & w/o --fast-list/listR:

rclone/fs/operations/operations.go

Lines 1316 to 1324 in 693112d

	func Rmdirs(ctx context.Context, f fs.Fs, dir string, leaveRoot bool) error {
	dirEmpty := make(map[string]bool)
	dirEmpty[dir] = !leaveRoot
	err := walk.Walk(ctx, f, dir, true, fs.Config.MaxDepth, func(dirPath string, entries fs.DirEntries, err error) error {
	if err != nil {
	fs.CountError(err)
	fs.Errorf(f, "Failed to list %q: %v", dirPath, err)
	return nil
	}

• without listR errors in walk.Walk() are trapped by the func passed to Walk() and are not propagated. (if a wrong directory is listed causing an error the error is printed here and the Rmdirs succeeds without error.)

That is so we do as much as possible and report a failure at the end.

• with listR errors in walk.Walk() are not trapped by the func and are propagated. (if a wrong directory is listed causing an error Rmdirserrors and fails.)

Hmm, have you got a demo I can try for this?

[yparitcher]

If the directory doesn't exist when we do rclone mkdir bbr:bucket/dir that is probably the moment we should be creating the zero length directory marker

exactly.

I think returning an error will break too much stuff - we want the directory made, we don't want an error returned saying that we can't make the directory.

i agree, just this is limiting 1. to be just in the list command.
also B2 does not allow objects ending in / so the directory marker would have to be a hidden subfile. this would require redoing all that stuff anyway to make it detect the markers properly.

i am not sure which way to go here.

[yparitcher]

Hmm, have you got a demo I can try for this?

try running the S3 integration tests on https://github.com/yparitcher/rclone/tree/BBR-empty-dir
the operations test purge will succeed without listR and fail with listR

also BBR give different behavior when run from the root or from a bucket you caught most of them in #3421 however some tests i had to change to work when running from a bucket, as the tests were designed for root

[shynome]

I backup my postgres db directory to s3 storage with rclone, today I have verify the backup, it is broken because lose some empty directory like pg_tblspc, pg_replslot, I think we should add some warning for someone who sync empty directory to s3 storage, because today we have not implemented this feature

I like support empty directories on BBR via a hidden empty file

the full log

2020-12-20 13:26:14.201 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2020-12-20 13:26:14.201 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2020-12-20 13:26:14.207 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-12-20 13:26:14.222 UTC [1] LOG:  could not open directory "pg_tblspc": No such file or directory
2020-12-20 13:26:14.226 UTC [19] LOG:  database system was interrupted; last known up at 2020-12-20 13:09:08 UTC
2020-12-20 13:26:14.226 UTC [19] LOG:  creating missing WAL directory "pg_wal/archive_status"
2020-12-20 13:26:14.467 UTC [19] LOG:  could not open directory "pg_tblspc": No such file or directory
2020-12-20 13:26:14.468 UTC [19] LOG:  could not open directory "pg_tblspc": No such file or directory
2020-12-20 13:26:14.468 UTC [19] FATAL:  could not open directory "pg_replslot": No such file or directory
2020-12-20 13:26:14.469 UTC [1] LOG:  startup process (PID 19) exited with exit code 1
2020-12-20 13:26:14.469 UTC [1] LOG:  aborting startup due to startup process failure
2020-12-20 13:26:14.472 UTC [1] LOG:  database system is shut down

[ncw]

We now have these to support empty directory markers

  --azureblob-directory-markers   Upload an empty object with a trailing slash when a new directory is created
  --gcs-directory-markers         Upload an empty object with a trailing slash when a new directory is created
  --s3-directory-markers          Upload an empty object with a trailing slash when a new directory is created

This covers the major bucket based backends.

I'm going to close this now. We might do swift and b2 in the future depending on demand.

[lemeshovich]

We now have these to support empty directory markers

please add --b2-directory-markers 🙏