Support arbitrary delimiters

[jtolio]

The S3 protocol supports arbitrary delimiters when asking to list objects. In other words, when using S3, you can pretend that something besides "/" separates path elements by asking to list all paths with a shared prefix up to a next delimiter of any type. Storj uses the forward slash as an encryption element delimiter, so it doesn't make a ton of sense for us to allow arbitrary delimiters in normal use cases. However, this means that we are incompatible with S3 for some applications.

We explicitly disable support for arbitrary delimiters here:

gateway-st/miniogw/gateway.go

Lines 211 to 213 in 9768ca2

	if delimiter != "" && delimiter != "/" {
	return minio.ListObjectsInfo{}, minio.UnsupportedDelimiter{Delimiter: delimiter}
	}

A recent consequence of this is that we are incompatible with Cockroach's backup tool. cockroachdb/cockroach#69165

Cockroach is probably not the last tool we won't work with, so as that Cockroach ticket points out, it's probably preferable we are slow over broken. So, we should mimic arbitrary delimiter support by listing recursively and then filtering. We will have to do this client-side because the arbitrary delimiter will be processed on the paths prior to encryption, so there won't be any opportunity for Satellite-side optimization.

Since this only applies to things trying to use us like S3 (libuplink doesn't have a delimiter option at all), the gateway-st miniogw package is the right place for this (as gateway-st and gateway-mt will pick it up).

tl;dr: miniogw's ListObjects method should, when given a delimiter besides "/" or "", internally list all objects, then filter down to the provided delimiter, mimicking support for arbitrary delimiters.

[jtolio]

cc @amwolff (thanks for the original cockroach ticket filing!) and also https://storjlabs.atlassian.net/browse/IN-607

[OlegOLK]

Upvote for this. It would be super nice to have that functionality. But I am a bit confused with the next things:
aws --endpoint-url=https://gateway.eu1.storjshare.io s3 cp s3://<bucket> . --recursive --exclude "*" --include "<one>" --include "<two>" --profile storj works like a charm but when I am trying to do the same through .net SDK
new ListObjectsRequest { BucketName = resp.Buckets.First().BucketName, Delimiter = "*", Prefix = "<one>*<two>" }
I got error:
A header you provided implies functionality that is not implemented

[storjBuildBot]

Change https://review.dev.storj.io/c/storj/gateway/+/6434 mentions this issue.

[amwolff]

@OlegOLK I believe the reason the former works and the latter doesn't is that in the first case, AWS CLI lists the whole bucket and filters it locally. Amazon's implementation of S3 doesn't support filtering using wildcards, and Storj's Gateway-MT and Gateway-ST also don't support it. So even if you tried it against, e.g., AWS S3, it would give you results different from the first command you posted. Fortunately, filtering by prefix and delimiter can be pretty powerful, too, and support for that in Gateway-MT and Gateway-ST is coming!

[amwolff]

Arbitrary prefix and delimiter support (or making the ListObjects and ListObjectsV2 actions fully S3-compatible) should be in the next Gateway-ST release (13th of December). For Gateway-MT, it should be there after the production deployment.

Here's the changeset associated with this issue:

1. https://review.dev.storj.io/c/storj/gateway/+/6434
2. https://review.dev.storj.io/c/storj/gateway/+/6448
3. https://review.dev.storj.io/c/storj/gateway/+/6449
4. https://review.dev.storj.io/c/storj/gateway/+/6450
5. https://review.dev.storj.io/c/storj/gateway/+/6452