Custom metadata is not saved when S3 MPU is used

[elek]

Custom metadata is not saved when S3 MPU is used (in this example the threshold is 8MB)

As you can see in the second case the metadata is empty, but with real AWS it's foo: bar in both cases.

> aws s3 cp --endpoint http://192.168.80.4:7777 --metadata foo=bar 1mb s3://bucket1/1mb
upload: ./1mb to s3://bucket1/1mb                                
 > aws s3 cp --endpoint http://192.168.80.4:7777 --metadata foo=bar 20mb s3://bucket1/20mb
upload: ./20mb to s3://bucket1/20mb                               
> aws s3api get-object --endpoint http://192.168.80.4:7777 --bucket bucket1 --key 1mb x 
{
    "AcceptRanges": "bytes",
    "LastModified": "Thu, 30 Sep 2021 11:36:10 GMT",
    "ContentLength": 1048576,
    "ETag": "\"f5f16eee8ad5a86a3439dc75429831ee\"",
    "ContentType": "application/octet-stream",
    "Metadata": {
        "foo": "bar"
    }
}
> aws s3api get-object --endpoint http://192.168.80.4:7777 --bucket bucket1 --key 20mb x
{
    "AcceptRanges": "bytes",
    "LastModified": "Thu, 30 Sep 2021 11:36:21 GMT",
    "ContentLength": 20971520,
    "ETag": "\"80f096bdc4c7fd9639307cd1174b398a-3\"",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

Credit:

The problem is identified by the rclone s3 unit tests during the conversation in rclone/rclone#5616 where @olefrost suggested to re-run s3 unit tests with storj gateway-mt and @ncw drew attention to this problem from many failures.

[mniewrzal]

From what I can tell bug applies also to storj/gatewa-st

[elek]

Looks to be a more generic problem, not just a gateway-mt one.

As far as I see, the uplink itself doesn't support custom metadata in BeginObject call, as it's not supported in protocol level either. Probably because during the normal upload the metadata is part of the ObjectCommitRequest

I checked again with aws cli --debug option that AWS expects the custom metadata in the CreateMultipartUpload project. Which means it can be fixed only with updating the protocol (storj/common) the satellites (storj/storj) the uplink and finally the gateways. (but fixme if I am wrong)

2021-10-01 10:26:57,978 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateMultipartUpload) with params: {'url_path': '/bucket1/out?uploads', 
'query_string': {}, 'method': 'POST', 'headers': {'x-amz-meta-foo': 'bar', 'User-Agent': 'aws-cli/1.20.34 Python/3.9.7 Linux/5.14.7-zen1-1-zen botocore/1.21.34'}, 'body': b'', 'url': 'http://
192.168.224.4:7777/bucket1/out?uploads', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x7fe0aae30430>, 'has_streaming_input': False, 'auth_type
': None, 'signing': {'bucket': 'bucket1'}}}
...
...
2021-10-01 10:26:57,978 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=http://192.168.224.4:7777/bucket1/out?uploads, headers={'x-amz-meta-foo': b'bar', 'User-Agent': b'aws-cli/1.20.34 Python/3.9.7 Linux/5.14.7-zen1-1-zen botocore/1.21.34', 'X-Amz-Date': b'20211001T082657Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=jumih53d2jk4z7opoodtvaxgmmaq/20211001/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-foo, Signature=c49794da9d56eba218bb264c36d3e55a75f7d6bf29f9d3dbca32d86e7ccf2ee2', 'Content-Length': '0'}>

[mniewrzal]

Unfortunately, I need to agree (https://review.dev.storj.io/c/storj/gateway/+/5906). We missed that case and we need to extend uplink API.

[amwolff]

A bit related to this, we have a test(s) that check for the existence of custom metadata with regards to MPU, but these checks are commented out, e.g. this one.

This one is directly related to committing not returning the "real" object (no metadata at all).

[wthorp]

This appears to still be awaiting at least https://review.dev.storj.io/c/storj/storj/+/6624

[shaupt131]

Blocker for this issue has been turned into milestone for Metainfo team: https://github.com/storj/team-metainfo/milestone/2

[storjBuildBot]

Change https://review.dev.storj.io/c/storj/gateway-mt/+/7422 mentions this issue.