Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom metadata is not saved when S3 MPU is used #39

Closed
elek opened this issue Sep 30, 2021 · 7 comments
Closed

Custom metadata is not saved when S3 MPU is used #39

elek opened this issue Sep 30, 2021 · 7 comments
Labels
blocked bug Something isn't working s3-tests

Comments

@elek
Copy link
Contributor

elek commented Sep 30, 2021

Custom metadata is not saved when S3 MPU is used (in this example the threshold is 8MB)

As you can see in the second case the metadata is empty, but with real AWS it's foo: bar in both cases.

> aws s3 cp --endpoint http://192.168.80.4:7777 --metadata foo=bar 1mb s3://bucket1/1mb
upload: ./1mb to s3://bucket1/1mb                                
 > aws s3 cp --endpoint http://192.168.80.4:7777 --metadata foo=bar 20mb s3://bucket1/20mb
upload: ./20mb to s3://bucket1/20mb                               
> aws s3api get-object --endpoint http://192.168.80.4:7777 --bucket bucket1 --key 1mb x 
{
    "AcceptRanges": "bytes",
    "LastModified": "Thu, 30 Sep 2021 11:36:10 GMT",
    "ContentLength": 1048576,
    "ETag": "\"f5f16eee8ad5a86a3439dc75429831ee\"",
    "ContentType": "application/octet-stream",
    "Metadata": {
        "foo": "bar"
    }
}
> aws s3api get-object --endpoint http://192.168.80.4:7777 --bucket bucket1 --key 20mb x
{
    "AcceptRanges": "bytes",
    "LastModified": "Thu, 30 Sep 2021 11:36:21 GMT",
    "ContentLength": 20971520,
    "ETag": "\"80f096bdc4c7fd9639307cd1174b398a-3\"",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

Credit:

The problem is identified by the rclone s3 unit tests during the conversation in rclone/rclone#5616 where @olefrost suggested to re-run s3 unit tests with storj gateway-mt and @ncw drew attention to this problem from many failures.

@mniewrzal
Copy link
Contributor

From what I can tell bug applies also to storj/gatewa-st

@elek
Copy link
Contributor Author

elek commented Oct 1, 2021

Looks to be a more generic problem, not just a gateway-mt one.

As far as I see, the uplink itself doesn't support custom metadata in BeginObject call, as it's not supported in protocol level either. Probably because during the normal upload the metadata is part of the ObjectCommitRequest

I checked again with aws cli --debug option that AWS expects the custom metadata in the CreateMultipartUpload project. Which means it can be fixed only with updating the protocol (storj/common) the satellites (storj/storj) the uplink and finally the gateways. (but fixme if I am wrong)

2021-10-01 10:26:57,978 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateMultipartUpload) with params: {'url_path': '/bucket1/out?uploads', 
'query_string': {}, 'method': 'POST', 'headers': {'x-amz-meta-foo': 'bar', 'User-Agent': 'aws-cli/1.20.34 Python/3.9.7 Linux/5.14.7-zen1-1-zen botocore/1.21.34'}, 'body': b'', 'url': 'http://
192.168.224.4:7777/bucket1/out?uploads', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x7fe0aae30430>, 'has_streaming_input': False, 'auth_type
': None, 'signing': {'bucket': 'bucket1'}}}
...
...
2021-10-01 10:26:57,978 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=http://192.168.224.4:7777/bucket1/out?uploads, headers={'x-amz-meta-foo': b'bar', 'User-Agent': b'aws-cli/1.20.34 Python/3.9.7 Linux/5.14.7-zen1-1-zen botocore/1.21.34', 'X-Amz-Date': b'20211001T082657Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=jumih53d2jk4z7opoodtvaxgmmaq/20211001/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-foo, Signature=c49794da9d56eba218bb264c36d3e55a75f7d6bf29f9d3dbca32d86e7ccf2ee2', 'Content-Length': '0'}>

@mniewrzal
Copy link
Contributor

Unfortunately, I need to agree (https://review.dev.storj.io/c/storj/gateway/+/5906). We missed that case and we need to extend uplink API.

@amwolff
Copy link
Member

amwolff commented Oct 1, 2021

A bit related to this, we have a test(s) that check for the existence of custom metadata with regards to MPU, but these checks are commented out, e.g. this one.

This one is directly related to committing not returning the "real" object (no metadata at all).

@wthorp
Copy link
Contributor

wthorp commented Feb 10, 2022

This appears to still be awaiting at least https://review.dev.storj.io/c/storj/storj/+/6624

@wthorp wthorp added the blocked label Feb 10, 2022
@shaupt131
Copy link

Blocker for this issue has been turned into milestone for Metainfo team: https://github.com/storj/team-metainfo/milestone/2

@Erikvv Erikvv pinned this issue Apr 7, 2022
@Erikvv Erikvv unpinned this issue Apr 7, 2022
@storjBuildBot
Copy link
Collaborator

Change https://review.dev.storj.io/c/storj/gateway-mt/+/7422 mentions this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked bug Something isn't working s3-tests
Projects
No open projects
Archived in project
Development

No branches or pull requests

6 participants