-
Notifications
You must be signed in to change notification settings - Fork 253
Specifying a port in s3cmd makes v4 fail #84
Description
Here is the s3cmd command and its output:
# s3cmd --debug -c s3cfg.mem-v4 mb s3://testbucket
DEBUG: s3cmd version 1.6.1
DEBUG: ConfigParser: Reading file 's3cfg.mem-v4'
DEBUG: ConfigParser: access_key->ac...7_chars...1
DEBUG: ConfigParser: secret_key->ve...11_chars...1
DEBUG: ConfigParser: host_base->{{HOST}}:80
DEBUG: ConfigParser: host_bucket->%(bucket)s.{{HOST}}:80
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: website_endpoint->http://192.168.1.101:80/answertoall
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'mb' using UTF-8
DEBUG: Unicodising 's3://testbucket' using UTF-8
DEBUG: Command: mb
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(testbucket): testbucket.{{HOST}}:80
DEBUG: canonical_headers = host:testbucket.{{HOST}}:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z
DEBUG: Canonical Request:
PUT
/
host:testbucket.{{HOST}}:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=accessKey1/20160628/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=44805e7ecae4b14c918a92142513f58b19dbc3051b175f66fe85d8c3b322b237', 'x-amz-date': '20160628T105414Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(testbucket): testbucket.{{HOST}}:80
DEBUG: ConnMan.get(): creating new connection: http://testbucket.{{HOST}}:80
DEBUG: non-proxied HTTPConnection(testbucket.{{HOST}}:80)
DEBUG: format_uri(): /
DEBUG: Sending request method_string='PUT', uri='/', headers={'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=accessKey1/20160628/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=44805e7ecae4b14c918a92142513f58b19dbc3051b175f66fe85d8c3b322b237', 'x-amz-date': '20160628T105414Z'}, body=(0 bytes)
DEBUG: Response: {'status': 403, 'headers': {'x-amz-id-2': '781591391d87c1f6a3d3', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'keep-alive', 'x-amz-request-id': '781591391d87c1f6a3d3', 'date': 'Tue, 28 Jun 2016 10:54:15 GMT', 'content-type': 'application/xml'}, 'reason': 'Forbidden', 'data': '<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message><Resource></Resource><RequestId>781591391d87c1f6a3d3</RequestId></Error>'}
DEBUG: ConnMan.put(): connection put back to pool (http://testbucket.{{HOST}}:80#1)
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: x-amz-id-2: 781591391d87c1f6a3d3
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: connection: keep-alive
DEBUG: HttpHeader: x-amz-request-id: 781591391d87c1f6a3d3
DEBUG: HttpHeader: date: Tue, 28 Jun 2016 10:54:15 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided.'
DEBUG: ErrorXML: Resource: None
DEBUG: ErrorXML: RequestId: '781591391d87c1f6a3d3'
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided.And here is the log from our Kibana instance (First, the CanonicalRequest, and second the constructed stringToSign):
{
"_index": "logstash-scality-2016.06.28",
"_type": "log",
"_id": "AVWWpUvFjIh-3to3jiRi",
"_score": null,
"_source": {
"name": "S3",
"bucketName": "testbucket",
"canonicalReqResult": "PUT\n/\n\nhost:testbucket.{{HOST}}\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20160628T105414Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"req_id": "781591391d87c1f6a3d3",
"level": "debug",
"message": "constructed canonicalRequest",
"hostname": "connect05.scality.riakpoc.local",
"pid": 85,
"@version": "1",
"@timestamp": "2016-06-28T10:54:15.022Z",
"fields": {},
"beat": {},
"type": "log",
"count": 1
},
"fields": {
"@timestamp": [
1467111255022
]
},
"highlight": {
"req_id": [
"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
],
"req_id.raw": [
"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
]
},
"sort": [
1467111255022
]
}
{
"_index": "logstash-scality-2016.06.28",
"_type": "log",
"_id": "AVWWpUvFjIh-3to3jiRl",
"_score": null,
"_source": {
"name": "S3",
"bucketName": "testbucket",
"stringToSign": "AWS4-HMAC-SHA256\n20160628T105414Z\n20160628/US/s3/aws4_request\n1dff0237362e0f8b069cba1748f99ccca9221cc54a6d37187a268071d38e65e4",
"req_id": "781591391d87c1f6a3d3",
"level": "trace",
"message": "constructed stringToSign",
"hostname": "connect05.scality.riakpoc.local",
"pid": 85,
"@version": "1",
"@timestamp": "2016-06-28T10:54:15.022Z",
"type": "log",
"count": 1,
"fields": {},
"beat": {}
},
"fields": {
"@timestamp": [
1467111255022
]
},
"highlight": {
"req_id": [
"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
],
"req_id.raw": [
"@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
]
},
"sort": [
1467111255022
]
}
As you can see, comparing the two canonical Requests:
(from s3cmd)
PUT
/
host:testbucket.{{HOST}}:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
(from S3)
PUT
/
host:testbucket.{{HOST}}
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
We can see that the only difference is that s3cmd includes the target's port into the canonical request, while the S3 server computes it without a port. In this setup here, we're using HAProxy to load-balance the requests only multiple S3 servers, so it makes sense to avoid using the port inside S3; but as s3cmd is actually using it, it may make the v4 auth fail for no real valid reason.
I do not think it is for us to fix, but we should at least document it properly.
@vrancurel @GiorgioRegni what is your take on this ?