Skip to content

Specifying a port in s3cmd makes v4 fail #84

@ghost

Description

Here is the s3cmd command and its output:

# s3cmd --debug -c s3cfg.mem-v4 mb s3://testbucket
DEBUG: s3cmd version 1.6.1
DEBUG: ConfigParser: Reading file 's3cfg.mem-v4'
DEBUG: ConfigParser: access_key->ac...7_chars...1
DEBUG: ConfigParser: secret_key->ve...11_chars...1
DEBUG: ConfigParser: host_base->{{HOST}}:80
DEBUG: ConfigParser: host_bucket->%(bucket)s.{{HOST}}:80
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: website_endpoint->http://192.168.1.101:80/answertoall
DEBUG: Updating Config.Config cache_file -> 
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'mb' using UTF-8
DEBUG: Unicodising 's3://testbucket' using UTF-8
DEBUG: Command: mb
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(testbucket): testbucket.{{HOST}}:80
DEBUG: canonical_headers = host:testbucket.{{HOST}}:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z

DEBUG: Canonical Request:
PUT
/

host:testbucket.{{HOST}}:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=accessKey1/20160628/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=44805e7ecae4b14c918a92142513f58b19dbc3051b175f66fe85d8c3b322b237', 'x-amz-date': '20160628T105414Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(testbucket): testbucket.{{HOST}}:80
DEBUG: ConnMan.get(): creating new connection: http://testbucket.{{HOST}}:80
DEBUG: non-proxied HTTPConnection(testbucket.{{HOST}}:80)
DEBUG: format_uri(): /
DEBUG: Sending request method_string='PUT', uri='/', headers={'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=accessKey1/20160628/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=44805e7ecae4b14c918a92142513f58b19dbc3051b175f66fe85d8c3b322b237', 'x-amz-date': '20160628T105414Z'}, body=(0 bytes)
DEBUG: Response: {'status': 403, 'headers': {'x-amz-id-2': '781591391d87c1f6a3d3', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'keep-alive', 'x-amz-request-id': '781591391d87c1f6a3d3', 'date': 'Tue, 28 Jun 2016 10:54:15 GMT', 'content-type': 'application/xml'}, 'reason': 'Forbidden', 'data': '<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message><Resource></Resource><RequestId>781591391d87c1f6a3d3</RequestId></Error>'}
DEBUG: ConnMan.put(): connection put back to pool (http://testbucket.{{HOST}}:80#1)
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: x-amz-id-2: 781591391d87c1f6a3d3
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: connection: keep-alive
DEBUG: HttpHeader: x-amz-request-id: 781591391d87c1f6a3d3
DEBUG: HttpHeader: date: Tue, 28 Jun 2016 10:54:15 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided.'
DEBUG: ErrorXML: Resource: None
DEBUG: ErrorXML: RequestId: '781591391d87c1f6a3d3'
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided.

And here is the log from our Kibana instance (First, the CanonicalRequest, and second the constructed stringToSign):

{
  "_index": "logstash-scality-2016.06.28",
  "_type": "log",
  "_id": "AVWWpUvFjIh-3to3jiRi",
  "_score": null,
  "_source": {
    "name": "S3",
    "bucketName": "testbucket",
    "canonicalReqResult": "PUT\n/\n\nhost:testbucket.{{HOST}}\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20160628T105414Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "req_id": "781591391d87c1f6a3d3",
    "level": "debug",
    "message": "constructed canonicalRequest",
    "hostname": "connect05.scality.riakpoc.local",
    "pid": 85,
    "@version": "1",
    "@timestamp": "2016-06-28T10:54:15.022Z",
    "fields": {},
    "beat": {},
    "type": "log",
    "count": 1
  },
  "fields": {
    "@timestamp": [
      1467111255022
    ]
  },
  "highlight": {
    "req_id": [
      "@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
    ],
    "req_id.raw": [
      "@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1467111255022
  ]
}
{
  "_index": "logstash-scality-2016.06.28",
  "_type": "log",
  "_id": "AVWWpUvFjIh-3to3jiRl",
  "_score": null,
  "_source": {
    "name": "S3",
    "bucketName": "testbucket",
    "stringToSign": "AWS4-HMAC-SHA256\n20160628T105414Z\n20160628/US/s3/aws4_request\n1dff0237362e0f8b069cba1748f99ccca9221cc54a6d37187a268071d38e65e4",
    "req_id": "781591391d87c1f6a3d3",
    "level": "trace",
    "message": "constructed stringToSign",
    "hostname": "connect05.scality.riakpoc.local",
    "pid": 85,
    "@version": "1",
    "@timestamp": "2016-06-28T10:54:15.022Z",
    "type": "log",
    "count": 1,
    "fields": {},
    "beat": {}
  },
  "fields": {
    "@timestamp": [
      1467111255022
    ]
  },
  "highlight": {
    "req_id": [
      "@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
    ],
    "req_id.raw": [
      "@kibana-highlighted-field@781591391d87c1f6a3d3@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1467111255022
  ]
}

As you can see, comparing the two canonical Requests:
(from s3cmd)

PUT
/

host:testbucket.{{HOST}}:80
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

(from S3)

PUT
/

host:testbucket.{{HOST}}
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160628T105414Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

We can see that the only difference is that s3cmd includes the target's port into the canonical request, while the S3 server computes it without a port. In this setup here, we're using HAProxy to load-balance the requests only multiple S3 servers, so it makes sense to avoid using the port inside S3; but as s3cmd is actually using it, it may make the v4 auth fail for no real valid reason.

I do not think it is for us to fix, but we should at least document it properly.

@vrancurel @GiorgioRegni what is your take on this ?

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions