Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS iOS SDK working with minio #4039

Closed
atom2ueki opened this issue Apr 3, 2017 · 54 comments · Fixed by #4053 or #4064
Closed

AWS iOS SDK working with minio #4039

atom2ueki opened this issue Apr 3, 2017 · 54 comments · Fixed by #4053 or #4064

Comments

@atom2ueki
Copy link

atom2ueki commented Apr 3, 2017

Error messgae

SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method./images/test.jpg3L1373L137
Upload failed with error: (The operation couldn’t be completed. (com.amazonaws.AWSServiceErrorDomain error 3.))

Here is how I configure

let accessKey = "xxxxxxx"
let secretKey = "xxxxxxx"       
let credentialsProvider = AWSStaticCredentialsProvider(accessKey: accessKey, secretKey: secretKey)
let configuration = AWSServiceConfiguration(region: AWSRegionType.USEast1, endpoint: AWSEndpoint(region: .USEast1, service: .APIGateway, url: URL( string:"http://xxxx.com:9000")),credentialsProvider: credentialsProvider)
AWSServiceManager.default().defaultServiceConfiguration = configuration

Possible Solution

refer to this issue, minio/mc#1707
maybe it caused by region issue?

Your Environment

  • Version used:

    • official docker latest
    • awss3 lib used: pod 'AWSS3', '~> 2.5'
@krishnasrinivas
Copy link
Contributor

@atom2ueki

  • Can you paste the server log when the error happens?
  • What operation did you try to do?
  • Can you paste the code for the operation you tried as well?

@deekoder deekoder added this to the Edge cache milestone Apr 3, 2017
@atom2ueki
Copy link
Author

atom2ueki commented Apr 4, 2017

@krishnasrinivas

  • Can you paste the server log when the error happens?
    time="2017-04-03T19:19:01Z" level=error msg="{\"method\":\"PUT\",\"path\":\"/images/test.jpg\",\"query\":\"\",\"header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-us\"],\"Authorization\":[\"AWS4-HMAC-SHA256 Credential=V0CZ2XP1SCEOEN2M2NNA/20170403/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-acl;x-amz-date, Signature=59d975f48adc3e1fd0cacb927eecbed366757b1baa6ead34a7d4fed9ee2c3a36\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"225491\"],\"Content-Type\":[\"image/jpeg\"],\"Host\":[\"atom2ueki.com\"],\"User-Agent\":[\"aws-sdk-iOS/2.5.1 iOS/10.3 en_US transfer-manager\"],\"X-Amz-Acl\":[\"public-read\"],\"X-Amz-Date\":[\"20170403T191901Z\"]}}" cause="Signature does not match" source="[object-handlers.go:472:objectAPIHandlers.PutObjectHandler()]"

  • What operation did you try to do?
    I'm trying to upload a jpeg format image to a bucket named images

  • Can you paste the code for the operation you tried as well?

let accessKey = "xxxxxxx"
let secretKey = "xxxxxxx"       
let credentialsProvider = AWSStaticCredentialsProvider(accessKey: accessKey, secretKey: secretKey)
let configuration = AWSServiceConfiguration(region: AWSRegionType.USEast1, endpoint: AWSEndpoint(region: .USEast1, service: .APIGateway, url: URL( string:"http://xxxx.com:9000")),credentialsProvider: credentialsProvider)
AWSServiceManager.default().defaultServiceConfiguration = configuration

let S3BucketName = "images"
let remoteName = "test.jpg"
let fileURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent(remoteName)
let image = UIImage(named: "test")
let data = UIImageJPEGRepresentation(image!, 0.9)
do {
	try data?.write(to: fileURL)
}
catch {}

let uploadRequest = AWSS3TransferManagerUploadRequest()!
uploadRequest.body = fileURL
uploadRequest.key = remoteName
uploadRequest.bucket = S3BucketName
uploadRequest.contentType = "image/jpeg"
uploadRequest.acl = .publicRead

transferManager.upload(uploadRequest).continueWithBlock { (task) -> AnyObject! in
	dispatch_async(dispatch_get_main_queue()) {
	  self.myActivityIndicator.stopAnimating()
	}

	if let error = task.error {
	    print("Upload failed with error: (\(error.localizedDescription))")
	}

	if let exception = task.exception {
	    print("Upload failed with exception (\(exception))")
	}

	if task.result != nil {
	    print("success!")
	}
	else {
	    print("Unexpected empty result.")
	}

	return nil
}

@harshavardhana
Copy link
Member

let configuration = AWSServiceConfiguration(region: AWSRegionType.USEast1, endpoint: AWSEndpoint(region: .USEast1, service: .APIGateway, url: URL( string:"http://xxxx.com:9000")),credentialsProvider: credentialsProvider)

Here the service looks like is pointing to .APIGateway and also the host header is wrong

\"Host\":[\"atom2ueki.com\"]

Are you using a proxy in between Minio and your iOS App?

@atom2ueki
Copy link
Author

@harshavardhana no, there is no proxy, so what should I put the services here if not .APIGateway , should I map http://xxxx.com:9000 to a subdomain ?

@krishnasrinivas
Copy link
Contributor

url: URL( string:"http://xxxx.com:9000"))

@atom2ueki this is expected to appear in Host header. But what you pasted was:

"Host":["atom2ueki.com"]

So we guessed a proxy might be meddling the Host header. If not we have to dig deeper.

BTW xxxx.com is atom2ueki.com right?

@atom2ueki
Copy link
Author

atom2ueki commented Apr 4, 2017

@krishnasrinivas yes, you're right, and now I've setup tls on s3.atom2ueki.com, and got another error

2017-04-05 00:47:45.388 MinioTest[1318:11670] AWSiOSSDK v2.5.1 [Debug] AWSURLSessionManager.m line:566 | -[AWSURLSessionManager printHTTPHeadersForResponse:] | Response headers:
{
    "Accept-Ranges" = bytes;
    "Content-Type" = "application/xml";
    Date = "Tue, 04 Apr 2017 16:47:45 GMT";
    Server = "nginx/1.11.10";
    Vary = Origin;
    "x-amz-request-id" = 14B23FB32BFCFE24;
}
2017-04-05 00:47:45.388 MinioTest[1318:11670] AWSiOSSDK v2.5.1 [Debug] AWSURLResponseSerialization.m line:240 | -[AWSXMLResponseSerializer responseObjectForResponse:originalRequest:currentRequest:data:error:] | Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>There were headers present in the request which were not signed</Message><Key></Key><BucketName></BucketName><Resource>/images/test.jpg</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>
Upload failed with error: (The operation couldn’t be completed. (com.amazonaws.AWSServiceErrorDomain error 11.))

says that There were headers present in the request which were not signed

@atom2ueki
Copy link
Author

atom2ueki commented Apr 4, 2017

@krishnasrinivas

configuration

let credentialsProvider = AWSStaticCredentialsProvider(accessKey: accessKey, secretKey: secretKey)
let configuration = AWSServiceConfiguration(region: AWSRegionType.USEast1, endpoint: AWSEndpoint(region: .USEast1, service: .S3, url: URL(string:"https://s3.atom2ueki.com")),credentialsProvider: credentialsProvider)

Logs from docker

time="2017-04-04T17:00:06Z" level=error msg="{\"method\":\"PUT\",\"path\":\"/images/test.jpg\",\"query\":\"\",\"header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-us\"],\"Authorization\":[\"AWS4-HMAC-SHA256 Credential=7X9BOW4CO78JFUD5EQ4D/20170404/us-east-1/s3/aws4_request, SignedHeaders=content-encoding;content-type;host;transfer-encoding;user-agent;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=aa77d8c12270e1de577152d830393fa7eee4752e11639658127853df562f823e\"],\"Connection\":[\"close\"],\"Content-Encoding\":[\"aws-chunked\"],\"Content-Length\":[\"230678\"],\"Content-Type\":[\"image/jpeg\"],\"Host\":[\"s3.atom2ueki.com\"],\"User-Agent\":[\"aws-sdk-iOS/2.5.1 iOS/10.3 en_US transfer-manager\"],\"X-Amz-Acl\":[\"public-read\"],\"X-Amz-Content-Sha256\":[\"STREAMING-AWS4-HMAC-SHA256-PAYLOAD\"],\"X-Amz-Date\":[\"20170404T170006Z\"],\"X-Amz-Decoded-Content-Length\":[\"225491\"],\"X-Forwarded-For\":[\"175.156.203.252\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Ssl\":[\"on\"],\"X-Real-Ip\":[\"175.156.203.252\"]}}" cause="Signature does not match" source="[object-handlers.go:457:objectAPIHandlers.PutObjectHandler()]"

@krishnasrinivas
Copy link
Contributor

SignedHeaders=content-encoding;content-type;host;transfer-encoding;user-agent;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length

but transfer-encoding header is not available in the headers list - I think nginx is removing it?

@harshavardhana
Copy link
Member

The transfer-encoding header is removed by Go server we might have to support this, just like how they remove "Expect" header.

  1334              if w.req.Method == "HEAD" || !bodyAllowedForStatus(code) {
  1335			// do nothing
  1336		} else if code == StatusNoContent {
  1337			delHeader("Transfer-Encoding")
  1338		} else if hasCL {
  1339			delHeader("Transfer-Encoding")
  1340		} else if w.req.ProtoAtLeast(1, 1) {
  1341			// HTTP/1.1 or greater: Transfer-Encoding has been set to identity,  and no
  1342			// content-length has been provided. The connection must be closed after the
  1343			// reply is written, and no chunking is to be done. This is the setup
  1344			// recommended in the Server-Sent Events candidate recommendation 11,
  1345			// section 8.
  1346			if hasTE && te == "identity" {
  1347				cw.chunking = false
  1348				w.closeAfterReply = true

@krishnasrinivas
Copy link
Contributor

But the above code is in func (cw *chunkWriter) writeHeader(p []byte) i.e for headers sent to the client.

It must be getting removed somewhere else.

@harshavardhana
Copy link
Member

But the above code is in func (cw *chunkWriter) writeHeader(p []byte) i.e for headers sent to the client.

It must be getting removed somewhere else.

Yes you are right, but i remember seeing it somewhere that they remove it. It could be a potential bug too since Transfer-Encoding can have different encoding mechanisms they shouldn't remove it because the value is necessary.

@krishnasrinivas
Copy link
Contributor

krishnasrinivas commented Apr 4, 2017

diff --git a/src/main/minio.js b/src/main/minio.js
index 549b615..89d1405 100644
--- a/src/main/minio.js
+++ b/src/main/minio.js
@@ -377,6 +377,7 @@ export class Client {
         if (!this.enableSHA256) sha256sum = 'UNSIGNED-PAYLOAD'
         reqOptions.headers['x-amz-date'] = Moment().utc().format('YYYYMMDDTHHmmss') + 'Z'
         reqOptions.headers['x-amz-content-sha256'] = sha256sum
+       reqOptions.headers['transfer-encoding'] = 'identity'
         var authorization = signV4(reqOptions, this.accessKey, this.secretKey, region)
         reqOptions.headers.authorization = authorization
       }

reproduces this problem without proxy

@krishnasrinivas
Copy link
Contributor

@atom2ueki we'll fix it.

@atom2ueki
Copy link
Author

atom2ueki commented Apr 6, 2017

@krishnasrinivas @harshavardhana
I've update my docker image with minio/minio:edge, but I still get error form server side, client side get a successful upload url, but server side have error

time="2017-04-06T03:21:13Z" level=error msg="{\"method\":\"PUT\",\"path\":\"/images/test.jpg\",\"query\":\"\",\"header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"zh-tw\"],\"Authorization\":[\"AWS4-HMAC-SHA256 Credential=TNZ0P8AZS8TZLRGNW1PL/20170406/us-east-1/s3/aws4_request, SignedHeaders=content-encoding;content-type;host;transfer-encoding;user-agent;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=ae1eb78ce3d32224cddb16f94da8294c61d909509f710c76404341f925e14128\"],\"Connection\":[\"close\"],\"Content-Encoding\":[\"aws-chunked\"],\"Content-Length\":[\"30917\"],\"Content-Type\":[\"image/jpeg\"],\"Host\":[\"s3.atom2ueki.com\"],\"User-Agent\":[\"aws-sdk-iOS/2.5.2 iOS/10.3 zh_HK transfer-manager\"],\"X-Amz-Acl\":[\"public-read\"],\"X-Amz-Content-Sha256\":[\"STREAMING-AWS4-HMAC-SHA256-PAYLOAD\"],\"X-Amz-Date\":[\"20170406T032113Z\"],\"X-Amz-Decoded-Content-Length\":[\"30098\"],\"X-Forwarded-For\":[\"132.147.98.29\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Ssl\":[\"on\"],\"X-Real-Ip\":[\"132.147.98.29\"]}}" cause="Signature does not match" source="[object-handlers.go:494:objectAPIHandlers.PutObjectHandler()]"

my configuration changed a bit, endpoint url changed from https://s3.atom2ueki.com to https://s3.atom2ueki.com/minio

request header

{
    Authorization = "AWS4-HMAC-SHA256 Credential=TNZ0P8AZS8TZLRGNW1PL/20170406/us-east-1/s3/aws4_request, SignedHeaders=content-encoding;content-type;host;transfer-encoding;user-agent;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f414b0f7014f0060953340564087e15dc893170da3e8b566aae4181465cb5a80";
    "Content-Encoding" = "aws-chunked";
    "Content-Type" = "image/jpeg";
    Host = "s3.atom2ueki.com";
    "Transfer-Encoding" = Chunked;
    "User-Agent" = "aws-sdk-iOS/2.5.2 iOS/10.3 zh_HK transfer-manager";
    "X-Amz-Date" = 20170406T042148Z;
    "x-amz-acl" = "public-read";
    "x-amz-content-sha256" = "STREAMING-AWS4-HMAC-SHA256-PAYLOAD";
    "x-amz-decoded-content-length" = 30098;
}

@harshavardhana harshavardhana reopened this Apr 6, 2017
@harshavardhana
Copy link
Member

Ping @atom2ueki the request header is set by you or AWS SDK. ? Looks like transfer-encoding has a wrong value is odd here?

@atom2ueki
Copy link
Author

atom2ueki commented Apr 6, 2017

@harshavardhana yes, this header is getting from

transferManager.upload(uploadRequest).continueWith { (task: AWSTask<AnyObject>) -> Any? in
    print(task)
}

if you free, you can have a try with the code I shared with you

@harshavardhana
Copy link
Member

Will do thanks @atom2ueki

krishnasrinivas added a commit to krishnasrinivas/minio that referenced this issue Apr 9, 2017
fixes minio#4039

When x-amz-content-sha256 header is missing assume e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
as the SHA256 value for calculating canonical-request.
@harshavardhana
Copy link
Member

Keeping this open until we validate the fix merged in with the aws-ios-sdk #4064

@krishnasrinivas
Copy link
Contributor

krishnasrinivas commented Apr 10, 2017

@atom2ueki can you check with minio:edge

docker pull minio/minio:edge

@atom2ueki
Copy link
Author

@krishnasrinivas @harshavardhana

server side error changed to

cause="Signature does not match" source="[object-handlers.go:513:objectAPIHandlers.PutObjectHandler()]"

client side error remain

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key></Key><BucketName></BucketName><Resource>/images/test.jpg</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>
<AWSTask: 0x600000077900; completed = YES; cancelled = NO; faulted = YES; result = (null)>

@harshavardhana
Copy link
Member

@atom2ueki we tested your MinioTest project which seems to be working fine. do you have a sample code which we can try?

@harshavardhana
Copy link
Member

cause="Signature does not match" source="[object-handlers.go:513:objectAPIHandlers.PutObjectHandler()]"

Can you paste also the headers printed this looks like a short message.

@atom2ueki
Copy link
Author

@harshavardhana I'm using same project I shared to you, let me ensure everything is setup correctly and test again

@deekoder
Copy link
Contributor

@hackintoshrao - Mint should test for this.

@krishnasrinivas
Copy link
Contributor

krishnasrinivas commented Apr 11, 2017

@atom2ueki does MinioTest repo have all the changes that you have locally? Because when we run from MinioTest it uses non-streaming signature, but the headers that you have pasted shows streaming signature. If you have any changes locally can you push it?

@harshavardhana
Copy link
Member

@atom2ueki here are the results

2017-04-11 18:55:35.279 MinioTest[25445:741850] AWSiOSSDK v2.5.2 [Verbose] AWSSignature.m line:327 | -[AWSSignatureV4Signer signRequestV4:credentials:] | AWS4 Canonical Request: [PUT
/testbucket/test.jpg

content-length:4194304
content-type:image/jpeg
host:churchofminio.com
user-agent:aws-sdk-iOS/2.5.2 iOS/10.2 en_US transfer-manager
x-amz-acl:public-read
x-amz-date:20170412T015535Z

content-length;content-type;host;user-agent;x-amz-acl;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855]
2017-04-11 18:55:35.280 MinioTest[25445:741850] AWSiOSSDK v2.5.2 [Verbose] AWSSignature.m line:328 | -[AWSSignatureV4Signer signRequestV4:credentials:] | payload 
2017-04-11 18:55:35.280 MinioTest[25445:741850] AWSiOSSDK v2.5.2 [Verbose] AWSSignature.m line:344 | -[AWSSignatureV4Signer signRequestV4:credentials:] | AWS4 String to Sign: [AWS4-HMAC-SHA256
20170412T015535Z
20170412/us-east-1/s3/aws4_request
3558115c5de5184bd2f4539674331ba3a8c0bd09bc20b4898e0201d12a9dbe05]
2017-04-11 18:55:35.282 MinioTest[25445:741850] AWSiOSSDK v2.5.2 [Debug] AWSURLSessionManager.m line:537 | -[AWSURLSessionManager printHTTPHeadersAndBodyForRequest:] | Request headers:
{
    Authorization = "AWS4-HMAC-SHA256 Credential=minio/20170412/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-acl;x-amz-date, Signature=aceb7fd3926f4052538151db41e1259afe05205b78b3ce2e3a508c507ee74721";
    "Content-Length" = 4194304;
    "Content-Type" = "image/jpeg";
    Host = "churchofminio.com";
    "User-Agent" = "aws-sdk-iOS/2.5.2 iOS/10.2 en_US transfer-manager";
    "X-Amz-Date" = 20170412T015535Z;
    "x-amz-acl" = "public-read";
}
2017-04-11 18:55:38.696 MinioTest[25445:741911] AWSiOSSDK v2.5.2 [Debug] AWSURLSessionManager.m line:566 | -[AWSURLSessionManager printHTTPHeadersForResponse:] | Response headers:
{
    "Accept-Ranges" = bytes;
    "Content-Length" = 0;
    "Content-Type" = "text/plain; charset=utf-8";
    Date = "Wed, 12 Apr 2017 01:55:38 GMT";
    Etag = "\"b5cfa9d6c8febd618f91ac2843d50a1c\"";
    Server = "Minio/DEVELOPMENT.GOGET (linux; amd64)";
    Vary = Origin;
    "X-Amz-Bucket-Region" = "us-east-1";
    "X-Amz-Request-Id" = 14B483A90D10E917;
}
<AWSTask: 0x608000466300; completed = YES; cancelled = NO; faulted = NO; result = <AWSS3TransferManagerUploadOutput: 0x60000008cbc0> {
    ETag = "\"b5cfa9d6c8febd618f91ac2843d50a1c\"";
    requestCharged = 0;
    serverSideEncryption = 0;
}>
Uploading to:Optional(https://churchofminio.com/testbucket/test.jpg)
Uploaded to:Optional(https://churchofminio.com/testbucket/test.jpg)

We used https://churchofminio.com with accessKey: minio and secretKey: minio123 - this is the latest code from MinioTest repo.

@atom2ueki
Copy link
Author

@harshavardhana @krishnasrinivas I get this works! you guys awesome, thanks

@harshavardhana
Copy link
Member

@harshavardhana @krishnasrinivas I get this works! you guys awesome, thanks

aws-amplify/aws-sdk-ios#638 here is the relevant PR with this change it works with nginx+minio on https://s3.atom2ueki.com

Thanks for all the help on this @atom2ueki

@harshavardhana
Copy link
Member

// cc @laurentgdh here since you opened #4101

@deekoder
Copy link
Contributor

deekoder commented Apr 13, 2017

Can this be closed then ? @atom2ueki @krishnasrinivas

@laurentgdh
Copy link

Thanks @harshavardhana ! It's working now.

@atom2ueki
Copy link
Author

@deekoder yes, this one can closed

@deekoder
Copy link
Contributor

File an issue on golang @harshavardhana and then close this issue.

@harshavardhana
Copy link
Member

aws-amplify/aws-sdk-ios#638 fixed upstream.

@lock
Copy link

lock bot commented Apr 26, 2020

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Apr 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.